DataFile Collector (5G Usecase)

Overview

DataFile Collector provides ONAP Operators the ability to upload PM XML Files from PNFs, by sending a fileReady VES Event. These files are published to the DataRouter for further use within ONAP.

Blueprint/model/image

Deployment Prerequisite/dependencies

1. Make sure that cfy is installed and configured to work with the Cloudify deployment.
2. Make sure the Message Router and Data Router are running.

Deployment Steps

1. Download the dfc blueprint.
2. Transfer blueprint file in DCAE bootstrap POD under /blueprints directory.

3. Execute bash on the DCAE bootstrap POD. 

   kubectl -n onap exec -it <dev-dcaegen2-dcae-bootstrap> bash

4. Run Cloudify install command to install dfc.

   cfy install <dfc-blueprint-path> --blueprint-id <blueprint-name> --deployment-id <deployment-name>

    Sample output

   $ cfy install /blueprints/k8s-datafile.yaml --blueprint-id datafile --deployment-id datafile

   Uploading blueprint /blueprints/k8s-datafile.yaml...
   k8s-datafile.yaml |###################################################| 100.0%
   Blueprint uploaded. The blueprint's id is datafile
   Creating new deployment from blueprint datafile...
   Deployment created. The deployment's id is datafile
   Executing workflow install on deployment datafile [timeout=900 seconds]
   Deployment environment creation is pending...
   2019-05-20 11:35:50.030 CFY <datafile> Starting 'create_deployment_environment' workflow execution
   2019-05-20 11:35:50.727 CFY <datafile> Installing deployment plugins
   2019-05-20 11:35:50.727 CFY <datafile> Sending task 'cloudify_agent.operations.install_plugins'
   2019-05-20 11:35:50.727 CFY <datafile> Task started 'cloudify_agent.operations.install_plugins'
   2019-05-20 11:35:51.327 LOG <datafile> INFO: Installing plugin: k8s
   2019-05-20 11:35:51.327 LOG <datafile> INFO: Using existing installation of managed plugin: c5ce40e9-2d3f-45f4-8fac-479f7de4f146 [package_name: k8splugin, package_version: 1.4.13, supported_platform: linux_x86_64, distribution: centos, distribution_release: core]
   2019-05-20 11:35:51.327 CFY <datafile> Task succeeded 'cloudify_agent.operations.install_plugins'
   2019-05-20 11:35:51.327 CFY <datafile> Skipping starting deployment policy engine core - no policies defined
   2019-05-20 11:35:51.327 CFY <datafile> Creating deployment work directory
   2019-05-20 11:35:52.049 CFY <datafile> 'create_deployment_environment' workflow execution succeeded
   2019-05-20 11:35:55.034 CFY <datafile> Starting 'install' workflow execution
   2019-05-20 11:35:56.265 CFY <datafile> [datafile-collector_lwvvxh] Creating node instance
   2019-05-20 11:35:56.985 CFY <datafile> [datafile-collector_lwvvxh.create] Sending task 'k8splugin.create_for_platforms'
   2019-05-20 11:35:58.707 LOG <datafile> [datafile-collector_lwvvxh.create] INFO: Added config for dcae-datafile-collector
   2019-05-20 11:35:59.214 LOG <datafile> [datafile-collector_lwvvxh.create] INFO: Done setting up: dcae-datafile-collector
   2019-05-20 11:35:59.723 CFY <datafile> [datafile-collector_lwvvxh.create] Task succeeded 'k8splugin.create_for_platforms'
   2019-05-20 11:35:59.723 CFY <datafile> [datafile-collector_lwvvxh] Node instance created
   2019-05-20 11:35:59.723 CFY <datafile> [datafile-collector_lwvvxh] Configuring node instance: nothing to do
   2019-05-20 11:36:00.332 CFY <datafile> [datafile-collector_lwvvxh] Starting node instance
   2019-05-20 11:36:00.332 CFY <datafile> [datafile-collector_lwvvxh.start] Sending task 'k8splugin.create_and_start_container_for_platforms'
   2019-05-20 11:36:02.383 LOG <datafile> [datafile-collector_lwvvxh.start] INFO: Passing k8sconfig: {'tls': {u'cert_path': u'/opt/tls/shared', u'image': u'nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.tls-init-container:1.0.3'}, 'filebeat': {u'config_map': u'dcae-filebeat-configmap', u'config_path': u'/usr/share/filebeat/filebeat.yml', u'log_path': u'/var/log/onap', u'image': u'docker.elastic.co/beats/filebeat:5.5.0', u'data_path': u'/usr/share/filebeat/data', u'config_subpath': u'filebeat.yml'}, 'consul_dns_name': u'consul-server.onap', 'image_pull_secrets': [u'onap-docker-registry-key'], 'namespace': u'onap', 'consul_host': 'consul-server:8500', 'default_k8s_location': u'central'}
   2019-05-20 11:36:01.709 LOG <datafile> [datafile-collector_lwvvxh.start] INFO: Starting k8s deployment for dcae-datafile-collector, image: nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server:1.1.2, env: {'CONSUL_HOST': u'consul-server.onap', 'CONFIG_BINDING_SERVICE': 'config-binding-service'}, kwargs: {'readiness': {u'endpoint': u'/heartbeat', u'type': u'http', u'timeout': u'1s', u'interval': u'15s'}, 'tls_info': {u'use_tls': True, u'cert_directory': u'/opt/app/datafile/etc/cert/'}, 'replicas': 1, 'labels': {'cfydeployment': u'datafile', 'cfynodeinstance': u'datafile-collector_lwvvxh', 'cfynode': u'datafile-collector'}, 'ctx': <cloudify.context.CloudifyContext object at 0x7f51a5c162d0>, 'always_pull_image': False, 'resource_config': {}, 'log_info': {u'log_directory': u'/opt/app/datafile/logs'}, u'ports': [u'8100:0', u'8433:0'], 'k8s_location': u'central'}
   2019-05-20 11:36:02.383 LOG <datafile> [datafile-collector_lwvvxh.start] INFO: k8s deployment initiated successfully for dcae-datafile-collector: {'services': ['dcae-datafile-collector'], 'namespace': u'onap', 'location': u'central', 'deployment': 'dep-dcae-datafile-collector'}
   2019-05-20 11:36:02.383 LOG <datafile> [datafile-collector_lwvvxh.start] INFO: Waiting up to 1800 secs for dcae-datafile-collector to become ready
   2019-05-20 11:36:57.264 LOG <datafile> [datafile-collector_lwvvxh.start] INFO: k8s deployment is ready for: dcae-datafile-collector
   2019-05-20 11:36:57.888 CFY <datafile> [datafile-collector_lwvvxh.start] Task succeeded 'k8splugin.create_and_start_container_for_platforms'
   2019-05-20 11:36:58.525 CFY <datafile> [datafile-collector_lwvvxh] Node instance started
   2019-05-20 11:36:58.525 CFY <datafile> 'install' workflow execution succeeded
   Finished executing workflow install on deployment datafile
   * Run 'cfy events list -e 5c39a15a-a30e-4a3e-8196-94139c5ef368' to retrieve the execution's events/logs

Validation

curl <dcaegen2-dcae-healthcheck> and check if datafile-collector is in 'ready' state. 

$ curl 10.42.181.66 | jq

{
"type": "summary",
"count": 12,
"ready": 11,
"items": [
{
"name": "dev-dcaegen2-dcae-pm-mapper",
"ready": 1,
"unavailable": 0
},
{
"name": "dev-dcaegen2-dcae-config-binding-service",
"ready": 1,
"unavailable": 0
},
{
"name": "dev-dcaegen2-dcae-inventory-api",
"ready": 1,
"unavailable": 0
},
{
"name": "dev-dcaegen2-dcae-servicechange-handler",
"ready": 0,
"unavailable": 1
},
{
"name": "dev-dcaegen2-dcae-deployment-handler",
"ready": 1,
"unavailable": 0
},
{
"name": "dev-dcaegen2-dcae-policy-handler",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-ves-collector",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-tca-analytics",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-prh",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-hv-ves-collector",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-datafile-collector",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-snmptrap-collector",
"ready": 1,
"unavailable": 0
}
]
}