Overview
DataFile Collector provides ONAP Operators the ability to upload PM XML Files from PNFs, by sending a fileReady VES Event. These files are published to the DataRouter for further use within ONAP.
Click here to expand...
Bulk PM Flow:
- NF notifies DCAE that a PM file is available for upload.
- DataFile Collector uploads PM Files from NF and stores them in Data Router.
- PM Mapper creates custom Performance events (VES) from the PM File data according to PM Mapping File.
- Analytics Applications use these performance events for targeted analysis of network operations.
Blueprint/model/image
Deployment Prerequisite/dependencies
- Make sure that cfy is installed and configured to work with the Cloudify deployment.
- Make sure the Message Router and Data Router are running.
Deployment Steps
Execute bash on the cloudify manager kubernetes pod.
kubectl -n onap exec -it <dev-dcaegen2-dcae-cloudify-manager> bash
Download the dfc blueprint.
Run Cloudify Install command to install dfc.
cfy install <dfc-blueprint-path>
Sample output
$ cfy install /blueprints/k8s-datafile.yaml --blueprint-id datafile --deployment-id datafile
Uploading blueprint /blueprints/k8s-datafile.yaml...
k8s-datafile.yaml |###################################################| 100.0%
Blueprint uploaded. The blueprint's id is datafile
Creating new deployment from blueprint datafile...
Deployment created. The deployment's id is datafile
Executing workflow install on deployment datafile [timeout=900 seconds]
Deployment environment creation is pending...
2019-05-20 11:35:50.030 CFY <datafile> Starting 'create_deployment_environment' workflow execution
2019-05-20 11:35:50.727 CFY <datafile> Installing deployment plugins
2019-05-20 11:35:50.727 CFY <datafile> Sending task 'cloudify_agent.operations.install_plugins'
2019-05-20 11:35:50.727 CFY <datafile> Task started 'cloudify_agent.operations.install_plugins'
2019-05-20 11:35:51.327 LOG <datafile> INFO: Installing plugin: k8s
2019-05-20 11:35:51.327 LOG <datafile> INFO: Using existing installation of managed plugin: c5ce40e9-2d3f-45f4-8fac-479f7de4f146 [package_name: k8splugin, package_version: 1.4.13, supported_platform: linux_x86_64, distribution: centos, distribution_release: core]
2019-05-20 11:35:51.327 CFY <datafile> Task succeeded 'cloudify_agent.operations.install_plugins'
2019-05-20 11:35:51.327 CFY <datafile> Skipping starting deployment policy engine core - no policies defined
2019-05-20 11:35:51.327 CFY <datafile> Creating deployment work directory
2019-05-20 11:35:52.049 CFY <datafile> 'create_deployment_environment' workflow execution succeeded
2019-05-20 11:35:55.034 CFY <datafile> Starting 'install' workflow execution
2019-05-20 11:35:56.265 CFY <datafile> [datafile-collector_lwvvxh] Creating node instance
2019-05-20 11:35:56.985 CFY <datafile> [datafile-collector_lwvvxh.create] Sending task 'k8splugin.create_for_platforms'
2019-05-20 11:35:58.707 LOG <datafile> [datafile-collector_lwvvxh.create] INFO: Added config for dcae-datafile-collector
2019-05-20 11:35:59.214 LOG <datafile> [datafile-collector_lwvvxh.create] INFO: Done setting up: dcae-datafile-collector
2019-05-20 11:35:59.723 CFY <datafile> [datafile-collector_lwvvxh.create] Task succeeded 'k8splugin.create_for_platforms'
2019-05-20 11:35:59.723 CFY <datafile> [datafile-collector_lwvvxh] Node instance created
2019-05-20 11:35:59.723 CFY <datafile> [datafile-collector_lwvvxh] Configuring node instance: nothing to do
2019-05-20 11:36:00.332 CFY <datafile> [datafile-collector_lwvvxh] Starting node instance
2019-05-20 11:36:00.332 CFY <datafile> [datafile-collector_lwvvxh.start] Sending task 'k8splugin.create_and_start_container_for_platforms'
2019-05-20 11:36:02.383 LOG <datafile> [datafile-collector_lwvvxh.start] INFO: Passing k8sconfig: {'tls': {u'cert_path': u'/opt/tls/shared', u'image': u'nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.tls-init-container:1.0.3'}, 'filebeat': {u'config_map': u'dcae-filebeat-configmap', u'config_path': u'/usr/share/filebeat/filebeat.yml', u'log_path': u'/var/log/onap', u'image': u'docker.elastic.co/beats/filebeat:5.5.0', u'data_path': u'/usr/share/filebeat/data', u'config_subpath': u'filebeat.yml'}, 'consul_dns_name': u'consul-server.onap', 'image_pull_secrets': [u'onap-docker-registry-key'], 'namespace': u'onap', 'consul_host': 'consul-server:8500', 'default_k8s_location': u'central'}
2019-05-20 11:36:01.709 LOG <datafile> [datafile-collector_lwvvxh.start] INFO: Starting k8s deployment for dcae-datafile-collector, image: nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server:1.1.2, env: {'CONSUL_HOST': u'consul-server.onap', 'CONFIG_BINDING_SERVICE': 'config-binding-service'}, kwargs: {'readiness': {u'endpoint': u'/heartbeat', u'type': u'http', u'timeout': u'1s', u'interval': u'15s'}, 'tls_info': {u'use_tls': True, u'cert_directory': u'/opt/app/datafile/etc/cert/'}, 'replicas': 1, 'labels': {'cfydeployment': u'datafile', 'cfynodeinstance': u'datafile-collector_lwvvxh', 'cfynode': u'datafile-collector'}, 'ctx': <cloudify.context.CloudifyContext object at 0x7f51a5c162d0>, 'always_pull_image': False, 'resource_config': {}, 'log_info': {u'log_directory': u'/opt/app/datafile/logs'}, u'ports': [u'8100:0', u'8433:0'], 'k8s_location': u'central'}
2019-05-20 11:36:02.383 LOG <datafile> [datafile-collector_lwvvxh.start] INFO: k8s deployment initiated successfully for dcae-datafile-collector: {'services': ['dcae-datafile-collector'], 'namespace': u'onap', 'location': u'central', 'deployment': 'dep-dcae-datafile-collector'}
2019-05-20 11:36:02.383 LOG <datafile> [datafile-collector_lwvvxh.start] INFO: Waiting up to 1800 secs for dcae-datafile-collector to become ready
2019-05-20 11:36:57.264 LOG <datafile> [datafile-collector_lwvvxh.start] INFO: k8s deployment is ready for: dcae-datafile-collector
2019-05-20 11:36:57.888 CFY <datafile> [datafile-collector_lwvvxh.start] Task succeeded 'k8splugin.create_and_start_container_for_platforms'
2019-05-20 11:36:58.525 CFY <datafile> [datafile-collector_lwvvxh] Node instance started
2019-05-20 11:36:58.525 CFY <datafile> 'install' workflow execution succeeded
Finished executing workflow install on deployment datafile
* Run 'cfy events list -e 5c39a15a-a30e-4a3e-8196-94139c5ef368' to retrieve the execution's events/logs
Validation
curl <dcaegen2-dcae-healthcheck> and check if datafile-collector is in 'ready' state.
$ curl 10.42.181.66 | jq
{
"type": "summary",
"count": 12,
"ready": 11,
"items": [
{
"name": "dev-dcaegen2-dcae-pm-mapper",
"ready": 1,
"unavailable": 0
},
{
"name": "dev-dcaegen2-dcae-config-binding-service",
"ready": 1,
"unavailable": 0
},
{
"name": "dev-dcaegen2-dcae-inventory-api",
"ready": 1,
"unavailable": 0
},
{
"name": "dev-dcaegen2-dcae-servicechange-handler",
"ready": 0,
"unavailable": 1
},
{
"name": "dev-dcaegen2-dcae-deployment-handler",
"ready": 1,
"unavailable": 0
},
{
"name": "dev-dcaegen2-dcae-policy-handler",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-ves-collector",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-tca-analytics",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-prh",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-hv-ves-collector",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-datafile-collector",
"ready": 1,
"unavailable": 0
},
{
"name": "dep-dcae-snmptrap-collector",
"ready": 1,
"unavailable": 0
}
]
}
