Integrating PNDA
The goal of this DCAE project is to provide the PNDA platform as a deployment option that delivers a big-data analytics platform as part of DCAE.
Meetings: PNDA-DCAE integration is discussed as part of the weekly DCAE call (Thr UTC 13:00 / China 21:00 / Eastern 9:00 / Pacific 06:00 zoom.us/j/824147956) check on DCAE Weekly Meetings
Overview
Overview presentation of .
High level summary of tasks:
Installation of PNDA within DCAE:
|
Health-Check (PNDA to integrate with DCAE health check):
|
Enable Application Deployment on PNDA via DCAE:
|
Data Integration (Enable PNDA to receive data from DCAE collectors like VES, etc.)
|
Applications for PNDA
|
Release related information
Casablanca M3 Milestone for PNDA integration into DCAE:
Support for HDFS API,
VES data available in HDFS
Support for Spark Streaming API
Support for Spark Batch API
Jupyter Notebook
PNDA 5.0 Components versions
The source of truth regarding versions is available in the PNDA 5.0 release note.
Component | Version | |
|---|---|---|
Kafka | 1.1.0 | |
Kafka Manager | 1.3.3.15 | |
PNDA Deployment Manager | XXX | |
PNDA Package Repository | XXX | |
PNDA Console | XXX | |
Gobblin | 0.11.0 | |
Flink | 1.4.2 | |
Knox | 1.1.0 | |
HortonWorks | 2.6.5 | |
| Hadoop | 2.7.3 |
HBase | 1.1.2 | |
Hive | 2.1.0 | |
Spark | 1.6.3 | |
Spark | 2.3.0 | |
Oozie | 4.2.0 | |
Grafana | 5.1.3 | |
OpenTSDB | 2.3.0 | |
Consul | 1.0.3 | |
Jupyter | 4.2.1 | |
pnda API's
As part of the ongoing dcae integration with the pnda data platform, here are some pointers defining the provided pnda API’s:
Platform Data Management: https://github.com/pndaproject/platform-data-mgmnt/blob/develop/data-service/README.md
Platform Deployment Manager https://github.com/pndaproject/platform-deployment-manager#api-documentation
Platform Package Repository https://github.com/pndaproject/platform-package-repository#repository-api
List of JIRA tickets associated with PNDA for DCAE - Casablanca
List of JIRA tickets associated with PNDA for DCAE - Backlog
Content presented at ONS.EU 2018
PNDA Integration into ONAP DCAE - Slideset
PNDA Analytics Application within DCAEgen2 - Video
Comments
Hi @Donald Hunter (Deactivated), when are you planning to introduce PNDA into DCAE? By the state of the stories it looks like this is Dublin content - correct?
Thanks,
Roger
Hi @Roger Maitland,
Actually we are planning to integrate several of these stories into Casablanca. #DCAEGEN2-367 is merged but the dcaegen2/deployments job is failing because it needs a larger VM flavour and we have an open helpdesk ticket for that. The stories which touch OOM are queued, waiting for the container artifacts from dcaegen2/deployments.
Cheers,
Donald.
Hi,
As I understand, there is a deployment manager in PNDA that is used to upload packages (Analytics applications), create applications from packages and start applications. Few questions on integration with rest of ONAP:
Which ONAP components be calling these APIs? Is there any sequence diagram that help in understand full LCM.
I see from the source code that it support OOzie plugin as well as Yarn plugin via spark-streaming. Any plans to support Kubernetes plugin?
Is Oozie the only one that is helping in defining the workflow that consists of multiple applications?
I understand that OOzie support Kubernetes in Oozie 5.0 release. I hope Oozie version 5.0 supported by PNDA.
In case of Oozie, workflow is defined using XML files. In ONAP, how these XML files get created. Do you expect them to be created outside of ONAP and upload them to ONAP. If so, would they need to be uploaded to PNDA?
How do we associate various workflow XML files to appropriate VNFs and Kafka topics and collectors. Is there any super XML that connects these all.
Srini
Adding notifications...
@Vijay Kumar, @Frank Brockners and @Donald Hunter (Deactivated)
Hi Srini,
Your understanding is correct, there is a deployment manager in PNDA that has REST APIs for package upload, application creation and control.
There are open issues relating to deployment integration for ONAP that we have still plan and see if we can scope them for Dublin.
Cheers,
Donald.
Thanks Donald. I guess I assumed that the integration aspects are worked out. As part of edge-automation, we are trying to see how we can bring up networking analytics apps in remote spark clusters (deployed at the edge or regional sites). Hence, this integration aspect is very important for us.
Guidance from OOM team:
Hi Donald,
Thanks for reaching out to the OOM team.
There is some guidance I can provide to expedite the code review/merge process for your new helm charts. A lot of time has been spent towards standardizing helm charts. Although not perfect, and still evolving, we strive for a level of consistency in the templates.
Creating Helm Charts
There is a “starter” helm chart that can be found here: onap-chart. This is not a one-size fits all, but provides a basis for most charts. The importance is in the values.yaml. An attempt at standardizing configuration parameter names (based on Helm best practices), that allows for centralized hierarchical configuration. I would try to pour your specific configuration into this example. I would have recommended to clone-and-own the dcae-bootstrap helm chart instead but I see it has eliminated some of the standardized config.
Enable/disable PNDA Deployment
There isn’t really a requirement to have every chart disabled by default and then opt-in. In fact, by default, all ONAP components are deployed out-of-the-box. It can be viewed as a “demo” deployment. Customized deployments can use an override file to disable components as necessary. For PNDA, however, it would make sense to either disable the PNDA bootstrap sub chart or have the sub chart deploy but not spin up PNDA VMs unless a configuration flag is enabled and/or openstack configuration is provided. The configuration flag can be added to the DCAE values.yaml.
Today, each ONAP project (ie. DCAE, SO) can be enabled/disabled via the values.yaml (+ requirements.yaml) inside the onap parent Helm chart. Unfortunately, this does not provide control over individual subcharts like the PNDA bootstrap sub chart you’re adding to DCAE.
OpenStack Configuration
There are a few projects that need OpenStack configuration. Take a look inside ONAP values.yaml for APPC, NBI and SO configuration. I would use these as examples on how to propagate the configuration you need. There is an effort to consolidate this configuration into a single shared configuration but that is a future deliverable.
Feel free to put up draft patches as soon as possible to get early feedback from the OOM team.
Please let me know if you have any questions, comments or concerns.
Thanks,
Mike.
--
Mike Elliott
ONAP OOM PTL
Senior Architect - Amdocs