Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Service Assurance with Big Data Analytics

Description

Purpose:

  • Big Data analytics to make sure that the analysis is accurate.

  • Big data frameworks to allow usage of machine learning and deep learning.

  • Avoid sending large amount of data to ONAP-Central for training,  by letting training happen near the data source (Clolud-regions).

  • ONAP scale-out performance, by distributing some functions out of ONAP-Central such as Analytics

  • Letting inferencing happen closer to the edges/cloud-regions for future closed loop operations, thereby reducing the latency for closed loop.

How it would be done?

  • Use PNDA as a base

  • Create/adapt Helm charts

  • Ensure that no HEAT based deployment is necessary.

  • Use components that are needed for normal analytics as well ML based analytics (Apache Spark latest stable release, HDFS, OpenTSDB, Kafka, Avro schema etc..)

  • Use some PNDA  specific packages - Deployment manager as one example.

  • Develop new (or enhance existing) software components 

    • that allow distribution of analytics applications to various analytics instances

    • that allow onboarding new analytics applications and  onboarding of ML/DL models.

    • that integrates with CLAMP and DCAE framework 

Dublin Scope (Activities)

  1. DCAE & OOM - Helm charts for analytics framework.  Two packages

    •  Standard package (with all SW) and inferencing package (Minimal).

  2. Use case  - Deploy Analytics framework in the cloud-regions that are based on K8S.

  3. DCAE - Update PNDA deployment manager for spark app and ML/DL model distribution

  4. DCAE -  Analytics application management & dispatcher (to dispatch application image to various cloud regions)

  5. DCAE -  ML/DL Model Management & Dispatcher

  6. MultiCloud/K8S -  Analytics application configuration profile support

  7. DCAE -  Collection and Distribution Service - CollectD to Kafka (CollectD-kafka/avro)

  8. DCAE  - Collection and  Distribution Service - Node-export & cAdvisor to Kafka (Stretch Goal)

  9. -DCAE -  Changes required to integrate with CLAMP and DCAE (E2E configuration, E2E stiching etc...) - Stretch Goal- (As per DCAE team recommendation, moving this to new EPIC story as it requires more discussion on architecture https://lf-onap.atlassian.net/browse/ONAPARC-372)

  10. DCAE -  ONAP TCA Event dispatcher micro-service (ONAP-TCA-event-dispatcher)

  11. DCAE -  Make TCA application generic or create a test TCA application (since it needs to run in cloud-region that does not have ONAP specific components) to run on any spark based framework (Get input using Kafka, Get configuration updates via Consul directly, Output via Kafka) : TCA-spark application

  12. DCAE -  Test analytics application (Before integration with DCAE)

    1. Onboard analytics framework (standard package)

    2. Instantiate analytics framework in cloud-region1 

    3. Make analytics application CSAR with helm charts to instantiate CollectD-Kafka/Avro micro service, spark-k8s-operator to submit TCA-spark-application and helm chart to bring up ONAP-TCA-event-dispatcher)

    4. Onboard analytics application 

    5. Upload TCA-spark image and get it synchronized with cloud-region1

    6. Add configuration profiles for each component of analytics application (configure to generate alarm if memory size consumption is more than XGbytes).

    7. Do something on a compute ode to use up all the memory.

    8. Check whether memory events are coming to the ONAP-TCA-event-dispatcher.

  13. DCAE - Test analytics application (after integration with DCAE) - Dependent on item 9 (stretch goal)

    1. Leverage DCAE infrastructure components 

Note: There is some discussion on doing activities 1 to 8 in PNDA project.  To be decided. But, for now,assumption is that these would be placed in ONAP repositories.  

Assumptions:

  • For Dublin, it is assumed that Analytics framework and applications are deployed where K8S is present.

  • For Dublin, only Spark+HDFS+OpenTSDB+Kafka framework is supported. Other frameworks such as flint etc.. are further study.

  • For Dublin, Kubeflow with tensorflow serving is not used for training and inferencing.  Only spark based ML Pipelines are supported.

  • For Dublin,  closed loop actions are performed only at the ONAP-Central level

 

 

Attachments

2
  • 19 Nov 2018, 04:18 PM
  • 16 Nov 2018, 08:01 PM
16% Done
0

Confluence content

mentioned on

Activity

Show:

Former user November 28, 2018 at 5:29 AM

Hi - I was suggesting separate JIRA was for tracking central CL event flow with regard to your comment.
>>we will do analytics in Edge and do CL in ONAP-Central first

The CL event flow involves other run-time ONAP components - Policy, AAI, APPC and SO for successful closed loop event processing.

It would be good to keep DCAE integration under this JIRA as main scope is dependent on PNDA/DCAE integration.

Former user November 28, 2018 at 12:59 AM

@Vijay,

I created new EPIC story as requested. Any integration with CLAMP and DCAE can be discussed here: https://lf-onap.atlassian.net/browse/ONAPARC-372

Srini

Former user November 26, 2018 at 4:53 PM

- Thanks for the updates and agree to dublin scope proposed and phased approach.  Containerization of pnda and introducing DCAE-Controller PNDA integration component will be top priority. For testing initial app/service deployment,  we can do one of the following instead of using CSAR container helm-charts

  • Build cloudify blueprint manually and deploy via Cloudify using new adapter (plugin/broker) for PNDA integration.
    This option will be more aligned to target and give a template to work with SDC/CLAMP/policy flow eventually

  • Leverage Cloudify helm plugin to deploy the application helm chart via cloudify.
    This integration could be quick if helm charts for pnda application are already exist

The dynamic service instantiation and configuration management will require PNDA application modelling and defining onboarding requirements for PNDA app (through SDC/CLAMP/POLicy). As this will be dependent on new PNDA-integration components to be introduced, we could target this post Dublin.

Btw, the above JIRA scope primarily discusses edge platform/application deployment; please note CL support in ONAP-central for edge services does requires further discussion with other teams involved in CL flow; possibly separate ARC Jira necessary to track it.

Former user November 26, 2018 at 3:37 PM

Hi Vijay,

Item 12 is for testing basic framework. Once item 9 is done, testing can happen with DCAE. I have introduced Item 13 to show case that.

It appears that (based on feedback from you all), integration with DCAE & CLAMP can take time and many things to be considered and hence may not be possible to do integration with DCAE in Dublin release and hence made Item 9 and Item 13 as stretch goals.  As I indicated in my presentation, many actions are going to be manual first in analytics-as-a-service  and integration with DCAE takes away those manual steps. Some of the points you raised are good points. We have some answers, but they need to be discussed in R4 time frame and come out with right integration.

In Dublin, we will concentrate on making PNDA work in containers and make it deployable on K8S in cloud regions. As part of that, few modules need to be developed and we will do them in Dublin.  We are hoping that these modules can go in ONAP DCAE project as a new repository. 

During Dublin time frame, we need to have discussions on integration aspects and come out with work items to do in R5. In Edge-Automation working group too, we thought of bringing minimal Policy, AAI and APPC, but we felt that there are no resources to do them in R4 time frame (as they are not modular and dependent on many other components ).  Hence we thought that we will do analytics in Edge and do CL in ONAP-Central first.

In any case, it appears that there is enough work to make PNDA work on containers first and we will focus on that in  R4.

Sounds reasonable?

 

Former user November 26, 2018 at 2:17 PM

>>12.3 – “Make analytics application CSAR with helm charts to instantiate CollectD-Kafka/Avro micro service, spark-k8s-operator to submit TCA-spark-application and helm chart to bring up ONAP-TCA-event-dispatcher)”

– 12.3 was not something we agreed upon in our discussion few weeks back; the MS deployment at edge should be deployed via cloudify tosca blueprint for consistency on LCM with central ONAP/DCAE-Controller deployed services and alignment with SDC/CLAMP/Policy flow.

 

Also IMO the proposed option of deploying analytics platform at edge via SO/multi-cloud for initial deployment simplicity introduces few challenges for central ONAP components in-addition to architecture divergence. For e.g

  • How will central AAI have information about the remote VNF/VM being monitored at edge? Event enrichment will be required for successful CL flows.

  • Policy operation config (directed toward xNF’s) should specify the appropriate target action required at edge; if the deployments are done standalone outside of ONAP – this will be not be known.

  • For application brought up standalone at edge, no ONAP central view exist for running application and CL flow they support

  • Designer/Devops will miss information of which services are stitched together and where they are being deployed.

Some of the differences can be addressed by deploying the edge instance itself consistently via OOM by deploying a subset of required run-time components (Policy, AAI, Dmaap, APP-C, besides DCAE-C/PNDA analytics platform). I strongly suggest we review this option further with OOM team. This will also ensure consistencies in deployment and application onboarding/management between central and edge. With this approach – all the edge analytics and  control loop flow can be supported by components at edge completely locally. For Cross-region cases, the interaction with central ONAP services through Central collectors (or dmaap bridging) could be leveraged.

 

Adding to discussion as well.

 

 

Unresolved

Details

Assignee

Reporter

Labels

Priority

Epic Name

Created October 31, 2018 at 12:02 PM
Updated April 28, 2020 at 5:41 PM
Resolved October 31, 2018 at 12:02 PM

Flag notifications