Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Overview

DataLake is a software component of ONAP that can systematically persist the events in DMaaP into supported Big Data storage systems. It has a Admin UI, where a system administrator configures which Topics to be monitored, and to which data storage to store the data. It is also used to manage the settings of the storage and associated data analytics tool. The second part is the Feeder, which does the data transfer work and is horizontal scalable. In the next release, R7, we will add the third component, Data Exposure Service (EDS), which will expose the data in the data storage via REST API for other ONAP components and external systems to consume. Each data exposure only requires simple configurations.

Architecture Diagram

...

Image Added

Data Exposure Service will be available in R7.

Artefacts

Βlueprint (deployment artifact) :

Input file (deployment input)    :

Docker image                            : nexus3.onap.org:10001/onap/<>

Deployment Prerequisite/dependencies

Deployment Steps

Deployment of dl-handler can be done using Dashboard UI or CloudifyUI or via CLI. Below steps are based on CLI.

...

k8s-datalake-feeder.yaml, k8s-datalake-admin-ui.yaml

Docker image:

feeder, onap/org.onap.dcaegen2.services.datalakefeeder:1.0.0

admin UI, onap/org.onap.dcaegen2.services.datalakeadminui:1.0.1

Deployment Prerequisite/dependencies

Since datalake can log the message from the DMaap to several different external databases, such as Elasticsearch, Couch Base, MongoDB, Relational databases...etc. Once Datalake is successfully deployed, you can start to configure the external databases through our admin UI. The following sections will guide you to deploy datalake microservice, including cloudify blueprint upload, deployment, and un-deployment.

Deployment Steps

DL-handler consists of two pods- the feeder and admin UI. It can be deployed by using cloudify blueprint. Datalake can be easily deployed through DCAE cloudify manager. The following steps guides you launch Datalake though cloudify manager.

Log-in to the DCAE bootstrap POD's main container

...

First, we should find the bootstrap pod name through the following command and make sure that DCAE coudify manager is properly deployed.

Image Added

Login to the DCAE bootstrap pod through the following command.

Code Block
languagebash
themeMidnight
title

...

Login to the bootstrap pod
linenumberstrue

...

cfy blueprints validate /blueprints/k8s-dl-handler.yaml
kubectl exec -it <DCAE bootstrap pod> /bin/bash -n onap

Validate blueprint

Code Block
languagebash
themeMidnight
title

...

Validate Blueprint
linenumberstrue
cfy blueprints 

...

If the version of plugin used are different, update the blueprint import to match.

Deploy Service

validate /blueprints/k8s-dl-handler.yaml

Upload the blueprint to cloudify manager.

Code Block
languagebash
themeMidnight
titleUpload

...

blueprint to cloudify manager
linenumberstrue
cfy blueprint 

...

upload -b 

...

datalake-

...

feeder 

...

/bluerints/k8s-datalake-feeder.yaml
cfy blueprint upload -b datalake-admin-ui /blueprints/k8s-

...

datalake-admin-

...

ui.yaml

To un-deploy

...

Verify Blueprint Upload

Code Block
languagebash
themeMidnight
title

...

Verify Upload
linenumberstrue
cfy 

...

Delete blueprint

blueprint list

You can see the following returned message to show the blueprints have been correctly uploaded.

Image Added

Verify Plugin versions in target Cloudify instance match to blueprint imports

If the version of the plugin used is different, update the blueprint import to match.

Code Block
languagebash
themeMidnight
title

...

Verify Plugin version
linenumberstrue
cfy 

...

Initial Validation

...

plugins list

Create Deployment

Here we are going to create deployments for both feeder and admin UI.

Code Block
languagebash
themeMidnight
titleVerify Heartbeat is runningInput file
linenumberstrue
root@k8s-rancher:~# kubectl get pods -n onap | egrep "dl-handler"

...

cfy deployments create -b datalake-feeder feeder-deploy
cfy deployments create -b datalake-admin-ui admin-ui-deploy

Launch Service

Next, we are going to launch the datalake.

Code Block
languagebash
themeMidnight
titleVerify Logs for Dmaap pollUpload and deploy blueprint
linenumberstrue

Functional tests

...

cfy executions start -d feeder-deploy install
cfy executions start -d admin-ui-deploy install

To Un-deploy

Uninstall running component and delete deployment

code
Code Block
languagebash
themeMidnight
titleConfiguration

<Add below steps to configure DL-Handler to subscribe and feed into external DL with step-by-step procedure>

Dynamic Configuration Update

As the dl-handler service periodically polls Consul KV using configbindingService api's - the run time configuration of dl-handler service can be updated dynamically without having to redeploy/restart the service. The updates to configuration can be triggered either from Policy (or CLAMP) or made directly in Consul.

Locate the servicename by executing into dl-handler Service pod and getting env HOSTNAME value

Uninstall component
linenumberstrue
cfy uninstall feeder-deploy
cfy uninstall admin-ui-deploy 

Delete blueprint

Code Block
languagebash
themeMidnight
titleServiceName
root@k8s-rancher:~# kubectl exec -it -n onap dep-s78f36f2daf0843518f2e25184769eb8b-dcae-dl-handler-servithzx2 /bin/bash
Defaulting container name to s78f36f2daf0843518f2e25184769eb8b-dcae-dl-handler-service.
Use 'kubectl describe pod/dep-s78f36f2daf0843518f2e25184769eb8b-dcae-dl-handler-servithzx2 -n onap' to see all of the containers in this pod.

misshtbt@s78f36f2daf0843518f2e25184769eb8b-dcae-dl-handler-service:~/bin$ env | grep HOSTNAME
HOSTNAME=s78f36f2daf0843518f2e25184769eb8b-dcae-dl-handler-service

...

Delete blueprint
linenumberstrue
cfy blueprints delete datalake-feeder
cfy blueprints deltet datalake-admin-ui

Deploy external database

Code Block
languagebash
themeMidnight
titleConsul URL
http://<k8snodeip>:30270/ui/#/dc1/kv/

...

Deploy MongoDB as the external database
linenumberstrue
docker run -d -p 27017:27017 --name mongodb mongo
docker start mongodb