DataLake Development Environment Setup

This page uses 2 VMs in China Mobile lab as an example to demonstrate how to setup a development environment. 

We have 2 VMs:

datalake01: 172.30.1.74
datalake02: 172.30.1.75

1. Setup host names in both VMs and your local PC
   In both VMs and your PC, sudo vi /etc/hosts, add these lines: (In windows, the file is C:\Windows\System32\drivers\etc\hosts.)
   

   172.30.1.74 message-router-zookeeper message-router-kafka 
   172.30.1.74 dl_druid dl_superset
   172.30.1.75 dl_es dl_couchbase

   
   

2. Install JDK 8 in both VMs
   sudo apt install openjdk-8-jdk-headless
   
   
3. Setup Kafka in datalake01
   (ref https://kafka.apache.org/quickstart)
   This and the following 2 steps describe setting up and using your own Kafka for testing. For using ONAP DMaaP, see step 11.

   mkdir ~/kafka
   cd ~/kafka
   wget https://www-eu.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz
   tar -xzf kafka_2.11-2.0.0.tgz
   cd ~/kafka/kafka_2.11-2.0.0
   
   sudo vi config/server.properties 
   add this line:

   listeners=PLAINTEXT://172.30.1.74:9092

   To start Zookeeper and Kafka:
   nohup bin/zookeeper-server-start.sh config/zookeeper.properties > zk.log &
   nohup bin/kafka-server-start.sh config/server.properties > kf.log &

   Here are commands to stop them:
   bin/zookeeper-server-stop.sh
   bin/kafka-server-stop.sh
   

   
   

4. Create test Kafka topics 
   In datalake01
   

   cd ~/kafka/kafka_2.11-2.0.0

   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic AAI-EVENT
   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic unauthenticated.DCAE_CL_OUTPUT
   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic unauthenticated.SEC_FAULT_OUTPUT
   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic msgrtr.apinode.metrics.dmaap
   
   

   In case you want to reset the topics, here are the scripts to delete them:
   

   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic AAI-EVENT
   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic unauthenticated.DCAE_CL_OUTPUT
   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic unauthenticated.SEC_FAULT_OUTPUT
   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic msgrtr.apinode.metrics.dmaap
   
   

5. Load test data to Kafka
   In datalake01
   
   See step 9 for how to get the source code and data files.
   
   cd ~/kafka/kafka_2.11-2.0.0
   

   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic AAI-EVENT < ~/git/onap/datalake/storage/src/main/resources/druid/AAI-EVENT-full.json
   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic unauthenticated.DCAE_CL_OUTPUT < ~/git/onap/datalake/storage/src/main/resources/druid/DCAE_CL_OUTPUT-full.json
   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic unauthenticated.SEC_FAULT_OUTPUT < ~/git/onap/datalake/storage/src/main/resources/druid/SEC_FAULT_OUTPUT-full.json
   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic msgrtr.apinode.metrics.dmaap < ~/git/onap/datalake/storage/src/main/resources/druid/msgrtr.apinode.metrics.dmaap-full.json

   
   To check if the data is successfully loaded, one can read the data: 

   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic AAI-EVENT --from-beginning
   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic unauthenticated.DCAE_CL_OUTPUT --from-beginning
   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic unauthenticated.SEC_FAULT_OUTPUT --from-beginning
   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic msgrtr.apinode.metrics.dmaap --from-beginning
   
   

6. Setup Couchbase in datalake02 
   • Start docker
     sudo docker run -d --name couchbase -p  8091-8094:8091-8094  -p 11210:11210 couchbase/server-sandbox:6.0.0
   • Create users and buckets
     

     Access http://dl_couchbase:8091/ , use login: "Administrator/password". 

     Create bucket "dl", with memory quota 100MB.
     Create user dl/dl1234 , with “Application access” to bucket "dl".

     Create bucket "dmaap", with memory quota 100MB.
     Create user dmaap/dmaap1234, with “Application Access” and "Views Admin" to bucket "dmaap".

   • Create a document for default topic in bucket dl 
     _DL_DEFAULT_
     {
     "enabled": true,
     "saveRaw": false,
     "dataFormat": "JSON",
     "ttl": 1000,
     "supportElasticsearch":true,
     "supportCouchbase":true
     }

   • Disable some topics
     To disable topic '__consumer_offsets', create this document in bucket dl:
     __consumer_offsets
     {
     "enabled": false
     }
     
     
7. Setup ElasticSearch & Kibana in datalake02 
   (ref https://docs.swiftybeaver.com/article/33-install-elasticsearch-kibana-via-docker)
   

   sudo docker pull docker.elastic.co/elasticsearch/elasticsearch:6.6.1
   sudo docker run -d --rm -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name elastic docker.elastic.co/elasticsearch/elasticsearch:6.6.1
   sudo docker pull docker.elastic.co/kibana/kibana:6.6.1
   sudo docker run -d --rm --link elastic:dl_es -e "ELASTICSEARCH_URL=http://dl_es:9200" -p 5601:5601 --name kibana docker.elastic.co/kibana/kibana:6.6.1
   
   

8. Create test Indices in ElasticSearch 
   Indices should be auto created, which will be addressed in JIRA xxx. For now we need to manually create them.

   curl -X PUT "dl_es:9200/aai-event?pretty"

   curl -X PUT "dl_es:9200/unauthenticated.dcae_cl_output?pretty"

   curl -X PUT "dl_es:9200/unauthenticated.sec_fault_output?pretty"

   curl -X PUT "dl_es:9200/msgrtr.apinode.metrics.dmaap?pretty"

   
   To access Kibana: http://dl_es:5601/ .
   In case you want to reset the Indices, here are the scripts to delete them:
   

   curl -X DELETE "dl_es:9200/aai-event?pretty"

   curl -X DELETE "dl_es:9200/unauthenticated.dcae_cl_output?pretty"

   curl -X DELETE "dl_es:9200/unauthenticated.sec_fault_output?pretty"

   curl -X DELETE "dl_es:9200/msgrtr.apinode.metrics.dmaap?pretty"

   
   
9. Run DataLake feeder in Eclipse
   Check out DataLake source code from http://172.30.3.87/SOTN/datalake.git to ~/git/onap/datalake or C:\git\onap\datalake. If you don't have access, please apply one from Guobiao Mo.

   The feeder is a Spring boot application. The entry point is org.onap.datalake.storage.Application. After started, the app reads the topic list from Zookeeper, and starts pulling data from these Kafka topics, and insert the data to Couchbase and Elasticsearch. 
   
   

10. Create Docker image for deployment
    To create Docker image in your local development environment
    cd ~/git/onap/dcaegen2/services2/components/datalake-handler/feeder
    mvn clean  package
    sudo docker build -t  moguobiao/datalake-feeder -f src/assembly/Dockerfile .  (You may want to replace 'moguobiao' with your name)
    
    Push docker image to dockerhub
    

    sudo docker login -u  moguobiao -p password
    sudo docker push moguobiao/datalake-feeder 

    
    To deploy to datalake01:
    sudo docker pull moguobiao/datalake-feeder
    sudo docker run --rm -p 1680:1680 --name dl_storage --add-host=message-router-kafka:172.30.1.74 --add-host=message-router-zookeeper:172.30.1.74 --add-host=dl_couchbase:172.30.1.75 --add-host=dl_es:172.30.1.75 moguobiao/datalake-feeder
    
    
11. Use DMaaP as data source
    
    
12. More to come on Druid, Superset etc. ...