Page Comparison

This page uses 2 VMs in China Mobile lab as an example to demonstrate how to setup a development environment. 

...

1. Setup host names in both VMs and your local PC
   In both VMs and your local PC, sudo vi /etc/hosts, add these lines: (In windows, the file is C:\Windows\System32\drivers\etc\hosts.)
   

   172.30.1.74 message-router-zookeeper message-router-kafka 
   172.30.1.74 dl_druid dl_superset
   172.30.1.75 dl_es dl_couchbase

   
   

2. Install JDK 8 and Docker in on both VMs and local
   sudo apt install openjdk-8-jdk-headless
   For Docker, I install it in on a Linux VM running in on my local Windows.
   
   
3. Setup ONAP development environment
   (ref Setting Up Your Development Environment)
   In your local PC,
   

   cd ~/.m2 (On Windows, it is C:\Users\your_name\.m2)
   mv settings.xml settings.xml-old
   wget https://raw.githubusercontent.com/onap/oparent/master/settings.xml
   
   

4. Check out source code
   
   From China Mobile gitlab
   Check out DataLake source code from http://172.30.3.87/SOTN/datalake.git to ~/git/onap/dcaegen2/datalake services2 or C:\git\onap\datalakedcaegen2\services2 on Windows. If you don't have access, please apply one from Guobiao Mo.let me know.
   
   From ONAP gerrit (After seed code is checked in, see https://gerrit.onap.org/r/#/c/79055/)
   Check out DataLake source code from https://gerrit.onap.org/r/#/admin/projects/dcaegen2/services to C:\git\onap\dcaegen2\services2 or ~/git/onap/dcaegen2/services2. Currently DataLake feeder is hosted in ONAP repo as a DCAE component handler. (ref https://gerrit.onap.org/r/#/c/79055/)
   
   
5. Setup Kafka in datalake01
   (ref https://kafka.apache.org/quickstart)
   This and the following 2 steps describe setting up and using your own Kafka for development and testing. For using ONAP DMaaP, see step "Use DMaaP as data source".
   In datalake01,
   

   mkdir ~/kafka
   cd ~/kafka
   wget https://www-eu.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz
   tar -xzf kafka_2.11-2.0.0.tgz
   cd ~/kafka/kafka_2.11-2.0.0
   
   sudo vi config/server.properties 
   add this line:

   listeners=PLAINTEXT://172.30.1.74:9092

   To start Zookeeper and Kafka:
   nohup bin/zookeeper-server-start.sh config/zookeeper.properties > zk.log &
   nohup bin/kafka-server-start.sh config/server.properties > kf.log &

   Here Btw, here are commands to stop them:
   bin/zookeeper-server-stop.sh
   bin/kafka-server-stop.sh
   

   
   

6. Create test Kafka topics 
   In datalake01
   

   cd ~/kafka/kafka_2.11-2.0.0

   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic AAI-EVENT
   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic unauthenticated.DCAE_CL_OUTPUT
   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic unauthenticated.SEC_FAULT_OUTPUT
   ./bin/kafka-topics.sh --create --zookeeper message-router-zookeeper:2181 --replication-factor 1 --partitions 1 --topic msgrtr.apinode.metrics.dmaap
   
   

   In case you want to reset the topics, here are the scripts to delete them:
   

   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic AAI-EVENT
   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic unauthenticated.DCAE_CL_OUTPUT
   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic unauthenticated.SEC_FAULT_OUTPUT
   ./bin/kafka-topics.sh --zookeeper message-router-zookeeper:2181 --delete --topic msgrtr.apinode.metrics.dmaap
   
   

7. Load test data to Kafka
   The test data files are checked out from source depot in previous step "Check out source code".
   In datalake01
   
   cd ~/kafka/kafka_2.11-2.0.0
   

   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic AAI-EVENT < ~/git/onap/dcaegen2/services2/components/datalake-handler/feeder/src/main/resources/druid/AAI-EVENT-100.json
   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic unauthenticated.DCAE_CL_OUTPUT < ~/git/onap/dcaegen2/services2/components/datalake-handler/feeder/src/main/resources/druid/DCAE_CL_OUTPUT-100.json
   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic unauthenticated.SEC_FAULT_OUTPUT < ~/git/onap/dcaegen2/services2/components/datalake-handler/feeder/src/main/resources/druid/SEC_FAULT_OUTPUT-100.json
   ./bin/kafka-console-producer.sh --broker-list message-router-kafka:9092 --topic msgrtr.apinode.metrics.dmaap < ~/git/onap/dcaegen2/services2/components/datalake-handler/feeder/src/main/resources/druid/msgrtr.apinode.metrics.dmaap-100.json

   
   To check if the data is successfully loaded, one can read the data: 

   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic AAI-EVENT --from-beginning
   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic unauthenticated.DCAE_CL_OUTPUT --from-beginning
   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic unauthenticated.SEC_FAULT_OUTPUT --from-beginning
   bin/kafka-console-consumer.sh --bootstrap-server message-router-kafka:9092 --topic msgrtr.apinode.metrics.dmaap --from-beginning
   
   

8. Setup Couchbase in datalake02 
   In datalake02,
   
   • Start docker
     sudo docker pull couchbase/server-sandbox:6.0.0
     sudo docker run -d --name couchbase -p  8091-8094:8091-8094  -p 11210:11210 couchbase/server-sandbox:6.0.0
   • Create users and buckets
     

     Access http://dl_couchbase:8091/ , use login: "Administrator/password". 

     Create bucket "dl", with memory quota 100MB.
     Create user dl/dl1234 , with “Application access” to bucket "dl".

     Create bucket "dmaap", with memory quota 100MB.
     Create user dmaap/dmaap1234, with “Application Access” and "Views Admin" to bucket "dmaap".

   • Create a document for default topic in bucket dl 
     _DL_DEFAULT_
     {
     "enabled": true,
     "saveRaw": false,
     "dataFormat": "JSON",
     "ttl": 1000,
     "supportElasticsearch":true,
     "supportCouchbase":true
     }

   • Disable some topics
     To disable topic '__consumer_offsets', create this document in bucket dl:
     __consumer_offsets
     {
     "enabled": false
     }
     
     
9. Setup ElasticSearch & Kibana in datalake02 
   (ref https://docs.swiftybeaver.com/article/33-install-elasticsearch-kibana-via-docker)
   In datalake02,
   

   sudo docker pull docker.elastic.co/elasticsearch/elasticsearch:6.6.1
   sudo docker run -d --rm -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name elastic docker.elastic.co/elasticsearch/elasticsearch:6.6.1
   sudo docker pull docker.elastic.co/kibana/kibana:6.6.1
   sudo docker run -d --rm --link elastic:dl_es -e "ELASTICSEARCH_URL=http://dl_es:9200" -p 5601:5601 --name kibana docker.elastic.co/kibana/kibana:6.6.1
   
   

10. Create test Indices in ElasticSearch 
    Indices should be auto created, which will be addressed in JIRA xxx. For now we need to manually create them.

    curl -X PUT "dl_es:9200/aai-event?pretty"

    curl -X PUT "dl_es:9200/unauthenticated.dcae_cl_output?pretty"

    curl -X PUT "dl_es:9200/unauthenticated.sec_fault_output?pretty"

    curl -X PUT "dl_es:9200/msgrtr.apinode.metrics.dmaap?pretty"

    Indices should be auto created by feeder.
    To access Kibana: http://dl_es:5601/ .
    In case you want to reset the Indices, here are the scripts to delete them:
    

    curl -X DELETE "dl_es:9200/aai-event?pretty"

    curl -X DELETE "dl_es:9200/unauthenticated.dcae_cl_output?pretty"

    curl -X DELETE "dl_es:9200/unauthenticated.sec_fault_output?pretty"

    curl -X DELETE "dl_es:9200/msgrtr.apinode.metrics.dmaap?pretty"

    
    
11. Run DataLake feeder in Eclipse
    

    The feeder is a Spring boot application. The entry point is org.onap.datalake.feeder.Application. After started, the app reads the topic list from Zookeeper, and starts pulling data from these Kafka topics, and insert the data to Couchbase and Elasticsearch. 
    
    

12. Create Docker image for deployment
    To create Docker image in your local development environment, it is required to install Docker in local.
    cd ~/git/onap/dcaegen2/services2/components/datalake-handler/feeder
    mvn clean  package (or mvn clean  package -DskipTests)
    sudo docker build -t  moguobiao/datalake-feeder -f src/assembly/Dockerfile .  (You may want to replace 'moguobiao' with your name)
    
    Push docker image to dockerhub
    

    sudo docker login -u  moguobiao -p password
    sudo docker push moguobiao/datalake-feeder 

    
    To deploy the image to datalake01:
    sudo docker pull moguobiao/datalake-feeder
    sudo docker run --rm -p 1680:1680 --name dl_feeder --add-host=message-router-kafka:172.30.1.74 --add-host=message-router-zookeeper:172.30.1.74 --add-host=dl_couchbase:172.30.1.75 --add-host=dl_es:172.30.1.75 moguobiao/datalake-feeder
    
    
13. Use DMaaP as data source
    
    
14. More to come on Druid, Superset etc. ...