dcae-cli Tool - Deploy Micro Service through Component Spec
Overview
The dcae-cli
is a Python command-line tool for component developers. With it, the developer can :
- validate the data formats and component specifications
- publish the validated data formats and component specifications into the
onboarding catalog
- access the
onboarding catalog
to search for existing data formats (for possible reuse) and component specs - deploy a component onto a local or remote DCAE platform for functional and pair-wise testing (This is done without Cloudify)
The git repository for the dcae_cli tool can be found here
For official info:
- https://onap.readthedocs.io/en/latest/submodules/dcaegen2.git/docs/sections/components/dcae-cli/dcae-cli-tool.html
- https://wiki.onap.org/display/DW/MicroServices+Onboarding
Requirements:
- ONAP deployment (3 node deployment is sufficient)
- A single Ubuntu instance
Step by Step to install , validate component spec and deploy on local env:
- Execute the commands as a root user to avoid permission issues.
Clone the repo onto one of the ONAP instanaces
root@excl-xxxxxx-lego-casablanca-node-3:~# git clone https://gerrit.onap.org/r/p/dcaegen2/platform/cli.git
Cloning into 'cli'...
remote: Counting objects: 2, done
remote: Finding sources: 100% (2/2)
remote: Total 507 (delta 0), reused 507 (delta 0)
Receiving objects: 100% (507/507), 384.30 KiB | 0 bytes/s, done.
Resolving deltas: 100% (179/179), done.
Checking connectivity... done.
Install required packages: Python, Pip, Virtualenv
Install python, pip (9.0.1 or higher), and virtualenv if they are not installed. Do these when not in a VPN to avoid possible network issues.
$ sudo apt-get -f install python
$ sudo apt-get -f install python-pip
$ sudo pip install virtualenv
Install dcae_cli
$ pip install onap-dcae-cli
Check dcae_cli version
You can verify the version of the dcae-cli with the following command. To get the latest version of the dcae_cli tool,
$ dcae_cli --version
Upgrade dcae_cli
Periodically, upgrade the dcae_cli to get the latest version
pip install --upgrade onap-dcae-cli
Setup postgres database
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# mkdir postgres
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# docker run --name mypostgres -e POSTGRES_PASSWORD=dcae -e PGDATA=/var/lib/postgresql/data/pgdata -v /home/ubuntu/postgres/:/var/lib/postgresql/data/pgdata -p 5432:5432 -d postgres:9.5.2
Unable to find image 'postgres:9.5.2' locally
9.5.2: Pulling from library/postgres
8b87079b7a06: Pull complete
a3ed95caeb02: Pull complete
ff6abb23e531: Pull complete
8364ca902ad3: Pull complete
84179c1b7ff6: Pull complete
be951654637c: Pull complete
4841dfc8333f: Pull complete
8e92fd62d485: Pull complete
13e5de4be2f2: Pull complete
d6aaf4d83b1c: Pull complete
3113f93aec6d: Pull complete
055e85b433f4: Pull complete
Digest: sha256:923591dfc3bf2e6e2ea35fee759e050340dcea580b78835cbf590e7209f0a4b1
Status: Downloaded newer image for postgres:9.5.2
...
root@excl-xxxxxx-lego-casablanca-node-3:~# docker ps | grep mypostgres
ab28ca5bcc52 postgres:9.5.2 "/docker-entrypoin..." 15 seconds ago Up 15 seconds 0.0.0.0:5432->5432/tcp mypostgres
Install psql
root@excl-xxxxxx-lego-casablanca-node-3:~# apt-get install postgresql-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libpq5 postgresql-client-9.5
Suggested packages:
postgresql-9.5 postgresql-doc-9.5
The following NEW packages will be installed:
libpq5 postgresql-client postgresql-client-9.5
0 upgraded, 3 newly installed, 0 to remove and 30 not upgraded.
Need to get 957 kB of archives.
After this operation, 3,512 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpq5 amd64 9.5.14-0ubuntu0.16.04 [78.8 kB]
Get:2 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-client-9.5 amd64 9.5.14-0ubuntu0.16.04 [872 kB]
Get:3 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-client all 9.5+173ubuntu0.2 [5,468 B]
Fetched 957 kB in 1s (806 kB/s)
Selecting previously unselected package libpq5:amd64.
(Reading database ... 88946 files and directories currently installed.)
Preparing to unpack .../libpq5_9.5.14-0ubuntu0.16.04_amd64.deb ...
Unpacking libpq5:amd64 (9.5.14-0ubuntu0.16.04) ...
Selecting previously unselected package postgresql-client-9.5.
Preparing to unpack .../postgresql-client-9.5_9.5.14-0ubuntu0.16.04_amd64.deb ...
Unpacking postgresql-client-9.5 (9.5.14-0ubuntu0.16.04) ...
Selecting previously unselected package postgresql-client.
Preparing to unpack .../postgresql-client_9.5+173ubuntu0.2_all.deb ...
Unpacking postgresql-client (9.5+173ubuntu0.2) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Setting up libpq5:amd64 (9.5.14-0ubuntu0.16.04) ...
Setting up postgresql-client-9.5 (9.5.14-0ubuntu0.16.04) ...
update-alternatives: using /usr/share/postgresql/9.5/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
Setting up postgresql-client (9.5+173ubuntu0.2) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Create required database in postgress
root@excl-xxxxxx-lego-casablanca-node-3:~# docker exec -it mypostgres psql -U postgres
psql (9.5.2)
Type "help" for help.
postgres=# CREATE DATABASE dcae_onboarding_db;
postgres-# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
--------------------+----------+----------+------------+------------+-----------------------
dcae_onboarding_db | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres-# \c dcae_onboarding_db
You are now connected to database "dcae_onboarding_db" as user "postgres".
Now exit from postgres database by pressing ctrl +d
Configure dcae_cli tool for initial configuration
Set up a virtual environment and activate
root@excl-xxxxxx-lego-casablanca-node-3:~/cli# virtualenv cli_tool
New python executable in /home/ubuntu/cli/cli_tool/bin/python
Installing setuptools, pip, wheel...
done.
root@excl-xxxxxx-lego-casablanca-node-3:~/cli# source cli_tool/bin/activate
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/cli#
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# dcae_cli --reinit
Warning! Reinitializing your dcae-cli configuration
Please enter the remote server url: https://git.onap.org/dcaegen2/platform/cli/plain (Taken from https://git.onap.org/dcaegen2/platform/cli/plain/dcae-cli/README.md)
Could not download initial configuration from remote server. Attempt manually setting up? [y/N]: y
Please enter your user id: root
Now we need to set up access to the onboarding catalog
Please enter the onboarding catalog hostname: localhost
Please enter the onboarding catalog user: postgres
Please enter the onboarding catalog password: dcae
Could not download initial profiles from remote server. Set empty default? [y/N]: y
Reinitialize done
Now you can list catalog if all the configuration is done.
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# dcae_cli catalog list
Components:
+------+---------+------+-------------+-------+--------+-----------+
| Name | Version | Type | Description | Owner | Status | Published |
+------+---------+------+-------------+-------+--------+-----------+
| | | | | | | |
+------+---------+------+-------------+-------+--------+-----------+
Data formats:
+------+---------+-------------+-------+--------+-----------+
| Name | Version | Description | Owner | Status | Published |
+------+---------+-------------+-------+--------+-----------+
| | | | | | |
+------+---------+-------------+-------+--------+-----------+
Set up profile on dcae_cli tool
List all the profiles:
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# dcae_cli profiles list
* default
Show configuration of default profile:
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# dcae_cli profiles show default
{
"cdap_broker": "cdap_broker",
"config_binding_service": "config_binding_service",
"consul_host": "",
"docker_host": ""
}
Keep this terminal open, we will go back to it.
Here we need two input configurations,
consul_host ---→ Needed running ONAP
docker_host ---→ Create a new ubuntu VM and install docker
For the consul_host:
We need to expose the consul as a service, here are the steps:
Open a new terminal to the ONAP instance.
root@excl-xxxxxx-lego-casablanca-node-1:~# kubectl -n onap get svc | grep consul
consul ExternalName <none> consul-server.onap.svc.cluster.local <none> 1d
consul-server ClusterIP None <none> 8301/TCP 1d
consul-server-ui NodePort 10.43.24.247 <none> 8500:30270/TCP 1d
msb-consul NodePort 10.43.148.68 <none> 8500:30285/TCP 1d
root@excl-xxxxxx-lego-casablanca-node-1:~# kubectl -n onap expose svc consul-server-ui --type=LoadBalancer --name=my-consul
service "my-consul" exposed
root@excl-xxxxxx-lego-casablanca-node-1:~# kubectl -n onap get svc | grep consul
consul ExternalName <none> consul-server.onap.svc.cluster.local <none> 1d
consul-server ClusterIP None <none> 8301/TCP 1d
consul-server-ui NodePort 10.43.24.247 <none> 8500:30270/TCP 1d
msb-consul NodePort 10.43.148.68 <none> 8500:30285/TCP 1d
my-consul LoadBalancer 10.43.230.247 10.209.63.6 8500:32655/TCP 5s
Now we got consul host, <consul_host=10.209.63.6 >
For the Docker Host
Login to the ubuntu VM, install docker and update it's configuration to accept tcp requests..here are the steps.
Login to the ubuntu instance.
Switch to a root user
install docker:
sudo curl https://releases.rancher.com/install-docker/17.03.sh | sh
sudo touch /etc/docker/daemon.json
sudo chmod 777 /etc/docker/daemon.json
sudo bash -c "cat > /etc/docker/daemon.json <<EOF
{
\"log-driver\": \"json-file\",
\"log-opts\": {
\"max-size\": \"20m\",
\"max-file\": \"3\"
}
}
EOF"
$ sudo service docker restart
Now docker is installed but required external tcp port is not enabled.
root@excl-xxxxxx-lego-xxx:/etc/default# sudo netstat -ntlp | grep LISTEN
sudo: unable to resolve host excl-xxxxxx-lego-xxx
tcp 0 0 127.0.0.1:8879 0.0.0.0:* LISTEN 12830/helm
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 16944/sshd
tcp6 0 0 :::22 :::* LISTEN 16944/sshd
Enable tcp port for external request
Follow the below commands:
root@excl-xxxxxx-lego-xxx:/etc/systemd/system# cd /etc/systemd/system
root@excl-xxxxxx-lego-xxx:/etc/systemd/system# mkdir docker.service.d
root@excl-xxxxxx-lego-xxx:/etc/systemd/system# cd docker.service.d/
Create a new file statup_options.conf
root@excl-xxxxxx-lego-xxx:/etc/systemd/system/docker.service.d# vi startup_options.conf
# /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376
root@excl-xxxxxx-lego-xxx:/etc/systemd/system/docker.service.d# sudo systemctl daemon-reload
root@excl-xxxxxx-lego-xxx:/etc/systemd/system/docker.service.d# systemctl restart docker.service
Check that the port is enabled:
root@excl-xxxxxx-lego-xxx:/etc/systemd/system/docker.service.d# sudo netstat -ntlp | grep LISTEN
sudo: unable to resolve host excl-xxxxxx-lego-xxx
tcp 0 0 127.0.0.1:8879 0.0.0.0:* LISTEN 12830/helm
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 16944/sshd
tcp6 0 0 :::2376 :::* LISTEN 15402/dockerd
tcp6 0 0 :::22 :::* LISTEN 16944/sshd
root@excl-xxxxxx-lego-xxx:~# ps -eaf | grep docker
root 15402 1 2 14:13 ? 00:00:51 /usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376
root 15413 15402 0 14:13 ? 00:00:01 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 17570 15259 0 14:45 pts/0 00:00:00 grep --color=auto docker
root@excl-xxxxxx-lego-xxx:~# docker ps
If you get the following error, you will need to define the DOCKER_HOST.
Cannot connect to the Docker daemon at tcp://localhost:4243. Is the docker daemon running?
root@excl-xxxxxx-lego-xxx:~# DOCKER_HOST=tcp://localhost:2376
root@excl-xxxxxx-lego-xxx:~# docker ( now docker command will work )
To get the host IP:
root@excl-xxxxxx-lego-xxx:~# echo $(ip route get 8.8.8.8 | awk '/8.8.8.8/ {print $NF}')
10.209.63.33
This Ubuntu Virtual Machine external IP is our docker host. So in this case, docker_host=10.209.63.33 ( VM host ip )
Finally login to nexus.
$ docker login -u docker -p docker nexus3.onap.org:10001
Update the dcae_cli default profile
Now , we need to configure the profile with correct consul_host and docker_host. ( or if you execute the reinit command)
So go back to your terminal on the the ONAP Deployment where you were previously connected to the dcae_cli
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli profiles set default consul_host 10.209.63.6
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli profiles set default docker_host 10.209.63.33:2376
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli profiles show default
{
"cdap_broker": "cdap_broker",
"config_binding_service": "config_binding_service",
"consul_host": "10.209.63.6",
"docker_host": "10.209.63.33:2376"
}
Add data format
To validate data format:
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# vi VES_dataformat.json (Insert the above into the file)
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli data_format add VES_dataformat.json
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli data_format list
Data formats for xxxx
+-------------------+---------+---------------------+--------+----------------------------+
| Name | Version | Description | Status | Modified |
+-------------------+---------+---------------------+--------+----------------------------+
| VES_specification | 7.30.1 | VES spec for v7.0.1 | staged | 2019-01-08 11:59:52.638336 |
+-------------------+---------+---------------------+--------+----------------------------+
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli data_format publish VES_specification
Data format has been published
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli data_format list
Data formats for xxxx
+-------------------+---------+---------------------+-----------+----------------------------+
| Name | Version | Description | Status | Modified |
+-------------------+---------+---------------------+-----------+----------------------------+
| VES_specification | 7.30.1 | VES spec for v7.0.1 | published | 2019-01-08 12:01:20.791865 |
+-------------------+---------+---------------------+-----------+----------------------------+
Add component spec
Validating the component spec
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# vi pmmapper-component-spec.json (Insert the above into the file)
if you wish you can change the docker image change the line
"uri": "nexus3.onap.org:10001/onap/dmaap/datarouter-node:1.0.3", to another docker image i.e nexus3.onap.org:10001/onap/XXXX:1.0.1
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli component add pmmapper-component-spec.json
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli component list
Active profile: default
+----------------+---------+--------+------------------------------+--------+----------------------------+-----------+
| Name | Version | Type | Description | Status | Modified | #Deployed |
+----------------+---------+--------+------------------------------+--------+----------------------------+-----------+
| dcae-pm-mapper | 1.0.0 | docker | PM Mapper Docker Application | staged | 2019-01-08 12:02:16.468432 | 0 |
+----------------+---------+--------+------------------------------+--------+----------------------------+-----------+
Use the "--deployed" option to see more details on deployments
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli component publish dcae-pm-mapper
Component has been published
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli component list
Active profile: default
+----------------+---------+--------+------------------------------+-----------+----------------------------+-----------+
| Name | Version | Type | Description | Status | Modified | #Deployed |
+----------------+---------+--------+------------------------------+-----------+----------------------------+-----------+
| dcae-pm-mapper | 1.0.0 | docker | PM Mapper Docker Application | published | 2019-01-10 16:58:10.029253 | 0 |
+----------------+---------+--------+------------------------------+-----------+----------------------------+-----------+
deploy component
Deploying the micro-service: Only published specs can be deployed. Check existing catalog with published components.
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli catalog list
Components:
+----------------+---------+--------+------------------------------+----------+-----------+------------+
| Name | Version | Type | Description | Owner | Status | Published |
+----------------+---------+--------+------------------------------+----------+-----------+------------+
| dcae-pm-mapper | 1.0.0 | docker | PM Mapper Docker Application | root | published | 2019-01-08 |
+----------------+---------+--------+------------------------------+----------+-----------+------------+
Data formats:
+-------------------+---------+---------------------+----------+-----------+------------+
| Name | Version | Description | Owner | Status | Published |
+-------------------+---------+---------------------+----------+-----------+------------+
| VES_specification | 7.30.1 | VES spec for v7.0.1 | root | published | 2019-01-08 |
+-------------------+---------+---------------------+----------+-----------+------------+
Create a dmaap-file ( dmaap.json ) with below content for deployment:
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# vi dmaap.json
{
"pm_mapper_handle_out":
{
"type": "message_router",
"dmaap_info": {
"topic_url": "https://we-are-message-router.us:3905/events/some-topic"
}
}
}
Here "pm_mapper_handle_out" → This is a config key in component spec ( check component spec )
Run the deployment:
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli -v component run --attached dcae-pm-mapper --dmaap-file dmaap.json
Running the Component.....
DCAE.Discovery | INFO | * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
DCAE.Discovery | INFO | * If you run a 'component reconfig' command, you must first execute the following
DCAE.Discovery | INFO | * export SERVICE_NAME=root.e677f7f8-ac75-4919-bdf7-6b794a8f2cc8.1-0-0.dcae-pm-mapper
DCAE.Discovery | INFO | * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
DCAE.Docker | INFO | Running image 'nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server:1.1.1' as 'root.ad8ef3fb-5aca-4095-ad48-44358bbdd082.1-1-0.dcae-pm-mapper'
Above command is run as attach mode, press ctrl+c to stop the micro service but keep it running while you verify the docker container is running in the next step before stopping the dcae_cli command.
A docker component can be run in either attached
or unattached
mode. (Default is unattached).
Mode | Description |
---|---|
attached | component is run in the ‘foreground’, container logs are streamed to stdout. Ctrl-C is used to terminate the dcae_cli session. |
unattached | component is run in the ‘background’, container logs are viewed via docker logs command, container runs until undeployed with dcae_cli undeploy command. |
Check micro service on the Docker host:
In the example component spec, datafile image is provided. Therefore it is deployed.
Goto the terminal thats previously logged into the ubuntu VM.
root@excl-xxxxxx-lego-xxx:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b7f6a219fe11 nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server:1.1.1 "java -jar /target..." 17 seconds ago Up 16 seconds 0.0.0.0:32771->8100/tcp, 0.0.0.0:32770->8433/tcp root.2d086686-b4ac-41c0-90ba-4fc5237efd71.1-1-0.dcae-pm-mapper-new