Useing the dcae-cli Tool to deploy a micro service through a component spec
Overview
The dcae-cli
is a Python command-line tool for component developers. With it, the developer can :
- validate the data formats and component specifications
- publish the validated data formats and component specifications into the
onboarding catalog
- access the
onboarding catalog
to search for existing data formats (for possible reuse) and component specs - deploy a component onto a local or remote DCAE platform for functional and pair-wise testing (This is done without Cloudify)
The git repository for the dcae_cli tool can be found here
For official info:
- https://onap.readthedocs.io/en/latest/submodules/dcaegen2.git/docs/sections/components/dcae-cli/dcae-cli-tool.html
- https://wiki.onap.org/display/DW/MicroServices+Onboarding
Requirements:
- ONAP deployment (3 node deployment is sufficient)
- Another single Ubuntu instance (which we will install docker on)
Follow the below instructions step by step to install , validate component spec and deploy.
- Execute the commands as a root user to avoid permission issues.
Clone the repo onto one of the ONAP instanaces
Login to an instance on the onap deployment and switch to a root user.
root@excl-xxxxxx-lego-casablanca-node-3:~# git clone https://gerrit.onap.org/r/p/dcaegen2/platform/cli.git
Cloning into 'cli'...
remote: Counting objects: 2, done
remote: Finding sources: 100% (2/2)
remote: Total 507 (delta 0), reused 507 (delta 0)
Receiving objects: 100% (507/507), 384.30 KiB | 0 bytes/s, done.
Resolving deltas: 100% (179/179), done.
Checking connectivity... done.
Install required packages: Python, Pip, Virtualenv
Install python, pip (9.0.1 or higher), and virtualenv if they are not installed. Do these when not in a VPN to avoid possible network issues.
$ sudo apt-get -f install python
$ sudo apt-get -f install python-pip
$ sudo pip install virtualenv
Install dcae_cli
$ pip install onap-dcae-cli
Check dcae_cli version
You can verify the version of the dcae-cli with the following command. To get the latest version of the dcae_cli tool,
$ dcae_cli --version
Upgrade dcae_cli
Periodically, upgrade the dcae_cli to get the latest version
pip install --upgrade onap-dcae-cli
Setup postgres database
Create a new postgres db.
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# mkdir postgres
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# docker run --name mypostgres -e POSTGRES_PASSWORD=dcae -e PGDATA=/var/lib/postgresql/data/pgdata -v /home/ubuntu/postgres/:/var/lib/postgresql/data/pgdata -p 5432:5432 -d postgres:9.5.2
Unable to find image 'postgres:9.5.2' locally
9.5.2: Pulling from library/postgres
8b87079b7a06: Pull complete
a3ed95caeb02: Pull complete
ff6abb23e531: Pull complete
8364ca902ad3: Pull complete
84179c1b7ff6: Pull complete
be951654637c: Pull complete
4841dfc8333f: Pull complete
8e92fd62d485: Pull complete
13e5de4be2f2: Pull complete
d6aaf4d83b1c: Pull complete
3113f93aec6d: Pull complete
055e85b433f4: Pull complete
Digest: sha256:923591dfc3bf2e6e2ea35fee759e050340dcea580b78835cbf590e7209f0a4b1
Status: Downloaded newer image for postgres:9.5.2
...
root@excl-xxxxxx-lego-casablanca-node-3:~# docker ps | grep mypostgres
ab28ca5bcc52 postgres:9.5.2 "/docker-entrypoin..." 15 seconds ago Up 15 seconds 0.0.0.0:5432->5432/tcp mypostgres
Install psql
root@excl-xxxxxx-lego-casablanca-node-3:~# apt-get install postgresql-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libpq5 postgresql-client-9.5
Suggested packages:
postgresql-9.5 postgresql-doc-9.5
The following NEW packages will be installed:
libpq5 postgresql-client postgresql-client-9.5
0 upgraded, 3 newly installed, 0 to remove and 30 not upgraded.
Need to get 957 kB of archives.
After this operation, 3,512 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libpq5 amd64 9.5.14-0ubuntu0.16.04 [78.8 kB]
Get:2 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-client-9.5 amd64 9.5.14-0ubuntu0.16.04 [872 kB]
Get:3 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-client all 9.5+173ubuntu0.2 [5,468 B]
Fetched 957 kB in 1s (806 kB/s)
Selecting previously unselected package libpq5:amd64.
(Reading database ... 88946 files and directories currently installed.)
Preparing to unpack .../libpq5_9.5.14-0ubuntu0.16.04_amd64.deb ...
Unpacking libpq5:amd64 (9.5.14-0ubuntu0.16.04) ...
Selecting previously unselected package postgresql-client-9.5.
Preparing to unpack .../postgresql-client-9.5_9.5.14-0ubuntu0.16.04_amd64.deb ...
Unpacking postgresql-client-9.5 (9.5.14-0ubuntu0.16.04) ...
Selecting previously unselected package postgresql-client.
Preparing to unpack .../postgresql-client_9.5+173ubuntu0.2_all.deb ...
Unpacking postgresql-client (9.5+173ubuntu0.2) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Setting up libpq5:amd64 (9.5.14-0ubuntu0.16.04) ...
Setting up postgresql-client-9.5 (9.5.14-0ubuntu0.16.04) ...
update-alternatives: using /usr/share/postgresql/9.5/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
Setting up postgresql-client (9.5+173ubuntu0.2) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Create required database in postgress
root@excl-xxxxxx-lego-casablanca-node-3:~# docker exec -it mypostgres psql -U postgres
psql (9.5.2)
Type "help" for help.
postgres=# CREATE DATABASE dcae_onboarding_db;
postgres-# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
--------------------+----------+----------+------------+------------+-----------------------
dcae_onboarding_db | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres-# \c dcae_onboarding_db
You are now connected to database "dcae_onboarding_db" as user "postgres".
Now exit from postgres database by pressing ctrl +d
Configure dcae_cli tool for initial configuration
Set up a virtual environment and activate
root@excl-xxxxxx-lego-casablanca-node-3:~/cli# virtualenv cli_tool
New python executable in /home/ubuntu/cli/cli_tool/bin/python
Installing setuptools, pip, wheel...
done.
root@excl-xxxxxx-lego-casablanca-node-3:~/cli# source cli_tool/bin/activate
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/cli#
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# dcae_cli --reinit
Warning! Reinitializing your dcae-cli configuration
Please enter the remote server url: https://git.onap.org/dcaegen2/platform/cli/plain (Taken from https://git.onap.org/dcaegen2/platform/cli/plain/dcae-cli/README.md)
Could not download initial configuration from remote server. Attempt manually setting up? [y/N]: y
Please enter your user id: root
Now we need to set up access to the onboarding catalog
Please enter the onboarding catalog hostname: localhost
Please enter the onboarding catalog user: postgres
Please enter the onboarding catalog password: dcae
Could not download initial profiles from remote server. Set empty default? [y/N]: y
Reinitialize done
Now you can list catalog if all the configuration is done.
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# dcae_cli catalog list
Components:
+------+---------+------+-------------+-------+--------+-----------+
| Name | Version | Type | Description | Owner | Status | Published |
+------+---------+------+-------------+-------+--------+-----------+
| | | | | | | |
+------+---------+------+-------------+-------+--------+-----------+
Data formats:
+------+---------+-------------+-------+--------+-----------+
| Name | Version | Description | Owner | Status | Published |
+------+---------+-------------+-------+--------+-----------+
| | | | | | |
+------+---------+-------------+-------+--------+-----------+
Set up profile on dcae_cli tool
List all the profiles:
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# dcae_cli profiles list
* default
Show configuration of default profile:
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~# dcae_cli profiles show default
{
"cdap_broker": "cdap_broker",
"config_binding_service": "config_binding_service",
"consul_host": "",
"docker_host": ""
}
Keep this terminal open, we will go back to it.
Here we need two input configurations,
consul_host ---→ Needed running ONAP
docker_host ---→ Create a new ubuntu VM and install docker
For the consul_host:
We need to expose the consul as a service, here are the steps:
Open a new terminal to the ONAP instance.
root@excl-xxxxxx-lego-casablanca-node-1:~# kubectl -n onap get svc | grep consul
consul ExternalName <none> consul-server.onap.svc.cluster.local <none> 1d
consul-server ClusterIP None <none> 8301/TCP 1d
consul-server-ui NodePort 10.43.24.247 <none> 8500:30270/TCP 1d
msb-consul NodePort 10.43.148.68 <none> 8500:30285/TCP 1d
root@excl-xxxxxx-lego-casablanca-node-1:~# kubectl -n onap expose svc consul-server-ui --type=LoadBalancer --name=my-consul
service "my-consul" exposed
root@excl-xxxxxx-lego-casablanca-node-1:~# kubectl -n onap get svc | grep consul
consul ExternalName <none> consul-server.onap.svc.cluster.local <none> 1d
consul-server ClusterIP None <none> 8301/TCP 1d
consul-server-ui NodePort 10.43.24.247 <none> 8500:30270/TCP 1d
msb-consul NodePort 10.43.148.68 <none> 8500:30285/TCP 1d
my-consul LoadBalancer 10.43.230.247 10.209.63.6 8500:32655/TCP 5s
Now we got consul host, <consul_host=10.209.63.6 >
We also need to expose the Config-Binding service
root@excl-xxxxxx-lego-casablanca-node-1:~# kubectl -n onap expose svc config-binding-service --type=LoadBalancer --name=mycbs
service "mycbs" exposed
root@excl-xxxxxx-lego-casablanca-node-1:~# kubectl -n onap get svc -o wide | grep config
config-binding-service ClusterIP 10.43.184.13 <none> 10000/TCP 6d app=config-binding-service
mycbs LoadBalancer 10.43.178.39 10.209.63.17 10000:31464/TCP 3d app=config-binding-service
Take a note of the exposed Config Binding Service ip-address above (10.209.63.17) and keep this terminal open.
For the Docker Host
Login to the ubuntu VM, install docker and update it's configuration to accept tcp requests.here are the steps.
Login to the ubuntu instance.
Switch to a root user
install docker:
sudo curl https://releases.rancher.com/install-docker/17.03.sh | sh
sudo touch /etc/docker/daemon.json
sudo chmod 777 /etc/docker/daemon.json
sudo bash -c "cat > /etc/docker/daemon.json <<EOF
{
\"log-driver\": \"json-file\",
\"log-opts\": {
\"max-size\": \"20m\",
\"max-file\": \"3\"
}
}
EOF"
$ sudo service docker restart
Now docker is installed but required external tcp port is not enabled.
root@excl-xxxxxx-lego-xxx:/etc/default# sudo netstat -ntlp | grep LISTEN
sudo: unable to resolve host excl-xxxxxx-lego-xxx
tcp 0 0 127.0.0.1:8879 0.0.0.0:* LISTEN 12830/helm
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 16944/sshd
tcp6 0 0 :::22 :::* LISTEN 16944/sshd
Enable tcp port for external request
Follow the below commands:
root@excl-xxxxxx-lego-xxx:/etc/systemd/system# cd /etc/systemd/system
root@excl-xxxxxx-lego-xxx:/etc/systemd/system# mkdir docker.service.d
root@excl-xxxxxx-lego-xxx:/etc/systemd/system# cd docker.service.d/
Create a new file statup_options.conf
root@excl-xxxxxx-lego-xxx:/etc/systemd/system/docker.service.d# vi startup_options.conf
# /etc/systemd/system/docker.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376
root@excl-xxxxxx-lego-xxx:/etc/systemd/system/docker.service.d# sudo systemctl daemon-reload
root@excl-xxxxxx-lego-xxx:/etc/systemd/system/docker.service.d# systemctl restart docker.service
Check that the port is enabled:
root@excl-xxxxxx-lego-xxx:/etc/systemd/system/docker.service.d# sudo netstat -ntlp | grep LISTEN
sudo: unable to resolve host excl-xxxxxx-lego-xxx
tcp 0 0 127.0.0.1:8879 0.0.0.0:* LISTEN 12830/helm
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 16944/sshd
tcp6 0 0 :::2376 :::* LISTEN 15402/dockerd
tcp6 0 0 :::22 :::* LISTEN 16944/sshd
root@excl-xxxxxx-lego-xxx:~# ps -eaf | grep docker
root 15402 1 2 14:13 ? 00:00:51 /usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2376
root 15413 15402 0 14:13 ? 00:00:01 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 17570 15259 0 14:45 pts/0 00:00:00 grep --color=auto docker
root@excl-xxxxxx-lego-xxx:~# docker ps
If you get the following error, you will need to define the DOCKER_HOST.
Cannot connect to the Docker daemon at tcp://localhost:4243. Is the docker daemon running?
root@excl-xxxxxx-lego-xxx:~# DOCKER_HOST=tcp://localhost:2376
root@excl-xxxxxx-lego-xxx:~# docker ( now docker command will work )
To get the host IP:
root@excl-xxxxxx-lego-xxx:~# echo $(ip route get 8.8.8.8 | awk '/8.8.8.8/ {print $NF}')
10.209.63.33
This Ubuntu Virtual Machine external IP is our docker host. So in this case, docker_host=10.209.63.33 ( VM host ip )
Finally login to nexus.
$ docker login -u docker -p docker nexus3.onap.org:10001
Update the dcae_cli default profile
Now , we need to configure the profile with correct consul_host and docker_host. ( or if you execute the reinit command)
So go back to your terminal on the the ONAP Deployment where you were previously connected to the dcae_cli
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli profiles set default consul_host 10.209.63.6
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli profiles set default docker_host 10.209.63.33:2376
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli profiles show default
{
"cdap_broker": "cdap_broker",
"config_binding_service": "config_binding_service",
"consul_host": "10.209.63.6",
"docker_host": "10.209.63.33:2376"
}
Add data format
To validate data format:
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# vi VES_dataformat.json (Insert the above into the file)
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli data_format add VES_dataformat.json
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli data_format list
Data formats for xxxx
+-------------------+---------+---------------------+--------+----------------------------+
| Name | Version | Description | Status | Modified |
+-------------------+---------+---------------------+--------+----------------------------+
| VES_specification | 7.30.1 | VES spec for v7.0.1 | staged | 2019-01-08 11:59:52.638336 |
+-------------------+---------+---------------------+--------+----------------------------+
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli data_format publish VES_specification
Data format has been published
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli data_format list
Data formats for xxxx
+-------------------+---------+---------------------+-----------+----------------------------+
| Name | Version | Description | Status | Modified |
+-------------------+---------+---------------------+-----------+----------------------------+
| VES_specification | 7.30.1 | VES spec for v7.0.1 | published | 2019-01-08 12:01:20.791865 |
+-------------------+---------+---------------------+-----------+----------------------------+
Add component spec
Validating the component spec
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# vi pmmapper-component-spec.json (Insert the above into the file)
if you wish you can change the docker image change the line
"uri": "nexus3.onap.org:10001/onap/dmaap/datarouter-node:1.0.3", to another docker image i.e nexus3.onap.org:10001/onap/XXXX:1.0.1
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli component add pmmapper-component-spec.json
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli component list
Active profile: default
+----------------+---------+--------+------------------------------+--------+----------------------------+-----------+
| Name | Version | Type | Description | Status | Modified | #Deployed |
+----------------+---------+--------+------------------------------+--------+----------------------------+-----------+
| dcae-pm-mapper | 1.0.0 | docker | PM Mapper Docker Application | staged | 2019-01-08 12:02:16.468432 | 0 |
+----------------+---------+--------+------------------------------+--------+----------------------------+-----------+
Use the "--deployed" option to see more details on deployments
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli component publish dcae-pm-mapper
Component has been published
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli component list
Active profile: default
+----------------+---------+--------+------------------------------+-----------+----------------------------+-----------+
| Name | Version | Type | Description | Status | Modified | #Deployed |
+----------------+---------+--------+------------------------------+-----------+----------------------------+-----------+
| dcae-pm-mapper | 1.0.0 | docker | PM Mapper Docker Application | published | 2019-01-10 16:58:10.029253 | 0 |
+----------------+---------+--------+------------------------------+-----------+----------------------------+-----------+
deploy component
Deploying the micro-service: Only published specs can be deployed. Check existing catalog with published components.
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli catalog list
Components:
+----------------+---------+--------+------------------------------+----------+-----------+------------+
| Name | Version | Type | Description | Owner | Status | Published |
+----------------+---------+--------+------------------------------+----------+-----------+------------+
| dcae-pm-mapper | 1.0.0 | docker | PM Mapper Docker Application | root | published | 2019-01-08 |
+----------------+---------+--------+------------------------------+----------+-----------+------------+
Data formats:
+-------------------+---------+---------------------+----------+-----------+------------+
| Name | Version | Description | Owner | Status | Published |
+-------------------+---------+---------------------+----------+-----------+------------+
| VES_specification | 7.30.1 | VES spec for v7.0.1 | root | published | 2019-01-08 |
+-------------------+---------+---------------------+----------+-----------+------------+
Create a dmaap-file ( dmaap.json ) with below content for deployment:
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# vi dmaap.json
{
"pm_mapper_handle_out":
{
"type": "message_router",
"dmaap_info": {
"topic_url": "https://we-are-message-router.us:3905/events/some-topic"
}
}
}
Here "pm_mapper_handle_out" → This is a config key in component spec ( check component spec )
Run the deployment:
(cli_tool) root@excl-xxxxxx-lego-casablanca-node-3:~/pmmapper# dcae_cli -v component run --attached dcae-pm-mapper --dmaap-file dmaap.json
Running the Component.....
DCAE.Discovery | INFO | * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
DCAE.Discovery | INFO | * If you run a 'component reconfig' command, you must first execute the following
DCAE.Discovery | INFO | * export SERVICE_NAME=root.e677f7f8-ac75-4919-bdf7-6b794a8f2cc8.1-0-0.dcae-pm-mapper
DCAE.Discovery | INFO | * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
DCAE.Docker | INFO | Running image 'nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server:1.1.1' as 'root.ad8ef3fb-5aca-4095-ad48-44358bbdd082.1-1-0.dcae-pm-mapper'
Above command is run as attach mode, press ctrl+c to stop the micro service but keep it running while you verify the docker container is running in the next step before stopping the dcae_cli command.
A docker component can be run in either attached
or unattached
mode. (Default is unattached).
Mode | Description |
---|---|
attached | component is run in the ‘foreground’, container logs are streamed to stdout. Ctrl-C is used to terminate the dcae_cli session. |
unattached | component is run in the ‘background’, container logs are viewed via docker logs command, container runs until undeployed with dcae_cli undeploy command. |
Go back to the previously opened terminal on the ONAP deployment and verify that the service is executing ( the output will be similar to this) by executing the command
# curl http://<IP of the exposed Config Binding Service>:10000/service_component/<SERVICE_NAME>
i.e.
root@excl-xxxxxx-lego-casablanca-node-1:~# curl http://10.209.63.17:10000/service_component/root.e677f7f8-ac75-4919-bdf7-6b794a8f2cc8.1-0-0.dcae-pm-mapper
{"pm-mapper-filter": "{ \"filters\":[]}", "3GPP.schema.file": "{\"3GPP_Schema\":\"./etc/3GPP_relaxed_schema.xsd\"}", "streams_publish.pm_mapper_handle_out.message_router_topic": "unauthenticated.PM_VES_OUTPUT", "streams_subscribes": {}, "streams_publishes": {"pm_mapper_handle_out": {"type": "message_router", "aaf_password": null, "dmaap_info": {"topic_url": "https://we-are-message-router.us:3905/events/some-topic", "client_role": null, "location": null, "client_id": null}, "aaf_username": null}}, "streams_subscribes.pm_mapper_handle_in.feed_id": "1", "services_calls": {}}ubuntu@masternightly-node-2:
Check micro service on the Docker host:
In the example component spec the datafile image is provided therefore it is deployed.
Goto the terminal thats previously logged into the ubuntu VM.
root@excl-xxxxxx-lego-xxx:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b7f6a219fe11 nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server:1.1.1 "java -jar /target..." 17 seconds ago Up 16 seconds 0.0.0.0:32771->8100/tcp, 0.0.0.0:32770->8433/tcp root.2d086686-b4ac-41c0-90ba-4fc5237efd71.1-1-0.dcae-pm-mapper-new