Introduction
The 72 hour Stability Test for policy distribution has the goal of introducing a steady flow of transactions initiated from a test client server running JMeter. The policy distribution is configured with a special FileSystemReception plugin to monitor a local directory for newly added csar files to be processed by itself. The input csar files will be generated by test client (JMeter) and the result will be pulled from backend PDP by test client(JMeter).
The test will be performed in an environment where JMeter will continuously generate random csar files into the special directory where policy distribution monitors and get the processed results from backend PDP by restful calls(10 retries with 1 seconds between each retry) for the duration of 72 hours.
Setup details
The stability test is performed on VM's running in the OpenStack cloud environment in the ONAP integration lab. There are 2 separate VMs, one for running backend policy services which policy distribution needs, and the other is for policy distribution service itself and jmeter.
OpenStack environment details
Version: Windriver Titanium
policy backend VM details (VM1)
OS:Ubuntu 16.04.4 LTS
CPU: 8 core
RAM: 32 GB
HardDisk: 160 GB
Docker Version: 17.03.2-ce, build f5ec1e2
Java: openjdk version "1.8.0_181"
JMeter VM details (VM2)
OS: Ubuntu 16.04.4LTS
CPU: 8 core
RAM: 32 GB
HardDisk: 160 GB
Docker Version: 17.03.2-ce, build f5ec1e2
Java: openjdk version "1.8.0_181"
JMeter: 5.0
Basic installation on both VM1 & VM2
Install java, docker, docker-compose and other utilities on both VMs.
$ sudo apt-get update $ sudo apt-get install -y openjdk-8-jdk docker docker-compose zip unzip curl wget ssh telnet maven # make sure docker can be launched with normal user $ sudo groupadd docker $ sudo gpasswd -a $USER docker $ newgrp docker
Install backend policy services needed by policy-distribution (VM1)
Get policy/docker code:
$ git clone https://git.onap.org/policy/docker policy-docker $ cd policy-docker/ $ chmod +x config/drools/drools-tweaks.sh $ IP_ADDRESS=$(ifconfig ens3 | grep "inet addr" | tr -s ' ' | cut -d' ' -f3 | cut -d':' -f2) $ echo $IP_ADDRESS > config/pe/ip_addr.txt
Pull docker images:
$ docker login -u docker -p docker nexus3.onap.org:10001 $ docker iamge pull mariadb:10.2.14 $ docker image pull sonatype/nexus:2.14.8-01 $ docker image pull nexus3.onap.org:10001/onap/policy-pe $ docker image tag nexus3.onap.org:10001/onap/policy-pe onap/policy-pe $ docker image pull nexus3.onap.org:10001/onap/policy-drools $ docker image tag nexus3.onap.org:10001/onap/policy-drools onap/policy-drools $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE onap/policy-pe latest 8d3b9cec2e48 35 hours ago 1.6 GB nexus3.onap.org:10001/onap/policy-pe latest 8d3b9cec2e48 35 hours ago 1.6 GB nexus3.onap.org:10001/onap/policy-drools latest aba8c4afcd0c 46 hours ago 1.11 GB onap/policy-drools latest aba8c4afcd0c 46 hours ago 1.11 GB mariadb 10.2.14 2cbeee8bd9cf 6 months ago 399 MB sonatype/nexus 2.14.8-01 ee28e4c8635c 8 months ago 468 MB
Build docker and launch various policy services:
$ export MTU=1500
$ docker-compose up -d
#wait for about 10 minutes until all the policy backend services needed by policy-distribution are all up
$ docker-compose logs
pap | * upload completely sent off: 96 out of 96 bytes
pap | Transaction ID: 78970237-42b0-45d5-9a87-ef9a16e0780f --Policy 'com.Config_OOF_PCI_CONFIG_POLICY.1.xml' was successfully pushed to the PDP group 'default'.< HTTP/1.1 200
pap | < Content-Type: text/plain;charset=ISO-8859-1
pap | < Content-Length: 154
pap | < Date: Thu, 15 Nov 2018 08:27:38 GMT
pap | <
pap | { [data not shown]
pap | * Connection #0 to host pdp left intact
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b3db20fc3d94 onap/policy-pe "bash ./do-start.s..." 21 hours ago Up 21 hours brmsgw
90f95d1bf2dd onap/policy-pe "bash ./do-start.s..." 21 hours ago Up 21 hours 0.0.0.0:8081->8081/tcp pdp
f21fb4237d3c onap/policy-drools "/bin/sh -c ./do-s..." 21 hours ago Up 21 hours 0.0.0.0:6969->6969/tcp, 0.0.0.0:9696->9696/tcp drools
e00181a48a2e onap/policy-pe "bash ./do-start.s..." 21 hours ago Up 21 hours 0.0.0.0:8443->8443/tcp, 0.0.0.0:9091->9091/tcp pap
f3149b6e7397 mariadb:10.2.14 "docker-entrypoint..." 21 hours ago Up 21 hours 0.0.0.0:3306->3306/tcp mariadb
f13bbe072186 sonatype/nexus:2.14.8-01 "/bin/sh -c '${JAV..." 21 hours ago Up 21 hours 0.0.0.0:9081->8081/tcp nexus
Special settings on policy-distribution/Jmeter VM (VM2)
Add ip address of VM1 /etc/hosts entries NOTE: please replace the correct IP address of VM1 in your environment
# Added the IP address of VM1 into /etc/hosts on VM2 $ echo 10.12.6.222 pdp | sudo tee -a /etc/hosts # get vm2 IP address $ export VM2IP=$(hostname -I | cut -d\ -f1) $ echo $VM2IP 10.12.6.201
Install root CA
$ curl -s https://git.onap.org/dmaap/buscontroller/plain/misc/cert-client-init.sh | sudo sh
Install JMeter (VM2)
Download & install JMeter
$ wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-5.0.zip $ unzip apache-jmeter-5.0.zip
Build and install policy-distribution in VM2
Here we'll build policy distribution docker by ourselves, or you can get the docker image directly from onap nexus3 repo and skip the next step:
# get onap maven settings: $ mkdir ~/.m2 $ cd ~/.m2/ $ wget https://git.onap.org/oparent/plain/settings.xml $ cd ~/ $ git clone https://git.onap.org/policy/distribution policy-distribution $ cd policy-distribution/ # build policy-distribution images $ mvn install -DskipTests=True -Pdocker $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE onap/policy-distribution 2.1.0-SNAPSHOT 564d652332e0 19 hours ago 1.1 GB onap/policy-distribution 2.1.0-SNAPSHOT-20181114T0952 564d652332e0 19 hours ago 1.1 GB onap/policy-distribution 2.1.0-SNAPSHOT-latest 564d652332e0 19 hours ago 1.1 GB onap/policy-distribution latest 564d652332e0 19 hours ago 1.1 GB nexus3.onap.org:10001/ubuntu 14.04 f216cfb59484 3 weeks ago 188 MB ubuntu 14.04 f216cfb59484 3 weeks ago 188 MB
Here we'll copy the policy-distribution binaries out of the docker container to run them in the VM directly, so we can made some special tweak for S3P test purpose:
$ sudo mkdir -p /opt/app $ sudo chown ubuntu:ubuntu /opt/app $ docker run -d -v /opt/app/:/tmp/abc --name policy-distribution -p 6969:6969 -it onap/policy-distribution # go into docker $ docker exec -it policy-distribution /bin/bash
In the docker, copy the binaries out and exit docker
$ cd /opt/app $ cp -r policy /tmp/abc/ $ exit
Stop the docker
$ docker kill policy-distribution $ docker rm policy-distribution
Modify the policy-distribution configuration for S3P test in VM2
Turn off debug output of policy-distribution
# open file /opt/app/policy/distribution/etc/logback.xml, search for the following:
<logger name="com.att.eelf.debug" level="debug" additivity="false">
<appender-ref ref="asyncEELFDebug" />
</logger>
# and change the level="debug" to level="warn"
$ vim /opt/app/policy/distribution/etc/logback.xml
Open policy-distribution startup script to open remote jmx support for visualVM
# add the following JVM options when launch policy-distribution where 10.12.6.201 is the IP address of VM2 # -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=16001 -Djava.rmi.server.hostname=10.12.6.201 # change the last line of the policy-dist.sh as the following: # $JAVA_HOME/bin/java -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=16001 -Djava.rmi.server.hostname=10.12.6.201 -cp "$POLICY_DISTRIBUTION_HOME/etc:$POLICY_DISTRIBUTION_HOME/lib/*" org.onap.policy.distribution.main.startstop.Main -c $CONFIG_FILE $ vim /opt/app/policy/distribution/bin/policy-dist.sh
Launch policy-distribution service (VM2)
$ cd /opt/app/policy/distribution/bin $ ./policy-dist.sh /opt/app/policy/distribution/etc/s3pConfig.json &
Install & Configure VisualVM (VM2)
VisualVM needs to be installed in the virtual machine having policy-distribution. It will be used to monitor CPU, Memory, GC for policy-distribution while stability test is running.
Install visualVM
$ sudo apt-get install visualvm
Run few commands to configure permissions
$ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/
$ sudo touch visualvm.policy
$ sudo chmod 777 visualvm.policy
$ vi visualvm.policy
Add the following in visualvm.policy
grant codebase "file:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar" {
permission java.security.AllPermission;
};
Run following commands to start jstatd using port 1111
$ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/ $ ./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy &
Run visualVM locally to connect to remote VM2
# On your windows machine or your linux box locally, launch visualVM
Connect to jstatd & remote apex-pdp JVM
- Right click on "Remote" in the left panel of the screen and select "Add Remote Host..."
Enter the IP address of VM2.
- Right click on IP address, select "Add JMX Connection..."
- Enter the VM2 IP Address (from step 2) <IP address>:16001 ( for example -10.12.6.201:9991) and click OK.
- Double click on the newly added nodes under "Remote" to start monitoring CPU, Memory & GC.
Sample Screenshot of visualVM
Test Plan
The 72 hours stability test will run the following steps in 20 threaded loop.
- Delete existing policy if it matches the policy we're about to generate - send DELETE restful API to pdp and pap
- Generate the csar file and move it into the directory where being monitored.
- Get the policy from pdp matching the generated csar - sent restful API to pdp to get the matching policy, retry if failed, until it succeed or retry count exceed.
Please check https://git.onap.org/policy/distribution/tree/s3p/README.stability.md for Jmeter parameter we can use for stability test
Run the following commands to start 72 hours test
$ cd /home/ubuntu/policy-distribution/s3p $ rm stability.log; ~/apache-jmeter-5.0/bin/jmeter.sh -n -t stability.jmx -Jhost=pdp -dretry=20 -Jduration=259200 -l stability.log # get the total number csar files generated: $ grep "Generate_CSAR" stability.log | wc -l 1057 # get the total number of failed policy to be stored in the pdp: $ grep "get policy failed" stability.log | wc -l 950