| Table of Contents |
|---|
Introduction
The 72 hour Stability Test for policy distribution has the goal of introducing a steady flow of transactions initiated from a test client server running JMeter. The policy distribution is configured with a special FileSystemReception plugin to monitor a local directory for newly added csar files to be processed by itself. The input csar files will be generated by test client (JMeter) and the result will be pulled from backend PDP by test client(JMeter).
The test will be performed in an environment where JMeter will continuously generate random csar files into the special directory where policy distribution monitors and get the processed results from backend PDP by restful calls(10 retries with 1 seconds between each retry) for the duration of 72 hours.
Setup details
The stability test is performed on VM's running in the OpenStack cloud environment in the ONAP integration lab. There are 2 separate VMs, one for running backend policy services which policy distribution needs, and the other is for policy distribution service itself and jmeter.
...
Java: openjdk version "1.8.0_181"
JMeter: 5.0
Basic installation on both VM1 & VM2
Install java, docker, docker-compose and other utilities on both VMs.
| Code Block |
|---|
$ sudo apt-get update $ sudo apt-get install -y openjdk-8-jdk docker docker-compose zip unzip curl wget ssh telnet maven # make sure docker can be launched with normal user $ sudo groupadd docker $ sudo gpasswd -a $USER docker $ newgrp docker |
Install backend policy services needed by policy-distribution (VM1)
Get policy/docker code:
| Code Block |
|---|
$ git clone https://git.onap.org/policy/docker policy-docker $ cd policy-docker/ $ chmod +x config/drools/drools-tweaks.sh $ IP_ADDRESS=$(ifconfig ens3 | grep "inet addr" | tr -s ' ' | cut -d' ' -f3 | cut -d':' -f2) $ echo $IP_ADDRESS > config/pe/ip_addr.txt |
...
| Code Block |
|---|
$ docker login -u docker -p docker nexus3.onap.org:10001 $ docker iamgeimage pull mariadb:10.2.14 $ docker image pull sonatype/nexus:2.14.8-01 $ docker image pull nexus3.onap.org:10001/onap/policy-pe $ docker image tag nexus3.onap.org:10001/onap/policy-pe onap/policy-pe $ docker image pull nexus3.onap.org:10001/onap/policy-drools $ docker image tag nexus3.onap.org:10001/onap/policy-drools onap/policy-drools $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE onap/policy-pe latest 8d3b9cec2e48 35 hours ago 1.6 GB nexus3.onap.org:10001/onap/policy-pe latest 8d3b9cec2e48 35 hours ago 1.6 GB nexus3.onap.org:10001/onap/policy-drools latest aba8c4afcd0c 46 hours ago 1.11 GB onap/policy-drools latest aba8c4afcd0c 46 hours ago 1.11 GB mariadb 10.2.14 2cbeee8bd9cf 6 months ago 399 MB sonatype/nexus 2.14.8-01 ee28e4c8635c 8 months ago 468 MB |
...
| Code Block |
|---|
$ export MTU=1500 $ docker-compose up &-d #wait for severalabout 10 minutes until all the policy backend services needed by policy-distribution are all up $ docker-compose pslogs CONTAINERpap ID | IMAGE* upload completely sent off: 96 out of 96 bytes pap | Transaction ID: COMMAND 78970237-42b0-45d5-9a87-ef9a16e0780f --Policy 'com.Config_OOF_PCI_CONFIG_POLICY.1.xml' was successfully pushed to the PDP group 'default'.< HTTP/1.1 200 pap | < Content-Type: text/plain;charset=ISO-8859-1 pap | < Content-Length: 154 pap | < Date: Thu, 15 Nov 2018 08:27:38 GMT pap | < pap | { [data not shown] pap | * Connection #0 to host pdp left intact $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b3db20fc3d94814e448d784e onap/policy-pe "bash ./do-start.s..." 2117 hoursminutes ago Up 2117 hours minutes brmsgw 90f95d1bf2ddabb3c28231ff onap/policy-pe "bash ./do-start.s..." 2117 hoursminutes ago Up 2117 hours minutes 0.0.0.0:8081->8081/tcp pdp f21fb4237d3ce1946ca9816a onap/policy-drools "/bin/sh -c ./do-s..." 2117 hoursminutes ago Up 2117 hoursminutes 0.0.0.0:6969->6969/tcp, 0.0.0.0:9696->9696/tcp drools e00181a48a2ea7ac8c3492c0 onap/policy-pe "bash ./do-start.s..." 2117 hoursminutes ago Up 2117 hoursminutes 0.0.0.0:8443->8443/tcp, 0.0.0.0:9091->9091/tcp pap f3149b6e7397c123b56cfa57 mariadbsonatype/nexus:10.2.14.8-01 "docker-entrypoint"/bin/sh -c '${JAV..." 2117 hoursminutes ago Up 2117 hours minutes 0.0.0.0:33069081->3306>8081/tcp mariadbnexus f13bbe072186cccf586f0ab9 sonatype/nexusmariadb:10.2.14.8-01 "/bin/sh -c '${JAVdocker-entrypoint..." 2117 hoursminutes ago Up 21 hoursUp 17 minutes 0.0.0.0:90813306->8081>3306/tcp nexusmariadb |
Special settings on policy-distribution/Jmeter VM (VM2)
Add ip address of VM1 /etc/hosts entries NOTE: please replace the correct IP address of VM1 in your environment
...
| Code Block |
|---|
$ curl -s https://git.onap.org/dmaap/buscontroller/plain/misc/cert-client-init.sh | sudo sh |
Install JMeter (VM2)
Download & install JMeter
| Code Block |
|---|
$ wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-5.0.zip $ unzip apache-jmeter-5.0.zip |
Build and install policy-distribution in VM2
Here we'll build policy distribution docker by ourselves, or you can get the docker image directly from onap nexus3 repo and skip the next step:
...
| Code Block |
|---|
$ docker kill policy-distribution $ docker rm policy-distribution |
Modify the policy-distribution configuration for S3P test in VM2
Turn off debug output of policy-distribution
...
| Code Block |
|---|
# add the following JVM options when launch policy-distribution where 10.12.6.201 is the IP address of VM2 # -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=16001 -Djava.rmi.server.hostname=10.12.6.201 # change the last line of the policy-dist.sh as the following: # $JAVA_HOME/bin/java -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=16001 -Djava.rmi.server.hostname=10.12.6.201 -cp "$POLICY_DISTRIBUTION_HOME/etc:$POLICY_DISTRIBUTION_HOME/lib/*" org.onap.policy.distribution.main.startstop.Main -c $CONFIG_FILE $ vim /opt/app/policy/distribution/bin/policy-dist.sh |
Launch policy-distribution service (VM2)
| Code Block |
|---|
# create the default directory which policy distribution service will monitor
$ mkdir -p /tmp/policy_distribution/csar
# launch policy distribution service
$ cd /opt/app/policy/distribution/bin
$ ./policy-dist.sh /opt/app/policy/distribution/etc/s3pConfig.json & |
Install & Configure VisualVM (VM2)
VisualVM needs to be installed in the virtual machine having policy-distribution. It will be used to monitor CPU, Memory, GC for policy-distribution while stability test is running.
...
Sample Screenshot of visualVM
Test Plan
The 72 hours stability test will run the following steps in 20 threaded a single thread loop.
- Delete existing policy if it matches the policy we're about to generate - send DELETE restful API to pdp and pap
- Generate the csar file and move it into the directory where being monitored.
- Get the policy from pdp matching the generated csar - sent restful API to pdp to get the matching policy, retry if failed, until it succeed or retry count exceed.
...
| Code Block |
|---|
$ cd /home/ubuntu/policy-distribution/s3p $ rm stability.log; ~/apache-jmeter-5.0/bin/jmeter.sh -n -t stability.jmx -Jhost=pdp -dretryJretry=20200 -Jduration=259200 -l stability.log # get the total number csar files generated: $ grep "Generate_CSAR" stability.log | wc -l 1057 # get the total number of failed policy to be stored in the pdp: $ grep "get policy failed" stability.log | wc -l 950 |
...