Table of Contents

Introduction

The 72 hour Stability Test for policy distribution has the goal of introducing a steady flow of transactions initiated from a test client server running JMeter. The policy distribution is configured with a special FileSystemReception plugin to monitor a local directory for newly added csar files to be processed by itself. The input csar files will be generated by test client (JMeter) and the result will be pulled from backend PDP by test client(JMeter).

The test will be performed in an environment where JMeter will continuously generate random csar files into the special directory where policy distribution monitors and get the processed results from backend PDP by restful calls(10 retries with 1 seconds between each retry) for the duration of 72 hours.

Setup details

The stability test is performed on VM's running in the OpenStack cloud environment in the ONAP integration lab. There are 2 separate VMs, one for running backend policy services which policy distribution needs, and the other is for policy distribution service itself and jmeter.

...

Java: openjdk version "1.8.0_181"

JMeter: 5.0

Basic installation on both VM1 & VM2

Install java, docker, docker-compose and other utilities on both VMs.

Code Block

$ sudo apt-get update
$ sudo apt-get install -y openjdk-8-jdk docker docker-compose zip unzip curl wget ssh telnet maven
# make sure docker can be launched with normal user
$ sudo groupadd docker
$ sudo gpasswd -a $USER docker
$ newgrp docker

Install backend policy services needed by policy-distribution (VM1)

Get policy/docker code:

Code Block

$ git clone https://git.onap.org/policy/docker policy-docker
$ cd policy-docker/
$ chmod +x config/drools/drools-tweaks.sh
$ IP_ADDRESS=$(ifconfig ens3 | grep "inet addr" | tr -s ' ' | cut -d' ' -f3 | cut -d':' -f2)
$ echo $IP_ADDRESS > config/pe/ip_addr.txt

...

Code Block

$ docker login -u docker -p docker nexus3.onap.org:10001
$ docker iamgeimage pull mariadb:10.2.14
$ docker image pull sonatype/nexus:2.14.8-01
$ docker image pull nexus3.onap.org:10001/onap/policy-pe
$ docker image tag nexus3.onap.org:10001/onap/policy-pe onap/policy-pe
$ docker image pull nexus3.onap.org:10001/onap/policy-drools
$ docker image tag nexus3.onap.org:10001/onap/policy-drools onap/policy-drools
$ docker images
REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
onap/policy-pe                             latest              8d3b9cec2e48        35 hours ago        1.6 GB
nexus3.onap.org:10001/onap/policy-pe       latest              8d3b9cec2e48        35 hours ago        1.6 GB
nexus3.onap.org:10001/onap/policy-drools   latest              aba8c4afcd0c        46 hours ago        1.11 GB
onap/policy-drools                         latest              aba8c4afcd0c        46 hours ago        1.11 GB
mariadb                                    10.2.14             2cbeee8bd9cf        6 months ago        399 MB
sonatype/nexus                             2.14.8-01           ee28e4c8635c        8 months ago        468 MB

...

Code Block

$ export MTU=1500
$ docker-compose up &-d
#wait for severalabout 10 minutes until all the policy backend services needed by policy-distribution are all up
$ docker-compose pslogs
CONTAINERpap ID       | IMAGE* upload completely sent off: 96 out of 96 bytes
pap        | Transaction ID:  COMMAND            78970237-42b0-45d5-9a87-ef9a16e0780f --Policy 'com.Config_OOF_PCI_CONFIG_POLICY.1.xml' was successfully pushed to the PDP group 'default'.< HTTP/1.1 200
pap      CREATED  | < Content-Type: text/plain;charset=ISO-8859-1
pap       STATUS | < Content-Length: 154
pap        | PORTS< Date: Thu, 15 Nov 2018 08:27:38 GMT
pap        | <
pap        | { [data not shown]
pap        | * Connection #0 to host pdp left intact
NAMES
b3db20fc3d94
$ docker ps
CONTAINER ID   onap/policy-pe     IMAGE        "bash ./do-start.s..."   21 hours ago        UpCOMMAND 21 hours                CREATED             STATUS              PORTS               brmsgw 90f95d1bf2dd        onap/policy-pe             "bash ./do-start.s..."   21 hours ago       NAMES
814e448d784e        onap/policy-pe   Up 21 hours         0.0.0.0:8081->8081/tcp"bash ./do-start.s..."   17 minutes ago      Up 17 minutes              pdp  f21fb4237d3c        onap/policy-drools         "/bin/sh -c ./do-s..."   21 hours ago        Up 21 hours      brmsgw
abb3c28231ff  0.0.0.0:6969->6969/tcp, 0.0.0.0:9696->9696/tcp   drools
e00181a48a2e        onap/policy-pe             "bash ./do-start.s..."   2117 hoursminutes ago        Up 2117 hours  minutes       0.0.0.0:84438081->8443>8081/tcp, 0.0.0.0:9091->9091/tcp   pap f3149b6e7397        mariadb:10.2.14            "docker-entrypoint..."  pdp
21e1946ca9816a hours ago      onap/policy-drools  Up 21 hours     "/bin/sh    0.0.0.0:3306->3306/tcp    -c ./do-s..."   17 minutes ago      Up 17 minutes       0.0.0.0:6969->6969/tcp, 0.0.0.0:9696->9696/tcp   drools
a7ac8c3492c0   mariadb    f13bbe072186 onap/policy-pe       sonatype/nexus:2.14.8-01   "/bin/sh -c '${JAV  "bash ./do-start.s..."   2117 hoursminutes ago        Up 2117 hoursminutes         0.0.0.0:8443->8443/tcp, 0.0.0.0:90819091->8081>9091/tcp   pap
c123b56cfa57        sonatype/nexus:2.14.8-01   "/bin/sh   -c '${JAV..."   17 minutes ago    nexus

Special settings on policy-distribution/Jmeter VM (VM2)

Add ip address of VM1 /etc/hosts entries NOTE: please replace the correct IP address of VM1 in your environment

Code Block

# Added theUp IP17 addressminutes of VM1 into /etc/hosts on VM2 $ echo 10.12.6.222 pdp | sudo tee -a /etc/hosts
# get vm2 IP address
$ export VM2IP=$(hostname -I | cut -d\  -f1)
$ echo $VM2IP
10.12.6.201

Install root CA

Code Block
$ curl -s https://git.onap.org/dmaap/buscontroller/plain/misc/cert-client-init.sh \| sudo sh

Install JMeter (VM2)

Download & install JMeter

Code Block
$ wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-5.0.zip $ unzip apache-jmeter-5.0.zip

Build and install policy-distribution in VM2

Here we'll build policy distribution docker by ourselves, or you can get the docker image directly from onap nexus3 repo and skip the next step:

Code Block

# get onap maven settings:
$ mkdir ~/.m2
$ cd ~/.m2/
$ wget https://git.onap.org/oparent/plain/settings.xml
$ cd ~/
$ git clone https://git.onap.org/policy/distribution policy-distribution
$ cd policy-distribution/

# build policy-distribution images
$ mvn install -DskipTests=True -Pdocker
$ docker images
REPOSITORY                     TAG                            IMAGE ID            CREATED             SIZE
onap/policy-distribution       2.1.0-SNAPSHOT0.0.0.0:9081->8081/tcp                           nexus
cccf586f0ab9        mariadb:10.2.14            "docker-entrypoint..."   17 minutes ago      Up 17 minutes       0.0.0.0:3306->3306/tcp                           mariadb

Special settings on policy-distribution/Jmeter VM (VM2)

Add ip address of VM1 /etc/hosts entries NOTE: please replace the correct IP address of VM1 in your environment

Code Block
# Added the IP address of VM1 into /etc/hosts on VM2 $ echo 10.12.6.222 pdp \| sudo tee -a /etc/hosts # get vm2 IP address $ export VM2IP=$(hostname -I \| cut -d\ -f1) $ echo $VM2IP 10.12.6.201

Install root CA

Code Block
$ curl -s https://git.onap.org/dmaap/buscontroller/plain/misc/cert-client-init.sh \| sudo sh

Install JMeter (VM2)

Download & install JMeter

Code Block
$ wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-5.0.zip $ unzip apache-jmeter-5.0.zip

Build and install policy-distribution in VM2

Here we'll build policy distribution docker by ourselves, or you can get the docker image directly from onap nexus3 repo and skip the next step:

Code Block

# get onap maven settings:
$ mkdir ~/.m2
$ cd ~/.m2/
$ wget https://git.onap.org/oparent/plain/settings.xml
$ cd ~/
$ git clone https://git.onap.org/policy/distribution policy-distribution
$ cd policy-distribution/

# build policy-distribution images
$ mvn install -DskipTests=True -Pdocker
$ docker images
REPOSITORY                     TAG     564d652332e0        19 hours ago        1.1 GB onap/policy-distribution   IMAGE ID   2.1.0-SNAPSHOT-20181114T0952   564d652332e0      CREATED  19 hours ago        1.1 GBSIZE
onap/policy-distribution       2.1.0-SNAPSHOT-latest          564d652332e0       564d652332e0 19        19 hours ago        1.1 GB
onap/policy-distribution       latest2.1.0-SNAPSHOT-20181114T0952                         564d652332e0        19 hours ago        1.1 GB
nexus3.onap.org:10001/ubuntu/policy-distribution   14.04    2.1.0-SNAPSHOT-latest          564d652332e0        19 hours ago  f216cfb59484        3 weeks ago1.1 GB
onap/policy-distribution       latest                188 MB ubuntu       564d652332e0        19 hours ago        1.1 GB
nexus3.onap.org:10001/ubuntu   14.04                          f216cfb59484        3 weeks ago         188 MB

...

ubuntu

...

14.04

...

Code Block

$ sudo mkdir -p /opt/app  $ sudo chown ubuntu:ubuntu /opt/app $ docker run -d -v /opt/app/:/tmp/abc --name policy-distribution -p 6969:6969 -it onap/policy-distribution
# go into docker
$ docker exec -it policy-distribution /bin/bash

In the docker, copy the binaries out and exit docker

Code Block
$ cd /opt/app $ cp -r policy /tmp/abc/ $ exit

Stop the docker

Code Block
$ docker kill policy-distribution $ docker rm policy-distribution

Modify the policy-distribution configuration for S3P test in VM2

Turn off debug output of policy-distribution

Code Block

# open file /opt/app/policy/distribution/etc/logback.xml, search for the following:
 <logger name="com.att.eelf.debug" level="debug" additivity="false">
        <appender-ref ref="asyncEELFDebug" />
  </logger>
# and change the level="debug" to level="warn"
$ vim /opt/app/policy/distribution/etc/logback.xml

Open policy-distribution startup script to open remote jmx support for visualVM

Code Block

# add the following JVM options when launch policy-distribution where 10.12.6.201 is the IP address of VM2
# -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=16001 -Djava.rmi.server.hostname=10.12.6.201
# change the last line of the policy-dist.sh as the following: 
# $JAVA_HOME/bin/java -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=16001 -Djava.rmi.server.hostname=10.12.6.201 -cp "$POLICY_DISTRIBUTION_HOME/etc:$POLICY_DISTRIBUTION_HOME/lib/*" org.onap.policy.distribution.main.startstop.Main -c $CONFIG_FILE


$ vim /opt/app/policy/distribution/bin/policy-dist.sh

Launch policy-distribution service (VM2)

Code Block
$ cd /opt/app/policy/distribution/bin $ ./policy-dist.sh /opt/app/policy/distribution/etc/s3pConfig.json &

Install & Configure VisualVM (VM2)

VisualVM needs to be installed in the virtual machine having policy-distribution. It will be used to monitor CPU, Memory, GC for policy-distribution while stability test is running.

Install visualVM

Code Block
$ sudo apt-get install visualvm

Run few commands to configure permissions

Code Block

$ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/ 
$ sudo touch visualvm.policy 
$ sudo chmod 777 visualvm.policy

$ vi visualvm.policy 

Add the following in visualvm.policy


grant codebase "file:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar" {
   permission java.security.AllPermission;
};

Run following commands to start jstatd using port 1111

Code Block
$ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/ $ ./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy &

...

Code Block
# On your windows machine or your linux box locally, launch visualVM

Connect to jstatd & remote apex-pdp JVM

Right click on "Remote" in the left panel of the screen and select "Add Remote Host..."
Enter the IP address of VM2.
Right click on IP address, select "Add JMX Connection..."
Enter the VM2 IP Address (from step 2) <IP address>:16001 ( for example -10.12.6.201:9991) and click OK.
Double click on the newly added nodes under "Remote" to start monitoring CPU, Memory & GC.

Sample Screenshot of visualVM

Image Removed

Test Plan

The 72 hours stability test will run the following steps in 20 threaded loop.

...

Please check https://git.onap.org/policy/distribution/tree/s3p/README.stability.md for Jmeter parameter we can use for stability test

Run the following commands to start 72 hours test

Code Block
$ cd /home/ubuntu/policy-distribution/s3p $ rm stability.log; ~/apache-jmeter-5.0/bin/jmeter.sh -n -t stability.jmx -Jhost=pdp -Jduration=259200 -l stability.log f216cfb59484 3 weeks ago 188 MB

Here we'll copy the policy-distribution binaries out of the docker container to run them in the VM directly, so we can made some special tweak for S3P test purpose:

Code Block

$ sudo mkdir -p /opt/app

$ sudo chown ubuntu:ubuntu /opt/app
$ docker run -d -v /opt/app/:/tmp/abc --name policy-distribution -p 6969:6969 -it onap/policy-distribution
# go into docker
$ docker exec -it policy-distribution /bin/bash

In the docker, copy the binaries out and exit docker

Code Block
$ cd /opt/app $ cp -r policy /tmp/abc/ $ exit

Stop the docker

Code Block
$ docker kill policy-distribution $ docker rm policy-distribution

Modify the policy-distribution configuration for S3P test in VM2

Turn off debug output of policy-distribution

Code Block

# open file /opt/app/policy/distribution/etc/logback.xml, search for the following:
 <logger name="com.att.eelf.debug" level="debug" additivity="false">
        <appender-ref ref="asyncEELFDebug" />
  </logger>
# and change the level="debug" to level="warn"
$ vim /opt/app/policy/distribution/etc/logback.xml

Open policy-distribution startup script to open remote jmx support for visualVM

Code Block

# add the following JVM options when launch policy-distribution where 10.12.6.201 is the IP address of VM2
# -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=16001 -Djava.rmi.server.hostname=10.12.6.201
# change the last line of the policy-dist.sh as the following: 
# $JAVA_HOME/bin/java -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=16001 -Djava.rmi.server.hostname=10.12.6.201 -cp "$POLICY_DISTRIBUTION_HOME/etc:$POLICY_DISTRIBUTION_HOME/lib/*" org.onap.policy.distribution.main.startstop.Main -c $CONFIG_FILE


$ vim /opt/app/policy/distribution/bin/policy-dist.sh

Launch policy-distribution service (VM2)

Code Block

# create the default directory which policy distribution service will monitor
$ mkdir -p /tmp/policy_distribution/csar
# launch policy distribution service
$ cd /opt/app/policy/distribution/bin
$ ./policy-dist.sh /opt/app/policy/distribution/etc/s3pConfig.json &

Install & Configure VisualVM (VM2)

VisualVM needs to be installed in the virtual machine having policy-distribution. It will be used to monitor CPU, Memory, GC for policy-distribution while stability test is running.

Install visualVM

Code Block
$ sudo apt-get install visualvm

Run few commands to configure permissions

Code Block

$ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/ 
$ sudo touch visualvm.policy 
$ sudo chmod 777 visualvm.policy

$ vi visualvm.policy 

Add the following in visualvm.policy


grant codebase "file:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar" {
   permission java.security.AllPermission;
};

Run following commands to start jstatd using port 1111

Code Block
$ cd /usr/lib/jvm/java-8-openjdk-amd64/bin/ $ ./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy &

Run visualVM locally to connect to remote VM2

Code Block
# On your windows machine or your linux box locally, launch visualVM

Connect to jstatd & remote apex-pdp JVM

Right click on "Remote" in the left panel of the screen and select "Add Remote Host..."
Enter the IP address of VM2.
Right click on IP address, select "Add JMX Connection..."
Enter the VM2 IP Address (from step 2) <IP address>:16001 ( for example -10.12.6.201:9991) and click OK.
Double click on the newly added nodes under "Remote" to start monitoring CPU, Memory & GC.

Sample Screenshot of visualVM

Image Added

Test Plan

The 72 hours stability test will run the following steps in a single thread loop.

Delete existing policy if it matches the policy we're about to generate - send DELETE restful API to pdp and pap
Generate the csar file and move it into the directory where being monitored.
Get the policy from pdp matching the generated csar - sent restful API to pdp to get the matching policy, retry if failed, until it succeed or retry count exceed.

Please check https://git.onap.org/policy/distribution/tree/s3p/README.stability.md for Jmeter parameter we can use for stability test

Run the following commands to start 72 hours test

Code Block

$ cd /home/ubuntu/policy-distribution/s3p
$ rm stability.log; ~/apache-jmeter-5.0/bin/jmeter.sh -n -t stability.jmx -Jhost=pdp -Jretry=200 -Jduration=259200 -l stability.log
# get the total number csar files generated:
$ grep "Generate_CSAR" stability.log  | wc -l
1057
# get the total number of failed policy to be stored in the pdp:
$ grep "get policy failed" stability.log  | wc -l
950

Versions Compared

Old Version 1

New Version Current

Key

Introduction

Setup details

Basic installation on both VM1 & VM2

Install backend policy services needed by policy-distribution (VM1)

Special settings on policy-distribution/Jmeter VM (VM2)

Install JMeter (VM2)

Build and install policy-distribution in VM2

Special settings on policy-distribution/Jmeter VM (VM2)

Install JMeter (VM2)

Build and install policy-distribution in VM2

Modify the policy-distribution configuration for S3P test in VM2

Launch policy-distribution service (VM2)

Install & Configure VisualVM (VM2)

Test Plan

Modify the policy-distribution configuration for S3P test in VM2

Launch policy-distribution service (VM2)

Install & Configure VisualVM (VM2)

Test Plan

Page Comparison

Versions Compared

Old Version 1

New Version Current

Key

Introduction

Setup details

Basic installation on both VM1 & VM2

Install backend policy services needed by policy-distribution (VM1)

Special settings on policy-distribution/Jmeter VM (VM2)

Install JMeter (VM2)

Build and install policy-distribution in VM2

Special settings on policy-distribution/Jmeter VM (VM2)

Install JMeter (VM2)

Build and install policy-distribution in VM2

Modify the policy-distribution configuration for S3P test in VM2

Launch policy-distribution service (VM2)

Install & Configure VisualVM (VM2)

Test Plan

Modify the policy-distribution configuration for S3P test in VM2

Launch policy-distribution service (VM2)

Install & Configure VisualVM (VM2)

Test Plan