Introduction

The 72 hour Stability Test for policy distribution has the goal of introducing a steady flow of transactions initiated from a test client server running JMeter. The policy distribution is configured with a special FileSystemReception plugin to monitor a local directory for newly added csar files to be processed by itself. The input csar files will be generated by test client (JMeter) and the result will be pulled from backend PDP by test client(JMeter).

The test will be performed in an environment where JMeter will continuously generate random csar files into the special directory where policy distribution monitors and get the processed results from backend PDP by restful calls(10 retries with 1 seconds between each retry) for the duration of 72 hours.

Setup details

The stability test is performed on VM's running in the OpenStack cloud environment in the ONAP integration lab. There are 2 separate VMs, one for running backend policy services which policy distribution needs, and the other is for policy distribution service itself and jmeter.

OpenStack environment details

Version: Windriver Titanium

policy backend VM details (VM1)

OS:Ubuntu 16.04.4 LTS

CPU: 8 core

RAM: 32 GB

HardDisk: 160 GB

Docker Version: 17.03.2-ce, build f5ec1e2

Java: openjdk version "1.8.0_181"

JMeter VM details (VM2)

OS: Ubuntu 16.04.4LTS

CPU: 8 core

RAM: 32 GB

HardDisk: 160 GB

Docker Version: 17.03.2-ce, build f5ec1e2

Java: openjdk version "1.8.0_181"

JMeter: 5.0

Basic installation on both VM1 & VM2

Install java, docker, docker-compose and other utilities on both VMs.

$ sudo apt-get update
$ sudo apt-get install -y openjdk-8-jdk docker docker-compose zip unzip curl wget ssh telnet maven
# make sure docker can be launched with normal user
$ sudo groupadd docker
$ sudo gpasswd -a $USER docker
$ newgrp docker

Install backend policy services needed by policy-distribution (VM1)

Get policy/docker code:

$ git clone https://git.onap.org/policy/docker policy-docker
$ cd policy-docker/
$ chmod +x config/drools/drools-tweaks.sh
$ IP_ADDRESS=$(ifconfig ens3 | grep "inet addr" | tr -s ' ' | cut -d' ' -f3 | cut -d':' -f2)
$ echo $IP_ADDRESS > config/pe/ip_addr.txt

Pull docker images:

$ docker login -u docker -p docker nexus3.onap.org:10001
$ docker image pull mariadb:10.2.14
$ docker image pull sonatype/nexus:2.14.8-01
$ docker image pull nexus3.onap.org:10001/onap/policy-pe
$ docker image tag nexus3.onap.org:10001/onap/policy-pe onap/policy-pe
$ docker image pull nexus3.onap.org:10001/onap/policy-drools
$ docker image tag nexus3.onap.org:10001/onap/policy-drools onap/policy-drools
$ docker images
REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
onap/policy-pe                             latest              8d3b9cec2e48        35 hours ago        1.6 GB
nexus3.onap.org:10001/onap/policy-pe       latest              8d3b9cec2e48        35 hours ago        1.6 GB
nexus3.onap.org:10001/onap/policy-drools   latest              aba8c4afcd0c        46 hours ago        1.11 GB
onap/policy-drools                         latest              aba8c4afcd0c        46 hours ago        1.11 GB
mariadb                                    10.2.14             2cbeee8bd9cf        6 months ago        399 MB
sonatype/nexus                             2.14.8-01           ee28e4c8635c        8 months ago        468 MB

Build docker and launch various policy services:

Special settings on policy-distribution/Jmeter VM (VM2)

Add ip address of VM1 /etc/hosts entries NOTE: please replace the correct IP address of VM1 in your environment

Install root CA

Install JMeter (VM2)

Download & install JMeter

Build and install policy-distribution in VM2

Here we'll build policy distribution docker by ourselves, or you can get the docker image directly from onap nexus3 repo and skip the next step:

^{Here we'll copy the policy-distribution binaries out of the docker container to run them in the VM directly, so we can made some special tweak for S3P test purpose:}

^{In the docker, copy the binaries out and exit docker}

^{Stop the docker}

Modify the policy-distribution configuration for S3P test in VM2

^{Turn off debug output of policy-distribution}

^{Open policy-distribution startup script to open remote jmx support for visualVM}

Launch policy-distribution service (VM2)

Install & Configure VisualVM (VM2)

VisualVM needs to be installed in the virtual machine having policy-distribution. It will be used to monitor CPU, Memory, GC for policy-distribution while stability test is running.

Install visualVM

Run few commands to configure permissions

Run following commands to start jstatd using port 1111

Run visualVM locally to connect to remote VM2

Connect to jstatd & remote apex-pdp JVM

Right click on "Remote" in the left panel of the screen and select "Add Remote Host..."
Enter the IP address of VM2.
Right click on IP address, select "Add JMX Connection..."
Enter the VM2 IP Address (from step 2) <IP address>:16001 ( for example -10.12.6.201:9991) and click OK.
Double click on the newly added nodes under "Remote" to start monitoring CPU, Memory & GC.

Sample Screenshot of visualVM

Test Plan

The 72 hours stability test will run the following steps in a single thread loop.

Delete existing policy if it matches the policy we're about to generate - send DELETE restful API to pdp and pap
Generate the csar file and move it into the directory where being monitored.
Get the policy from pdp matching the generated csar - sent restful API to pdp to get the matching policy, retry if failed, until it succeed or retry count exceed.

Please check https://git.onap.org/policy/distribution/tree/s3p/README.stability.md for Jmeter parameter we can use for stability test

Run the following commands to start 72 hours test

ONAP Wiki

72 Hours Stability Test Plan of Policy SDC Service Distribution