Introduction

The 72 hour Stability Test for Drools PDP has the goal of introducing a steady flow of transactions initiated from a Test Client server running JMeter. The onset messages will be submitted through REST interface of Drools PDP. The results are verified through REST interface of Drools PDP using facts, source & sink topics. The JMeter testing will exercise the following use cases:

vCPE
vFirewall
vDNS
VOLTE

The stability test will loop through these use cases sequentially in two threaded environment (for each use case) for the duration of 72 hours.

Setup details

The stability test is performed on VM's running in OpenStack cloud environment. There are 2 seperate VM's, one for running policy framework & other one for running JMeter to simulate steady flow of transactions. For this test, the components involved in the flow outside of Policy (i.e. VFC, A&AI & SO) will be simulated using "features enable controlloop-utils".

OpenStack environment details

Version: Mitaka 9.2

Policy Framework VM details

OS: Linux 4.4.0-119-generic #143-Ubuntu SMP
CPU: 4core
RAM: 8GB
HardDisk: 80GB
Docker Version: 18.03.0-ce, build 0520e24
Java: openjdk version "1.8.0_162"

JMeter VM details

OS: Linux 4.4.0-119-generic #143-Ubuntu SMP
CPU: 4core
RAM: 8GB
HardDisk: 80GB
Java: openjdk version "1.8.0_162"

Install JMeter in virtual machine

Make the etc/hosts entries

echo $(hostname -I | cut -d\  -f1) $(hostname) | sudo tee -a /etc/hosts

Make the DNS entries

echo "nameserver <PrimaryDNSIPIP>" >> /etc/resolvconf/resolv.conf.d/head
 
echo "nameserver <SecondaryDNSIP>" >> /etc/resolvconf/resolv.conf.d/head
 
resolvconf -u

Update the ubuntu software installer

apt-get update

Check & Install Java

apt-get install -y openjdk-8-jdk
 
java -version

Download & install JMeter

mkdir jMeter


cd jMeter


wget http://mirrors.whoishostingthis.com/apache//jmeter/binaries/apache-jmeter-4.0.zip


unzip apache-jmeter-4.0.zip 


Follow the instructions in this link for installing plugin manager - https://jmeter-plugins.org/wiki/PluginsManager/

Run JMeter

/home/ubuntu/jMeter/apache-jmeter-4.0/bin/jmeter

The above command will load the JMeter UI. Then navigate to File → Open → Browse and select the test plan jmx file to open.

Install ONAP Policy Framework in virtual machine

Follow the steps mentioned in the below link to install ONAP Policy Framework with controllers & policies in a virtual machine.

/wiki/spaces/DW/pages/16283669

Install & Configure VisualVM

VisualVM needs to be installed in the virtual machine having policy framework. It will be used to monitor CPU, Memory, GC for drools PDP while stability test is running.

Install visualVM

sudo apt-get install visualvm

Login to docker container (using root)

docker exec -e -it -u 0 drools su - root

Run few commands to configure permissions

cd /usr/lib/jvm/java-8-openjdk-amd64/bin/ 

touch visualvm.policy 

vi visualvm.policy 

Add the following in visualvm.policy


grant codebase "file:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar" {
   permission java.security.AllPermission;
};


chmod 777 visualvm.policy


exit

Login to docker container (using policy)

docker exec -e -it -u 0 drools su - policy

Run following commands to start jstatd using port 1111

cd /usr/lib/jvm/java-8-openjdk-amd64/bin/ 


./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy  &


exit

Login to VM using graphical interface in separate terminal window.

ssh -X <user>@<VM-IP-ADDRESS>

Open visualVM

visualvm &

Connect to jstatd & remote drools PDP JVM

Right click on "Remote" in the left panel of the screen and select "Add Remote Host..."
Enter the IP address of drools docker container. Run the following command to check the IP address of drools container in VM.
```
 docker inspect --format '{{ .NetworkSettings.Networks.docker_default.IPAddress}}' drools
```
Right click on IP address, select "Add jstatd Connection..."
In "jstatd Connections" tab, enter port 1111 and click OK.
Right click on IP address, select "Add JMX Connection..."
Enter the drools docker container IP Address (from step 2) <IP address>:9991 ( for example - 172.18.0.7:9991) and click OK.
Double click on the newly added nodes under "Remote" to start monitoring CPU, Memory & GC.

Sample Screenshot of visualVM

Test Plan

The 72 hours stability test will run the following steps one by one sequentially in two threaded loop.

Check facts in memory - Before starting the test, we need to make sure that there are no stuck events in engine.

vCPE controller - The vCPE controller is responsible for trigerring the vCPE success test cases.
VCPE VNF ID Generator - generates a new VNF ID for every request.
vCPE Request ID Generator - generates a new Request ID for every request.
vCPE ONSET Request - sends a vCPE onset message to REST interface of drools PDP.
vCPE APPC Success Response - sends the APPC success response message to REST interface of drools PDP.
vCPE ABATEMENT Request - sends the vCPE ABATEMENT message to REST interface of drools PDP.
vCPE Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.
vCPE Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of vCPE onset message sent earlier.
vCPE Failure Controller - The vCPE controller is responsible for trigerring the vCPE failure test cases.
vCPE Failure VNF ID Generator - generates a new VNF ID for every request.
vCPE Failure Request ID Generator - generates a new Request ID for every request.
vCPE Failure ONSET Request - sends a vCPE onset message to REST interface of drools PDP.
vCPE Failure APPC Response - sends the APPC failure response message to REST interface of drools PDP.
vCPE Failure Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.
vCPE Failure Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL FAILURE notification against the request ID of vCPE onset message sent earlier.
vDNS Controller - The vDNS controller is responsible for trigerring the vDNS test flow.
vDNS vServer Name Generator - generates a new vServer name for every request.
vDNS Request ID Generator - generates a new Request ID for every request.
vDNS ONSET Request - sends a vDNS onset message to REST interface of drools PDP.
vDNS Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.
vDNS Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of vDNS onset message sent earlier.
vFW controller - The vFW controller is responsible for trigerring the vFirewall test flow.
vFW VNF ID Generator - generate a new VNF ID for every request.
vFW Request ID Generator - generate a new Request ID for every request.
vFW ONSET Request - sends a vFirewall onset message to REST interface of drools PDP.
vFW APPC Legacy Success Response - sends the APPC success response message to REST interface of drools PDP.
vFW Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.
vFW Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of vFirewall onset message sent earlier.
vFW Failure Controller - The vFW controller is responsible for trigerring the vFirewall failure test cases.
vFW Failure VNF ID Generator - generates a new VNF ID for every request.
vFW Failure Request ID Generator - generates a new Request ID for every request.
vFW Failure ONSET Request - sends a vFirewall onset message to REST interface of drools PDP.
vFW Failure APPC Response - sends the APPC failure response message to REST interface of drools PDP.
vFW Failure Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.
vFW Failure Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL FAILURE notification against the request ID of vFirewall onset message sent earlier.
VOLTE Controller - The VOLTE controller is responsible for triggering the VOLTE test flow.
VOLTE Request ID Generator - generates a new Request ID for every request.
VOLTE vServer Name Generator - generates a new vServer name for every request.
VOLTE VNF ID Generator - generates a new VNF ID for every request.
VOLTE ONSET Request - sends a VOLTE onset message to REST interface of drools PDP.
VOLTE Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.
VOLTE Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of VOLTE onset message sent earlier.
Check facts in memory - Before ending the test, we need to make sure that there are no stuck events in engine.

The following steps can be used to configure the parameters of test plan.

HTTP Authorization Manager - used to store user credentials which will be used for making HTTP requests.
HTTP Request Defaults - used to store HTTP request details like Server Name or IP, Port, Protocol etc.

User Defined Variables - used to store following parameters.

Name	Description	Default Value
size	Indicates the maximum number of VNF instances. Used for generating VNF IDs.	2000
wait	Wait time after each request so that the same gets processed by engine (in milliseconds)	500
vCPEControlLoop	vCPE Control loop name	ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e
vDNSControlLoop	vDNS Control loop name	ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3
vFWControlLoop	vFirewall Controll loop name	ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a
VOLTEControlLoop	VOLTE Control loop name	ControlLoop-VOLTE-2179b738-fd36-4843-a71a-a8c24c70c55b
threads	Number of threads to run test cases in parallel. It should not be more than 2 as the sink queue size is 10.	2
threadsTimeOutInMs	Synchronization timer for threads running in parallel (in milliseconds).	5000

Screenshot of drools PDP stability test plan

Important Note: There is wait introduced after sending every ONSET, Response, Request messages to drools PDP so that it is able to process the message before getting the next one. If needed, the wait time can be suitably adjusted based on the setup and time taken by drools PDP to process the messages.