Introduction

The 72 hour Stability Test for Drools PDP has the goal of introducing a steady flow of transactions initiated from a Test Client server running JMeter. The onset messages will be submitted through REST interface of Drools PDP. The results are verified through REST interface of Drools PDP using facts, source & sink topics. The JMeter testing will exercise the following use cases:

...

The stability test will loop through these use cases sequentially in a single two threaded environment (for each use case) for the duration of 72 hours.

Setup details

The stability test is performed on VM's running in OpenStack cloud environment. There are 2 seperate VM's, one for running policy framework & other one for running JMeter to simulate steady flow of transactions. For this test, the components involved in the flow outside of Policy (i.e. VFC, A&AI & SO) will be simulated using "features enable controlloop-utils".

OpenStack environment details

...

OS: Linux 4.4.0-119-generic #143-Ubuntu SMP
CPU: 4core
RAM: 8GB
HardDisk: 80GB
Java: openjdk version "1.8.0_162"

Install JMeter in virtual machine

Make the etc/hosts entries

...

The above command will load the JMeter UI. Then navigate to File → Open → Browse and select the test plan jmx file to open.

Install ONAP Policy Framework in virtual machine

Follow the steps mentioned in the below link to install ONAP Policy Framework with controllers & policies in a virtual machine.ONAP Policy Framework: Standalone installation in Virtual Machine

/wiki/spaces/DW/pages/16283669

Install & Configure VisualVM

VisualVM needs to be installed in the virtual machine having policy framework. It will be used to monitor CPU, Memory, GC for drools PDP while stability test is running.

Install visualVM

sudo apt-get install visualvm

Login to docker container (using root)

docker exec -e -it -u 0 drools su - root

Run few commands to configure permissions

cd /usr/lib/jvm/java-8-openjdk-amd64/bin/ 

touch visualvm.policy 

vi visualvm.policy 

Add the following in visualvm.policy


grant codebase "file:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar" {
   permission java.security.AllPermission;
};


chmod 777 visualvm.policy


exit

Login to docker container (using policy)

docker exec -e -it -u 0 drools su - policy

Run following commands to start jstatd using port 1111

cd /usr/lib/jvm/java-8-openjdk-amd64/bin/ 


./jstatd -p 1111 -J-Djava.security.policy=visualvm.policy  &


exit

Login to VM using graphical interface in separate terminal window.

ssh -X <user>@<VM-IP-ADDRESS>

Open visualVM

visualvm &

Connect to jstatd & remote drools PDP JVM

1. Right click on "Remote" in the left panel of the screen and select "Add Remote Host..."

2. Enter the IP address of drools docker container. Run the following command to check the IP address of drools container in VM.

   Code Block
   docker inspect --format '{{ .NetworkSettings.Networks.docker_default.IPAddress}}' drools

3. Right click on IP address, select "Add jstatd Connection..."
4. In "jstatd Connections" tab, enter port 1111 and click OK.
5. Right click on IP address, select "Add JMX Connection..."
6. Enter the drools docker container IP Address (from step 2) <IP address>:9991 ( for example - 172.18.0.7:9991) and click OK.
7. Double click on the newly added nodes under "Remote" to start monitoring CPU, Memory & GC.

Sample Screenshot of visualVM

Image Added

Test Plan

The 72 hours stability test will run the following steps one by one sequentially in single two threaded loop.

• Check facts in memory - Before starting the test, we need to make sure that there are no stuck events in engine.

• vCPE controller - The vCPE controller will be is responsible for trigerring the vCPE success test cases.
  
  
• VCPE VNF ID Generator - This will generate generates a new VNF ID for every request.
  
  
• vCPE Request ID Generator - This will generate generates a new Request ID for every request.
  
  
• vCPE ONSET Request - This step will send sends a vCPE onset message to REST interface of drools PDP.
  
  
• vCPE APPC Success Response - This step will send sends the APPC success response message to REST interface of drools PDP.
  
  
• vCPE ABATEMENT Request - This step will send sends the vCPE ABATEMENT message to REST interface of drools PDP.
  
  
• vCPE Assertion Controller - This controller will be is responsible for performing assertions for the request sent earlier.
  
  
• Get vCPE Sink Events - This step will fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of vCPE onset message sent earlier.
  
  
• vCPE Failure Controller - The vCPE controller will be is responsible for trigerring the vCPE failure test cases.
  
  
• vCPE Failure VNF ID Generator - This will generate generates a new VNF ID for every request.
  
  
• vCPE Failure Request ID Generator - This will generate generates a new Request ID for every request.
  
  
• vCPE Failure ONSET Request - This step will send sends a vCPE onset message to REST interface of drools PDP.
  
  
• vCPE Failure APPC Failure Response - This step will send sends the APPC failure response message to REST interface of drools PDP.
  
  
• vCPE Failure Assertion Controller - This controller will be is responsible for performing assertions for the request sent earlier.
  
  
• Get vCPE Failure Sink Events - This step will fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL FAILURE notification against the request ID of vCPE onset message sent earlier.
  
  
• vDNS Controller - The vDNS controller will be is responsible for trigerring the vDNS test flow.
  
  
• vDNS vServer Name Generator - This step will generate generates a new vServer name for every request.
  
  
• vDNS Request ID Generator - This will generate generates a new Request ID for every request.
  
  
• vDNS ONSET Request - This step will send sends a vDNS onset message to REST interface of drools PDP.
  
  
• vDNS Assertion Controller - This controller will be is responsible for performing assertions for the request sent earlier.
  
  
• Get vDNS Sink Events - This step will fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of vDNS onset message sent earlier.
  
  
• vFW controller - The vFW controller will be is responsible for trigerring the vFirewall test flow.
  
  
• vFW VNF ID Generator - This will generate a new VNF ID for every request.
  
  
• vFW Request ID Generator - This will generate a new Request ID for every request.
  
  
• vFirewall vFW ONSET Request - This step will send sends a vFirewall onset message to REST interface of drools PDP.
  
  
• vFW APPC Legacy Success Response - This step will send sends the APPC success response message to REST interface of drools PDP.
  
  
• vFW Assertion Controller - This controller will be is responsible for performing assertions for the request sent earlier.
  
  
• Get vFW Sink Events - This step will fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of vFirewall onset message sent earlier.
  
  
• vFW Failure Controller - The vFW controller is responsible for trigerring the vFirewall failure test cases.
  
  
• vFW Failure VNF ID Generator - generates a new VNF ID for every request.
  
  
• vFW Failure Request ID Generator - generates a new Request ID for every request.
  
  
• vFW Failure ONSET Request - sends a vFirewall onset message to REST interface of drools PDP.
  
  
• vFW Failure APPC Response - sends the APPC failure response message to REST interface of drools PDP.
  
  
• vFW Failure Assertion Controller - This controller is responsible for performing assertions for the request sent earlier.
  
  
• vFW Failure Sink Events - fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL FAILURE notification against the request ID of vFirewall onset message sent earlier.
  
  
• VOLTE Controller - The VOLTE controller will be is responsible for triggering the VOLTE test flow.
  
  
• VOLTE Request ID Generator - This will generate generates a new Request ID for every request.
  
  
• VOLTE vServer Name Generator - This step will generate generates a new vServer name for every request.
  
  
• VOLTE VNF ID Generator - This will generate generates a new VNF ID for every request.
  
  
• VOLTE ONSET Request - This step will send sends a VOLTE onset message to REST interface of drools PDP.
  
  
• VOLTE Assertion Controller - This controller will be is responsible for performing assertions for the request sent earlier.
  
  
• Get VOLTE Sink Events - This step will fetch all the events from sink topic POLICY-CL-MGT and then verify the FINAL SUCCESS notification against the request ID of VOLTE onset message sent earlier.
  
  
• Check facts in memory - Before ending the test, we need to make sure that there are no stuck events in engine.

The following steps can be used to configure the parameters of test plan.

• HTTP Authorization Manager - used to store user credentials which will be used for making HTTP requests.
• HTTP Request Defaults -  used to store HTTP request details like Server Name or IP, Port, Protocol etc.

• User Defined Variables -  used to store following parameters.

  Name	Description	Default Value
  size	Indicates the maximum number of VNF instances. Used for generating VNF IDs.	2000
  wait	Wait time after each request so that the same gets processed by engine (in milliseconds)	500
  vCPEControlLoop	vCPE Control loop name	ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e
  vDNSControlLoop	vDNS Control loop name	ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3
  vFWControlLoop	vFirewall Controll loop name	ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a
  VOLTEControlLoop	VOLTE Control loop name	ControlLoop-VOLTE-2179b738-fd36-4843-a71a-a8c24c70c55b
  threads	Number of threads to run test cases in parallel. It should not be more than 2 as the sink queue size is 10.

  
  Image Removed

  	2
  threadsTimeOutInMs	Synchronization timer for threads running in parallel (in milliseconds).	5000

               

Screenshot of drools PDP stability test plan

Image Added

Important Note: There is wait introduced after sending every ONSET, Response, Request messages to drools PDP so that it is able to process the message before getting the next one. If needed, the wait time can be suitably adjusted based on the setup and time taken by drools PDP to process the messages.