The intent of the 72 hour stability test is not to exhaustively test all functions but to run a steady load against the system and look for issues like memory leaks that aren't found in the short duration install and functional testing during the development cycle.
This page will collect notes on the 72 hour stability test run for Frankfurt.
See El Alto Stability Run Notes for comparison to previous runs.
Summary of Results
WORK IN PROGRESS
Setup
The integration-longevity tenant in Intel/Windriver environment was used for the 72 hour tests.
The onap-ci job for "Project windriver-longevity-release-manual" was used for the deployment with the OOM set to frankfurt and Integration branches set to master. Integraiton master was used so we could catch the latest updates to integration scripts and vnf heat templates.
The jenkins job needs a couple of updates for each release:
- Set the integration branch to 'origin/master'
- Modify the parameters to deploy.sh to specify "-i master" and "-o frankfurt" to get integration master an oom frankfurt clones onto the nfs server.
The path for robot logs on dockerdata-nfs changed in Frankfurt so the /dev-robot/ becomes /dev/robot
The stability tests used robot container image 1.6.1-STAGING-20200519T201214Z
robot container updates:
API_TYPE was set to GRA_API since we have deprecated VNF_API.
Shakedown consists of creating some temporary tags for stability72hrvLB, stability72hrvVG,stability72hrVFWCL to make sure each sub test ran successfully (including cleanup) in the environment before the jenkins job started with the higher level testsuite tag stability72hr that covers all three test types.
Clean out the old buid jobs using a jenkins console script (manage jenkins)
def jobName = "windriver-longevity-stability72hr"
def job = Jenkins.instance.getItem(jobName)
job.getBuilds().each { it.delete() }
job.nextBuildNumber = 1
job.save()
appc.properties updated to apply the fix for DMaaP message processing to call http://localhost:8181 for the streams update.
VNF Orchestration Tests
This test uses the onap-ci job "Project windriver-longevity-stability72hr" to automatically onboard, distribute and instantiate the ONAP opensource test VNFs vLB, vVG and vFWCL.
The scripts run validation tests after the install.
The scripts then delete the VNFs and cleans up the environment for the next run.
The script tests AAF, DMaaP, SDC, VID, AAI, SO, SDNC, APPC with the open source VNFs.
There was a problem with the robot scripts for vLB where it was not finding the base_lb.yaml file in the artifacts due to a change in the structure. A two line change to the vnf orchestration script to look for the 'heat3' key was made to resolve the issue. A Jira was created to track the changes to the robot scrips. INT-1598 - Getting issue details... STATUS
These tests started at jenkins job #1
Each test run generates over 500 MB of data on the test through robot framework.
| Test # | Comment | Message |
|---|---|---|
Closed Loop Tests
This test uses the onap-ci job "Project windriver-longevity-vfwclosedloop".
The test uses the robot test script "demo-k8s.sh vfwclosedloop ". The script sets the number of streams on the vPacket Generator to 10 , waits for the change from 10 set sreams to 5 streams by the control loop then sets the stream to 1 and again waits for the 5 streams.
Success tests the loop from VNF through DCAE, DMaaP, Policy, AAI , AAF and APPC.
In the jenkins job:
Modify the NFS_IP and PKG_IP in the jenkins job to point to the current nfs server and packet generator in the tenant
NFS_IP=10.12.5.205
PKG_IP=10.12.5.247
The tests start with #1
| Test # | Comment | Message |
|---|---|---|