The APPC OOM deployment consists of a 3 node ODL Cluster plus a DB node. The details for the environment can be found at APPC WindRiver Lab. From a testing perspective a directory has been created on the Master APPC node (10.12.5.171). The directory is called "testing."
testing directory
ubuntu@k8s-master:~/testing$ ls -l total 36 drwxrwxr-x 10 ubuntu ubuntu 4096 May 5 00:11 apache-jmeter-4.0 drwxrwxr-x 3 ubuntu ubuntu 4096 May 4 15:16 ONAP-Testing -rw-rw-r-- 1 ubuntu ubuntu 1039 May 4 12:32 rebuild-VM3.json -rw-rw-r-- 1 ubuntu ubuntu 755 May 1 16:40 restart-test-dmaap.json -rw-rw-r-- 1 ubuntu ubuntu 1041 May 4 15:19 snapshot-VM3.json -rw-rw-r-- 1 ubuntu ubuntu 1031 May 4 12:11 start-VM2.json -rw-rw-r-- 1 ubuntu ubuntu 1031 May 4 11:50 stop-VM2.json -rw-rw-r-- 1 ubuntu ubuntu 1004 May 4 11:50 test.json -rw-rw-r-- 1 ubuntu ubuntu 675 Apr 30 17:56 test-restart.json ubuntu@k8s-master:~/testing/apache-jmeter-4.0/APPC-Tests$ ls -l total 167696 -rw-rw-r-- 1 ubuntu ubuntu 104327 May 5 20:34 APPC-LCM-Action-4272018.jmx -rw-rw-r-- 1 ubuntu ubuntu 48609 May 4 14:53 APPC-LCM-Action-Generic_Restart.jmx -rw-rw-r-- 1 ubuntu ubuntu 48673 May 5 20:42 APPC-LCM-Action-Rebuild.jmx -rw-rw-r-- 1 ubuntu ubuntu 56469 May 5 00:37 APPC-LCM-Action-Restart.jmx -rw-rw-r-- 1 ubuntu ubuntu 56425 May 4 14:53 APPC-LCM-Action-Start.jmx -rw-rw-r-- 1 ubuntu ubuntu 56406 May 4 18:02 APPC-LCM-Action-Stop.jmx -rw-rw-r-- 1 ubuntu ubuntu 95 May 4 23:48 appc-lcm-test.csv -rw-rw-r-- 1 ubuntu ubuntu 95 May 4 23:48 appc-lcm-test-Stability-Test-VM2.csv -rw-rw-r-- 1 ubuntu ubuntu 95 May 4 23:48 appc-lcm-test-Stability-Test-VM3.csv -rw-rw-r-- 1 ubuntu ubuntu 1451122 May 7 09:23 appc-oom-72hr.jtl -rw-rw-r-- 1 ubuntu ubuntu 2695327 May 7 09:25 appc-oom-rebuild-72hr.jtl -rw-rw-r-- 1 ubuntu ubuntu 167156886 May 7 09:30 jmeter.log drwxr-xr-x 2 root root 4096 May 4 23:48 temp -rw-r--r-- 1 root root 159 May 4 23:48 test-scott.jtl ubuntu@k8s-master:~/testing/ONAP-Testing$ ls -l total 12 drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:17 keys -rw-rw-r-- 1 ubuntu ubuntu 26 May 4 15:17 login.txt drwxrwxr-x 5 ubuntu ubuntu 4096 May 4 15:16 Test Data ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data$ ls -l total 24 drwxrwxr-x 4 ubuntu ubuntu 4096 May 4 15:17 APPC-Stability-Test -rw-rw-r-- 1 ubuntu ubuntu 999 May 4 15:16 Stop-Stability-Test-VM1-VM.json drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:16 vCPE drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:16 vDNS -rw-rw-r-- 1 ubuntu ubuntu 6144 May 4 15:16 vTESTVM1.xls ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data$ cd APPC-Stability-Test ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data/APPC-Stability-Test$ ls -l total 24 drwxrwxr-x 5 ubuntu ubuntu 4096 May 4 15:16 A&AI Data -rw-rw-r-- 1 ubuntu ubuntu 10079 May 4 15:17 APPC-AAI-Data.txt -rw-rw-r-- 1 ubuntu ubuntu 3587 May 4 15:17 APPC_Test_VM_SAMPLE.tar.gz drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:17 vDNS ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data/APPC-Stability-Test$ cd "A&AI Data" ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data/APPC-Stability-Test/A&AI Data$ ls -l total 12 drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:17 Stability-Test-VM1 drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:17 Stability-Test-VM2 drwxrwxr-x 2 ubuntu ubuntu 4096 May 4 15:17 Stability-Test-VM3 ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data/APPC-Stability-Test/A&AI Data/Stability-Test-VM1$ ls -l total 76 -rw-rw-r-- 1 ubuntu ubuntu 3325 May 4 15:17 cloud-region.json -rw-rw-r-- 1 ubuntu ubuntu 300 May 4 15:17 generic-vnf.json -rw-rw-r-- 1 ubuntu ubuntu 289 May 4 15:17 model.json -rw-rw-r-- 1 ubuntu ubuntu 314 May 4 15:17 named-query.json -rw-rw-r-- 1 ubuntu ubuntu 3381 May 4 15:17 put_closed_loop.sh -rw-rw-r-- 1 ubuntu ubuntu 1103 May 4 15:17 put_named_query.sh -rw-rw-r-- 1 ubuntu ubuntu 794 May 4 15:17 update-esr.sh -rw-rw-r-- 1 ubuntu ubuntu 518 May 4 15:17 update-esr-status.json -rw-rw-r-- 1 ubuntu ubuntu 125 May 4 15:17 update-identity-url.json -rw-rw-r-- 1 ubuntu ubuntu 492 May 4 15:17 update.sh -rw-rw-r-- 1 ubuntu ubuntu 337 May 4 15:17 update-vnf-status.json -rw-rw-r-- 1 ubuntu ubuntu 597 May 4 15:17 update-vserver-selflink.json -rw-rw-r-- 1 ubuntu ubuntu 799 May 4 15:17 update-vserver.sh -rw-rw-r-- 1 ubuntu ubuntu 450 May 4 15:17 verify.sh -rw-rw-r-- 1 ubuntu ubuntu 950 May 4 15:17 VNFC_APPC_Test_VM.json -rw-rw-r-- 1 ubuntu ubuntu 10901 May 4 15:17 vnf-to-esr-info-1.0.json -rw-rw-r-- 1 ubuntu ubuntu 115 May 4 15:17 vserver-generic-vnf-relationship.json ubuntu@k8s-master:~/testing/ONAP-Testing/Test Data/APPC-Stability-Test/A&AI Data/Stability-Test-VM1$
Within the testing directory there are sample test json requests along with jmeter and A&AI data load scripts. The breakdown for these directories is below:
apache-jmeter-4.0
This directory contains APACHE Jmeter along with APPC-Tests. The APPC-Tests subfolder contains jmx files that can be utilized to run specific APPC LCM Actions. The tests can be executed either command line or via the UI.
- Command Line (The example below is executing the test from the testing->apache-jmeter-4.0->APPC-Tests folder):
- ../bin/jmeter.sh -n -L DEBUG -t APPC-LCM-Action-Rebuild.jmx l APPCLCM-Action-Rebuild.jtl
- The jmx file actually contains the test "script" and the jtl file is the formatted test results file.
- ../bin/jmeter.sh -n -L DEBUG -t APPC-LCM-Action-Rebuild.jmx l APPCLCM-Action-Rebuild.jtl
- Executing JMeter from the UI:
- /home/ubuntu/testing/apache-jmeter-4.0/jmeter
- This will pull up the UI. However, you will need to have X11 setup on your desktop.
ONAP-Testing
ONAP Testing directory contains the data and scripts that will allow the user to update a particular A&AI instance.
put_closed_loop_sh is the script that actually "puts" news data into A&AI.
./put_closed_loop.sh aaiporhost |python -m json.tool
The below files need to be configured for the specific VNF being loaded:
cloud-region.json
model.json
generic-vnf-.json
vserver-generic-vnf-relationship.json
Additionally, for the Stability Testing 3 VMs were created. Each VM has a separate directory named accordingly. These files for the specific VNFs do not require modification if there is need to load the data for that VNF in a new or cleaned out A&AI>
verify.sh is the script that can be utilized to verify the VNF is loaded into A&AI. In order to query for the specific VNF simply modify the named-query.json file to reflect that VNF.
Pre-Defined data has been created for:
Stability-Test-VM1
Stability-Test-VM2
Stability-Test-VM3
vDNS
Common commands utilzied in the OOM env:
- kubectl get pods --all-namespaces -o wide -w
Displays all the running pods and related information.
POD InformationCubuntu@k8s-master:~$ kubectl get pods --all-namespaces -o wide -w NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system etcd-k8s-master 1/1 Running 1 6d 10.12.5.171 k8s-master kube-system kube-apiserver-k8s-master 1/1 Running 1 6d 10.12.5.171 k8s-master kube-system kube-controller-manager-k8s-master 1/1 Running 1 6d 10.12.5.171 k8s-master kube-system kube-dns-86f4d74b45-px44s 3/3 Running 12 19d 10.32.0.3 k8s-master kube-system kube-proxy-25tm5 1/1 Running 4 19d 10.12.5.171 k8s-master kube-system kube-proxy-6dt4z 1/1 Running 3 19d 10.12.5.174 k8s-appc1 kube-system kube-proxy-jmv67 1/1 Running 3 19d 10.12.5.193 k8s-appc2 kube-system kube-proxy-l8fks 1/1 Running 4 19d 10.12.5.194 k8s-appc3 kube-system kube-scheduler-k8s-master 1/1 Running 1 6d 10.12.5.171 k8s-master kube-system tiller-deploy-84f4c8bb78-p8b6j 1/1 Running 0 6d 10.36.0.4 k8s-appc3 kube-system weave-net-bz7wr 2/2 Running 13 19d 10.12.5.194 k8s-appc3 kube-system weave-net-c2pxd 2/2 Running 10 19d 10.12.5.174 k8s-appc1 kube-system weave-net-jw29c 2/2 Running 10 19d 10.12.5.171 k8s-master kube-system weave-net-kxxpl 2/2 Running 10 19d 10.12.5.193 k8s-appc2 onap onap-appc-0 2/2 Running 0 3d 10.44.0.5 k8s-appc1 onap onap-appc-1 2/2 Running 0 1h 10.47.0.1 k8s-appc2 onap onap-appc-2 2/2 Running 0 1h 10.36.0.5 k8s-appc3 onap onap-appc-cdt-5474c7cb88-8sfqp 1/1 Running 0 3d 10.36.0.10 k8s-appc3 onap onap-appc-db-0 2/2 Running 0 3d 10.47.0.2 k8s-appc2 onap onap-appc-dgbuilder-5dccdf57b5-qntzg 1/1 Running 0 3d 10.47.0.6 k8s-appc2 onap onap-dbcl-db-0 1/1 Running 0 3d 10.47.0.4 k8s-appc2 onap onap-dbcl-db-1 1/1 Running 0 3d 10.44.0.8 k8s-appc1 onap onap-dmaap-7dfc8f84d6-qmqmx 1/1 Running 1 3d 10.36.0.8 k8s-appc3 onap onap-dmaap-bus-controller-6c4d4d55cf-jrf4w 1/1 Running 0 3d 10.47.0.8 k8s-appc2 onap onap-global-kafka-6bb8cf75c8-dr5rm 1/1 Running 2 3d 10.36.0.11 k8s-appc3 onap onap-log-elasticsearch-855f94ccc4-jsc8t 1/1 Running 0 3d 10.47.0.7 k8s-appc2 onap onap-log-kibana-764bd47f-wxhhh 1/1 Running 0 3d 10.47.0.9 k8s-appc2 onap onap-log-logstash-776fd9fbd6-7hzhg 1/1 Running 0 3d 10.44.0.2 k8s-appc1 onap onap-robot-d7c758c5f-fz68f 1/1 Running 0 3d 10.44.0.4 k8s-appc1 onap onap-zookeeper-7774c66fb9-5nwmd 1/1 Running 0 3d 10.44.0.9 k8s-appc1
- kubectl exec -ti onap-appc-0 -c appc -n onap bash
- This command is executed from the Master node and opens a bash shell into the particular pod.
New or Clean APPC DB
Data for the different actions must be loaded into the PROTOCOL_REFERENCE table whenever a new instance of the APPC DB is instantiated or cleaned. In order to load the Stability Tests VNFs use the following:
1. kubectl exec -ti onap-appc-db-0 -c appc-db -n onap bash 2. mysql -u sdnctl -p 3. Enter the following SQL command to insert the data (First column is the unique key for each record...number is not as important as the fact it should be unique within the table): INSERT INTO `PROTOCOL_REFERENCE` VALUES (1,'Stop','vTESTVM','OS',now(),'NO','vm'); INSERT INTO `PROTOCOL_REFERENCE` VALUES (2,'Start','vTESTVM','OS',now(),'NO','vm'); INSERT INTO `PROTOCOL_REFERENCE` VALUES (4,'Rebuild','vTESTVM','OS',now(),'NO','vm'); INSERT INTO `PROTOCOL_REFERENCE` VALUES (3,'Restart','vTESTVM','OS',now(),'NO','vnf'); INSERT INTO `PROTOCOL_REFERENCE` VALUES (5,'Restart','vTESTVM','OS',now(),'NO','vm'); INSERT INTO `PROTOCOL_REFERENCE` VALUES (18,'Snapshot','vTESTVM','OS',now(),'NO','vm');
KIbana Queries
- Querying for a specific string in the Message filed:
- message:"REBUILD_STATUS -- SUCCESS"
- Querying for all transactions for a RequestId:
- RequestId:"RequestId you are searching for"
- View for Restart and Rebuilds can be found here.