Versions Compared

Old Version 2

changes.mady.by.user FREEMAN, BRIAN D

Saved on

compared with

New Version Current

changes.mady.by.user FREEMAN, BRIAN D

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This page will collect notes on the 72 hour stability test run for El Alto.

...


Summary

The integration-longevity tenant in Intel/Windriver environment was used for the 72 hour tests.

The onap-ci job for  "Project windriver-longevity-release-manual" was used for the deployment with the OOM and Integration branches set to elalto.

The deployment was fairly clean but there was an environment issue that required a few pods to be recycled by the normal k8 delete pod due to a what looked like a network blimp during the install.

We also hit the environment dhcp bug where the VMs would get an external dhcp address from a different network than openstack's dhcp. The symptom is not being able to log into the external IP of the VM.

This is solved by a force reboot of the VM from the horizon portal but unfortunately this prevents the installation of the demo VNF config files so the VM install script has to be re-run from inside the VM.

Changes were made to the testsuite robot scripts for   instantiateDemoVFWCL robot flows to fix changes in the customer name/stack name generation to match the jenkins job setup for closed loop.

These were a side affect of the El Alto refactoring for python 2.7/3 migration that hadnt been detected in the previous test cases due to the need for unique Naming requirements in the jenkins jobs.

Shakedown consistent of creating some temporary tags for stability72hrvLB, stability72hrvVG,stability72hrVFWCL to make sure each sub test ran successfully (including cleanup) in the environment before the jenkins job started with the higher level testsuite tag stability72hr that covers all three test types.

During shake down of the environment we exceeded the quota on key pairs again (a recurring problem due to testing in the environment where the keypair delete is not run after deleting the VMs).

We used the horizon portal to delete keypairs for a large set of the previous robot test runs using the common admin tenant to free up quota space which should be sufficent for the duration of the tests but we will delete key pairs during the run just in case if needed.

VNF Orchestration Tests

This test uses the onap-ci job "Project windriver-longevity-stability72hr" to automatically onboard, distribute and instantiate the ONAP opensource test VNFs vLB, vVG and vFWCL.

The scripts run validation tests after the install.

The scripts then delete the VNFs and cleans up the environment for the next run.

The script tests 72 hour stability test was a PASS.

Overall memory checks did not show any memory leaks.

Robot container has the highest memory utilization.

CPU at the node and port level stayed within norms.


Overall Success Rate for the onboard / instantiation tests was 94%.

Test failures were mostly due to issues with slowness in the Openstack Environment. Saturday/Sunday had fewer errors than Monday.

Tuning of the Mariadb-Galera server engine for Camunda may also be an issue (

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyOOM-2132
)


Overall Success Rate for the closed loop tests was 100%.


In comparison to Dublin this stability run was more stable and showed fewer tooling issues.

Like in Dublin the longevity tenant will continue to run these tests.



Onboard/Instantiate Tests

Image Added

Image Added





Setup

The integration-longevity tenant in Intel/Windriver environment was used for the 72 hour tests.

The onap-ci job for  "Project windriver-longevity-release-manual" was used for the deployment with the OOM and Integration branches set to elalto.

The deployment was fairly clean but there was an environment issue that required a few pods to be recycled by the normal k8 delete pod due to a what looked like a network blimp during the install.

We also hit the environment dhcp bug where the VMs would get an external dhcp address from a different network than openstack's dhcp. The symptom is not being able to log into the external IP of the VM.

This is solved by a force reboot of the VM from the horizon portal but unfortunately this prevents the installation of the demo VNF config files so the VM install script has to be re-run from inside the VM.


Changes were made to the testsuite robot scripts for   instantiateDemoVFWCL robot flows to fix changes in the customer name/stack name generation to match the jenkins job setup for closed loop.

These were a side affect of the El Alto refactoring for python 2.7/3 migration that hadnt been detected in the previous test cases due to the need for unique Naming requirements in the jenkins jobs.


Shakedown consistent of creating some temporary tags for stability72hrvLB, stability72hrvVG,stability72hrVFWCL to make sure each sub test ran successfully (including cleanup) in the environment before the jenkins job started with the higher level testsuite tag stability72hr that covers all three test types.


During shake down of the environment we exceeded the quota on key pairs again (a recurring problem due to testing in the environment where the keypair delete is not run after deleting the VMs).

We used the horizon portal to delete keypairs for a large set of the previous robot test runs using the common admin tenant to free up quota space which should be sufficent for the duration of the tests but we will delete key pairs during the run just in case if needed.



VNF Orchestration Tests

This test uses the onap-ci job "Project windriver-longevity-stability72hr" to automatically onboard, distribute and instantiate the ONAP opensource test VNFs vLB, vVG and vFWCL.

The scripts run validation tests after the install.

The scripts then delete the VNFs and cleans up the environment for the next run.

The script tests AAF, DMaaP, SDC, VID, AAI, SO, SDNC, APPC with the open source VNFs.

...

Each test run generates over 500 MB of data on the test through robot framework.


Test #CommentMessage

Test start #243 1 PM Oct 12
245Validate vServer in testsuite HeatBridge needed to wait for the AAI index update. Wrapped this step in a  Wait For Keyword Successpost response: {"requestError":{"serviceException":{"messageId":"SVC3001","text":"Resource not found for %1 using id %2 (msg=%3) (ec=%4)","variables":["POST Search","getNamedQueryResponse","Node Not Found:No Node of type vserver found for properties","ERR.5.4.6114"]}}}
260

Environment Issue/Test tool issue.

robot test script created  conflicting data


Received failure response
Tooling or SO to Openstack interface transient


Received failure response from so {"request":{"requestId":"79264729-04ab-4738-a27d-29013c59218c","startTime":"Sun, 13 Oct 2019 09:38:20 GMT","finishTime":"Sun, 13 Oct 2019 09:39:14 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"VfwclVfwsnk0f6a8e47E64e..base_vfw..module-0","modelInvariantId":"e994097b-6285-49e1-a87c-76ba6e0371ab","modelType":"vfModule","modelName":"VfwclVfwsnk0f6a8e47E64e..base_vfw..module-0","modelVersion":"1","modelCustomizationUuid":"6ce786ef-31e8-4f00-bdb4-1c66f54eaffd","modelVersionId":"72f56293-fbf2-49fa-bb13-1df8f5f88548","modelCustomizationId":"6ce786ef-31e8-4f00-bdb4-1c66f54eaffd","modelUuid":"72f56293-fbf2-49fa-bb13-1df8f5f88548","modelInvariantUuid":"e994097b-6285-49e1-a87c-76ba6e0371ab","modelInstanceName":"VfwclVfwsnk0f6a8e47E64e..base_vfw..module-0"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vFWCLvFWSNK_031aaae1_0","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"fc4a3aac-e15e-4cf2-b85c-93eee3cdf3cc","modelInfo":{"modelInvariantId":"ed6ca1d8-cf38-455b-bb0a-75ae84d51715","modelType":"service","modelName":"vFWCL 2019-10-13 09:29:","modelVersion":"1.0","modelVersionId":"1c3dece0-945e-4f38-b5d2-f1d3fe7579e1","modelUuid":"1c3dece0-945e-4f38-b5d2-f1d3fe7579e1","modelInvariantUuid":"ed6ca1d8-cf38-455b-bb0a-75ae84d51715"}}},{"relatedInstance":{"instanceId":"d4cc80c3-367c-4de2-8dd2-52904466b60a","modelInfo":{"modelCustomizationName":"vFWCL_vFWSNK 0f6a8e47-e64e 0","modelInvariantId":"dcbe3ca3-b9c3-4042-a06f-5ad83f1be089","modelType":"vnf","modelName":"vFWCL_vFWSNK 0f6a8e47-e64e","modelVersion":"1.0","modelCustomizationUuid":"9eaff9be-ac20-4872-9804-7bd45515a351","modelVersionId":"2de4b9dd-b6d6-4822-92c0-670c9329557f","modelCustomizationId":"9eaff9be-ac20-4872-9804-7bd45515a351","modelUuid":"2de4b9dd-b6d6-4822-92c0-670c9329557f","modelInvariantUuid":"dcbe3ca3-b9c3-4042-a06f-5ad83f1be089","modelInstanceName":"vFWCL_vFWSNK 0f6a8e47-e64e 0"}}}],"cloudConfiguration":{"tenantId":"28481f6939614cfd83e6767a0e039bcc","cloudOwner":"CloudOwner","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true,"testApi":"VNF_API"}},"instanceReferences":{"serviceInstanceId":"fc4a3aac-e15e-4cf2-b85c-93eee3cdf3cc","vnfInstanceId":"d4cc80c3-367c-4de2-8dd2-52904466b60a","vfModuleInstanceName":"Vfmodule_Ete_vFWCLvFWSNK_031aaae1_0","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"STATUS: Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF org.onap.so.openstack.utils.StackCreationException: Stack Creation Failed Openstack Status: CREATE_FAILED Status Reason: Resource CREATE failed: Conflict: resources.vsn_0_onap_private_port_0: IP address 10.0.235.102 already allocated in subnet 4ed99c09-aed6-4eca-8f94-48357ab4e5d1\nNeutron server returns request_ids: ['req-f60a93ff-ecbf-4c5e-b149-8ebdf64e38f2'] , Rollback of Stack Creation completed with status: DELETE_COMPLETE Status Reason: Stack DELETE completed successfully' rolledBack='true'","percentProgress":100,"timestamp":"Sun, 13 Oct 2019 09:39:14 GMT"}}}

Test Status : #261 7 AM Oct 13

No left over VMs or Stacks from delete

Docker-data-nfs at 21% of available capacity

robot container:   10.0.0.4:/dockerdata-nfs/dev-robot/robot/logs 162420736 33509376 128894976 21% /share/logs.


17 keypairs under demo account


Environment Spot Check when tests are not running look okay.

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2240m 28% 12448Mi 78%
long-k8s-02 1367m 17% 13446Mi 84%
long-k8s-03 505m 6% 9422Mi 59%
long-k8s-04 677m 8% 12128Mi 76%
long-k8s-05 1733m 21% 12895Mi 80%
long-k8s-06 1300m 16% 14170Mi 88%
long-k8s-07 1171m 14% 12040Mi 75%
long-k8s-08 3019m 37% 13843Mi 86%
long-k8s-09 4046m 50% 11023Mi 69%
long-orch-1 174m 8% 1369Mi 35%
long-orch-2 169m 8% 1305Mi 33%
long-orch-3 284m 14% 1190Mi 30%


root@long-nfs:/home/ubuntu# kubectl -n onap top pods | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2961m 713Mi 
dev-uui-uui-server-6cf5b6bb7f-2g88z 1004m 236Mi 
dev-oof-music-tomcat-98645f744-bdvp7 782m 262Mi 
dev-oof-music-cassandra-0 565m 761Mi 
dev-consul-consul-7556c76b57-rh7rs 497m 29Mi 
dev-oof-oof-has-controller-755d446d46-7hzb4 449m 144Mi 
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 385m 1096Mi 
dev-contrib-awx-0 258m 1375Mi 
dev-oof-music-cassandra-2 221m 766Mi 
dev-portal-portal-cassandra-75479b4646-8srbs 213m 2787Mi

Test Status: #267 12:00 PM Oct 13

/dev/vda1 162420480 36636868 125767228 23% /

No left over VMs or Stacks from previous runs

RegionOne_ONAP-NF_20191013T150300143Z_olc-key_PlYL style keypairs added in the morning. Up to 27 keypairs

Closed Loop Tests

This test uses the onap-ci job "Project windriver-longevity-vfwclosedloop".

...


268


Same signature as #260

,"requestStatus":{"requestState":"FAILED","statusMessage":"STATUS: Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF org.onap.so.openstack.utils.StackCreationException: Stack Creation Failed Openstack Status: CREATE_FAILED Status Reason: Resource CREATE failed: Conflict: resources.vdns_0_onap_private_port_0: IP address 10.0.236.25 already allocated in subnet 4ed99c09-aed6-4eca-8f94-48357ab4e5d1\nNeutron server returns request_ids: ['req-2a9b2ed0-0502-4377-b209-59f36a27d8dd'] , Rollback of Stack Creation completed with status: DELETE_COMPLETE Status Reason: Stack DELETE completed successfully'

Test Status #271  4:00 PM Oct 13

/dev/vda1 162420480 40495064 121909032 25% /


No left over VMs or Stacks

Up to 33 keypairs


root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 
long-k8s-01 2388m 29% 12455Mi 78% 
long-k8s-02 1439m 17% 13371Mi 83% 
long-k8s-03 502m 6% 9342Mi 58% 
long-k8s-04 836m 10% 12083Mi 75% 
long-k8s-05 1508m 18% 12989Mi 81% 
long-k8s-06 1238m 15% 14234Mi 89% 
long-k8s-07 1264m 15% 12020Mi 75% 
long-k8s-08 1275m 15% 13968Mi 87% 
long-k8s-09 4309m 53% 11170Mi 70% 
long-orch-1 190m 9% 1384Mi 36% 
long-orch-2 167m 8% 1410Mi 36% 
long-orch-3 300m 15% 1224Mi 31% 



root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2976m 700Mi 
dev-uui-uui-server-6cf5b6bb7f-2g88z 1005m 237Mi 
dev-oof-music-tomcat-98645f744-bdvp7 728m 262Mi 
dev-consul-consul-7556c76b57-rh7rs 621m 27Mi 
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 578m 1118Mi 
dev-cassandra-cassandra-1 565m 1839Mi 
dev-oof-oof-has-controller-755d446d46-7hzb4 476m 145Mi 
dev-sdc-sdc-onboarding-be-75fbbb65c9-d95zh 342m 752Mi 
dev-contrib-awx-0 303m 1438Mi 
dev-cassandra-cassandra-2 286m 1809Mi


276Same signature as #260questParameters":{"usePreload":true,"testApi":"VNF_API"}},"instanceReferences":{"serviceInstanceId":"7374c399-e4af-4cc8-81b3-cb0ff810ac7c","vnfInstanceId":"06888576-bd1b-4b30-b27f-3b61a0898bee","vfModuleInstanceName":"Vfmodule_Ete_vFWCLvPKG_3cd57462_1","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"STATUS: Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF org.onap.so.openstack.utils.StackCreationException: Stack Creation Failed Openstack Status: CREATE_FAILED Status Reason: Resource CREATE failed: Conflict: resources.vpg_0_onap_private_port_0: IP address 10.0.158.103 already allocated in subnet 4ed99c09-aed6-4eca-8f94-48357ab4e5d1\nNeutron server returns request_ids: ['req-3e4fd376-8698-4211-95a1-eb1312a71c28'] , Rollback of Stack Creation completed with status: DELETE_COMPLETE Status Reason: Stack DELETE completed successfully' rolledBack='true'","percentProgress":100,"timestamp":"Mon, 14 Oct 2019 01:35:58 GMT"}}

Test Status   #276  10:00 PM Oct 13

No stranded VMs or Stacks


robot  log storage up to 28% usage

10.0.0.4:/dockerdata-nfs/dev-robot/robot/logs 162420736 44839936 117564416 28% /share/logs

root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2658m 33% 12480Mi 78%
long-k8s-02 296m 3% 13336Mi 83%
long-k8s-03 514m 6% 9328Mi 58%
long-k8s-04 694m 8% 12168Mi 76%
long-k8s-05 1436m 17% 13196Mi 82%
long-k8s-06 1154m 14% 14280Mi 89%
long-k8s-07 1288m 16% 12068Mi 75%
long-k8s-08 3080m 38% 14026Mi 87%
long-k8s-09 4068m 50% 11226Mi 70%
long-orch-1 197m 9% 1331Mi 34%
long-orch-2 163m 8% 1393Mi 36%
long-orch-3 432m 21% 1203Mi 31%
root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2968m 700Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1002m 236Mi
dev-robot-robot-5f67d595b6-cl6bc 830m 4327Mi
dev-oof-music-cassandra-0 734m 763Mi
dev-oof-music-tomcat-98645f744-bdvp7 717m 262Mi
dev-consul-consul-7556c76b57-rh7rs 673m 28Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 460m 146Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 317m 1139Mi
dev-contrib-awx-0 239m 1380Mi
dev-oof-music-cassandra-2 202m 770Mi

281Same signature as #260vFWCL Heat address 10.0.241.102 already allocated
282Same signature as #260vFWCL Heat vfmodule duplicate
285Same signature as #260vVWCL Duplicate IP address 10.0.187.102
286Same signature as #260

Potential cause is conflict with vfwclosedloop vnf in preload / test tool data

10.0.251.101, 102, 103 are used by vfwclosed loop but not excluded from test data for instantiate tests.

Still need to look at ip address removal in openstack to see if ip address aging is affecting the tests


2019-10-14T10:35:21.610Z||org.onap.so.adapters.vnf.VnfAdapterRest - Create VfModule enter inside VnfAdapterRest: {"createVfModuleRequest":{"messageId":"94679b5b-a360-4f78-a9c3-d097e1b2ec25-1571049321389","skipAAI":true,"notificationUrl":"http://so-bpmn-infra.onap:8081/mso/WorkflowMessage/VNFAResponse/94679b5b-a360-4f78-a9c3-d097e1b2ec25-1571049321389",

....

--------------------------------------
2019-10-14T10:35:22.014Z|94679b5b-a360-4f78-a9c3-d097e1b2ec25|org.onap.so.openstack.utils.MsoCommonUtils - Config values RetryDelay:5 RetryCount:3 RetryCodes:504 ResponseCode:404
2019-10-14T10:35:22.015Z|94679b5b-a360-4f78-a9c3-d097e1b2ec25|org.onap.so.openstack.utils.MsoHeatUtils - Error in Query Stack
com.woorea.openstack.base.client.OpenStackResponseException: Not Found
at com.woorea.openstack.connector.HttpClientConnector.request(HttpClientConnector.java:186)
at com.woorea.openstack.base.client.OpenStackClient.request(OpenStackClient.java:71)
at com.woorea.openstack.base.client.OpenStackClient.execute(OpenStackClient.java:87)

...

2019-10-14T10:35:22.015Z|94679b5b-a360-4f78-a9c3-d097e1b2ec25|org.onap.so.openstack.utils.MsoHeatUtils - queryHeatStack - stack not found: Vfmodule_Ete_vFWCLvFWSNK_b0857107_0


2019-10-14T10:35:22.021Z||org.onap.so.adapters.vnf.VnfAdapterRest - Create VfModule enter inside VnfAdapterRest: {"createVfModuleRequest":{"messageId":"94679b5b-a360-4f78-a9c3-d097e1b2ec25-1571049321778","skipAAI":true,"notificationUrl":"http://so-bpmn-infra.onap:8081/mso/WorkflowMessage/VNFAResponse/94679b5b-a360-4f78-a9c3-d097e1b2ec25-1571049321778",

Looks like openstack does not respond with status correctly after a vfModule create.


Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keySO-2447

vFWCL Duplicate IP address 10.0.251.103


Test Status: 8 AM Oct 14

/dev/vda1 162420480 52463904 109940192 33% /


root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2667m 33% 12582Mi 78%
long-k8s-02 1447m 18% 13325Mi 83%
long-k8s-03 554m 6% 9325Mi 58%
long-k8s-04 654m 8% 12090Mi 75%
long-k8s-05 1419m 17% 13221Mi 82%
long-k8s-06 1077m 13% 14284Mi 89%
long-k8s-07 1350m 16% 12302Mi 77%
long-k8s-08 2907m 36% 14036Mi 88%
long-k8s-09 4157m 51% 11142Mi 69%
long-orch-1 207m 10% 1424Mi 37%
long-orch-2 198m 9% 1449Mi 37%
long-orch-3 320m 16% 1253Mi 32%


root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2985m 673Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1005m 236Mi
dev-oof-music-tomcat-98645f744-bdvp7 712m 263Mi
dev-consul-consul-7556c76b57-rh7rs 659m 29Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 548m 145Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 467m 1141Mi
dev-oof-music-cassandra-0 412m 766Mi
dev-contrib-awx-0 322m 1378Mi
dev-oof-music-cassandra-2 258m 772Mi
dev-cassandra-cassandra-0 206m 1813Mi

288Same problem as #260

vVG : Stack Vfmodule_Ete_vVG_b6aa2967_0 already exists in Tenant

duplicate stack name instead of duplicate ip address but same problem

289Heatbridge Validation

AAI query on reverse heat bridge

Testsuite should wrap in Wait For Keyword Success instead of justh query - cassandra replication delay


290Same problem as #260vFWCL:  vfmodule name duplicate

Test Status 1 PM Oct 14

/dev/vda1 162420480 56555456 105848640 35% /


No stranded VMs or Stacks

36 keypairs


(during active instantiate phase)

root@long-nfs:/tmp# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2823m 35% 12580Mi 78%
long-k8s-02 1540m 19% 13353Mi 83%
long-k8s-03 584m 7% 9364Mi 58%
long-k8s-04 653m 8% 12108Mi 75%
long-k8s-05 1781m 22% 13207Mi 82%
long-k8s-06 1471m 18% 14279Mi 89%
long-k8s-07 867m 10% 12362Mi 77%
long-k8s-08 3024m 37% 14075Mi 88%
long-k8s-09 4352m 54% 11064Mi 69%
long-orch-1 192m 9% 1359Mi 35%
long-orch-2 146m 7% 1386Mi 36%
long-orch-3 301m 15% 1159Mi 30%


root@long-nfs:/tmp# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2936m 668Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1004m 236Mi
dev-oof-music-tomcat-98645f744-bdvp7 670m 263Mi
dev-consul-consul-7556c76b57-rh7rs 542m 69Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 525m 145Mi
dev-oof-music-cassandra-0 443m 768Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 383m 1158Mi
dev-cassandra-cassandra-1 265m 1847Mi
dev-robot-robot-5f67d595b6-cl6bc 224m 4311Mi
dev-contrib-awx-0 207m 1378Mi

292Same problem as #260

vFWCL :  IP address 10.0.227.101 already allocated in subnet

Latest analysis indicates it may be a problem with shared mariadb-galera server.cnf that is not providing the right locking to Camunda  under load.

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyOOM-2132


Test Status  5 PM Oct 14

/dev/vda1 162420480 59795972 102608124 37% /


No stranded VMs or Stacks


root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2902m 36% 12577Mi 78%
long-k8s-02 1508m 18% 13292Mi 83%
long-k8s-03 546m 6% 9337Mi 58%
long-k8s-04 702m 8% 12079Mi 75%
long-k8s-05 1621m 20% 13283Mi 83%
long-k8s-06 1239m 15% 14345Mi 90%
long-k8s-07 860m 10% 12420Mi 77%
long-k8s-08 3142m 39% 14058Mi 88%
long-k8s-09 4366m 54% 11121Mi 69%
long-orch-1 167m 8% 1453Mi 37%
long-orch-2 182m 9% 1476Mi 38%
long-orch-3 367m 18% 1258Mi 32%


root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2982m 699Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1008m 235Mi
dev-oof-music-tomcat-98645f744-bdvp7 762m 263Mi
dev-consul-consul-7556c76b57-rh7rs 671m 46Mi
dev-oof-music-cassandra-0 671m 770Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 546m 145Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 431m 1163Mi
dev-contrib-awx-0 198m 1378Mi
dev-oof-music-cassandra-2 193m 773Mi
dev-oof-music-cassandra-1 166m 778Mi


304Same problem as #260vFWCL : IP address 192.168.10.200 already allocated in subnet

Test Status 8 AM Oct 15


/dev/vda1 162420480 72287180 90116916 45% /


No stranded VMs or Stacks



root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 3023m 37% 12621Mi 79%
long-k8s-02 1578m 19% 13298Mi 83%
long-k8s-03 548m 6% 9344Mi 58%
long-k8s-04 643m 8% 12039Mi 75%
long-k8s-05 1891m 23% 13415Mi 84%
long-k8s-06 1739m 21% 14462Mi 90%
long-k8s-07 700m 8% 12654Mi 79%
long-k8s-08 3355m 41% 14055Mi 88%
long-k8s-09 4288m 53% 10840Mi 68%
long-orch-1 345m 17% 1486Mi 38%
long-orch-2 196m 9% 1476Mi 38%
long-orch-3 391m 19% 1327Mi 34%


root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2980m 606Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1010m 235Mi
dev-oof-music-tomcat-98645f744-bdvp7 696m 263Mi
dev-consul-consul-7556c76b57-rh7rs 680m 28Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 628m 145Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 558m 1193Mi
dev-oof-music-cassandra-0 413m 771Mi
dev-oof-music-cassandra-1 370m 880Mi
dev-contrib-awx-0 280m 1387Mi
dev-cassandra-cassandra-1 239m 1861Mi

312Looks like Openstack Problem on Querying for Heat Stack

vVVG


On Delete Phase

Openstack Error:  {'code': 404, 'error': {'message': 'The Stack (Vfmodule_Ete_vVG_f9488c12_0) could not be found.', 'traceback': None, 'type': 'EntityNotFound'}, 'explanation': 'The resource could not be found.', 'title': 'Not Found'}




Test Status 2 PM Oct 15

/dev/vda1 162420480 76216720 86187376 47% /


No stranded VMs or Stacks


root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2366m 29% 12585Mi 78%
long-k8s-02 1554m 19% 13226Mi 82%
long-k8s-03 561m 7% 9275Mi 58%
long-k8s-04 670m 8% 12013Mi 75%
long-k8s-05 1830m 22% 13307Mi 83%
long-k8s-06 1120m 14% 14514Mi 91%
long-k8s-07 720m 9% 12719Mi 79%
long-k8s-08 3131m 39% 14015Mi 87%
long-k8s-09 4366m 54% 10872Mi 68%
long-orch-1 155m 7% 1425Mi 37%
long-orch-2 213m 10% 1379Mi 35%
long-orch-3 358m 17% 1267Mi 32%


root@long-nfs:~/oom/kubernetes/robot# kubectl -n onap top pods | sort -k3,3nr | head -20

dev-robot-robot-5f67d595b6-cl6bc 21m 4164Mi

dev-portal-portal-cassandra-75479b4646-8srbs 168m 2791Mi

dev-appc-appc-2 30m 2677Mi

dev-appc-appc-0 146m 2672Mi

dev-log-log-elasticsearch-5b99986585-fpc27 39m 2633Mi

dev-appc-appc-1 32m 2389Mi

dev-policy-pap-6bbfb7955-mnr26 7m 2042Mi

dev-sdc-sdc-be-f8c9bddf6-p6ml8 67m 1963Mi

dev-cassandra-cassandra-1 420m 1880Mi

dev-sdnc-sdnc-1 26m 1875Mi

dev-sdnc-sdnc-2 50m 1833Mi

dev-cassandra-cassandra-0 132m 1821Mi

dev-cassandra-cassandra-2 94m 1814Mi

dev-mariadb-galera-mariadb-galera-1 12m 1737Mi

dev-aai-aai-elasticsearch-665b4859c8-p295f 2m 1446Mi

dev-contrib-awx-0 312m 1387Mi

dev-vid-vid-7776f457d6-56msw 12m 1349Mi

dev-clamp-clamp-dash-es-6ff9cf4cf-5mkxc 7m 1344Mi

dev-dcaegen2-dcae-cloudify-manager-f476f7d59-2jrrw 167m 1344Mi

dev-pomba-pomba-elasticsearch-5d5f7d544b-qt29p 17m 1310Mi



root@long-nfs:~/oom/kubernetes/robot# kubectl -n onap top pods | sort -k2,2nr | head -20

dev-log-log-logstash-7d6c57f746-2b9gb 2998m 606Mi

dev-uui-uui-server-6cf5b6bb7f-2g88z 1006m 236Mi

dev-consul-consul-7556c76b57-rh7rs 839m 33Mi

dev-oof-music-tomcat-98645f744-bdvp7 766m 263Mi

dep-dcae-tca-analytics-5fdcc57cc9-78f9d 597m 1192Mi

dev-oof-music-cassandra-1 589m 782Mi

dev-oof-music-cassandra-0 526m 772Mi

dev-oof-oof-has-controller-755d446d46-7hzb4 471m 145Mi

dev-cassandra-cassandra-1 307m 1874Mi

dev-contrib-awx-0 228m 1387Mi

dev-portal-portal-cassandra-75479b4646-8srbs 190m 2796Mi

dev-oof-music-cassandra-2 187m 778Mi

dev-appc-appc-0 149m 2672Mi

dev-sdnc-sdnc-0 146m 1115Mi

dev-dcaegen2-dcae-cloudify-manager-f476f7d59-2jrrw 144m 1378Mi

dev-cassandra-cassandra-0 126m 1821Mi

dev-vfc-vfc-huawei-vnfm-driver-775cc4fd4d-6b9pd 112m 637Mi

dev-dmaap-message-router-0 111m 488Mi

dev-msb-msb-consul-744f6ccbdd-9xjbh 102m 47Mi

dev-cassandra-cassandra-2 92m 1813Mi






==============================================================================

Testsuites

==============================================================================

Testsuites.Health-Check :: Test that ONAP components are available via basi...

==============================================================================

Basic A&AI Health Check | PASS |

------------------------------------------------------------------------------

Basic AAF Health Check | PASS |

------------------------------------------------------------------------------

Basic AAF SMS Health Check | PASS |

------------------------------------------------------------------------------

Basic APPC Health Check | PASS |

------------------------------------------------------------------------------

Basic CLI Health Check | PASS |

------------------------------------------------------------------------------

Basic CLAMP Health Check | PASS |

------------------------------------------------------------------------------

Basic DCAE Health Check | PASS |

------------------------------------------------------------------------------

Basic DMAAP Data Router Health Check | PASS |

------------------------------------------------------------------------------

Basic DMAAP Message Router Health Check | PASS |

------------------------------------------------------------------------------

Basic DMAAP Message Router PubSub Health Check | PASS |

------------------------------------------------------------------------------

Basic DMAAP Bus Controller Health Check With Basic Auth | PASS |

------------------------------------------------------------------------------

Basic Log Elasticsearch Health Check | PASS |

------------------------------------------------------------------------------

Basic Log Kibana Health Check | PASS |

------------------------------------------------------------------------------

Basic Log Logstash Health Check | PASS |

------------------------------------------------------------------------------

Basic Microservice Bus Health Check | PASS |

------------------------------------------------------------------------------

Basic Multicloud API Health Check | PASS |

------------------------------------------------------------------------------

Basic Multicloud-pike API Health Check | PASS |

------------------------------------------------------------------------------

Basic Multicloud-starlingx API Health Check | PASS |

------------------------------------------------------------------------------

Basic Multicloud-titanium_cloud API Health Check | PASS |

------------------------------------------------------------------------------

Basic Multicloud-vio API Health Check | PASS |

------------------------------------------------------------------------------

Basic Multicloud-k8s API Health Check | PASS |

------------------------------------------------------------------------------

Basic OOF-Homing Health Check | PASS |

------------------------------------------------------------------------------

Basic OOF-SNIRO Health Check | PASS |

------------------------------------------------------------------------------

Basic OOF-CMSO Health Check | PASS |

------------------------------------------------------------------------------

Basic Policy Health Check | PASS |

------------------------------------------------------------------------------

Basic Pomba AAI-context-builder Health Check | PASS |

------------------------------------------------------------------------------

Basic Pomba SDC-context-builder Health Check | PASS |

------------------------------------------------------------------------------

Basic Pomba Network-discovery-context-builder Health Check | PASS |

------------------------------------------------------------------------------

Basic Pomba Service-Decomposition Health Check | PASS |

------------------------------------------------------------------------------

Basic Pomba Network-Discovery-MicroService Health Check | PASS |

------------------------------------------------------------------------------

Basic Pomba Pomba-Kibana Health Check | PASS |

------------------------------------------------------------------------------

Basic Pomba Elastic-Search Health Check | PASS |

------------------------------------------------------------------------------

Basic Pomba Sdnc-Context-Builder Health Check | PASS |

------------------------------------------------------------------------------

Basic Pomba Context-Aggregator Health Check | PASS |

------------------------------------------------------------------------------

Basic Portal Health Check | PASS |

------------------------------------------------------------------------------

Basic SDC Health Check (DMaaP:UP)

| PASS |

------------------------------------------------------------------------------

Basic SDNC Health Check | PASS |

------------------------------------------------------------------------------

Basic SO Health Check | PASS |

------------------------------------------------------------------------------

Basic UseCaseUI API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC catalog API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC emsdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC gvnfmdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC huaweivnfmdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC jujuvnfmdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC multivimproxy API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC nokiav2driver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC nslcm API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC resmgr API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC vnflcm API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC vnfmgr API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC vnfres API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC workflow API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC ztesdncdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VFC ztevnfmdriver API Health Check | PASS |

------------------------------------------------------------------------------

Basic VID Health Check | PASS |

------------------------------------------------------------------------------

Basic VNFSDK Health Check | PASS |

------------------------------------------------------------------------------

Basic Holmes Rule Management API Health Check | PASS |

------------------------------------------------------------------------------

Basic Holmes Engine Management API Health Check | PASS |

------------------------------------------------------------------------------

Basic Multicloud-fcaps API Health Check | PASS |

------------------------------------------------------------------------------

Basic Modeling genericparser API Health Check | FAIL |

502 != 200

------------------------------------------------------------------------------

Basic CDS Health Check | PASS |

------------------------------------------------------------------------------

Testsuites.Health-Check :: Test that ONAP components are available... | FAIL |

61 critical tests, 60 passed, 1 failed

61 tests total, 60 passed, 1 failed

==============================================================================

Testsuites | FAIL |

61 critical tests, 60 passed, 1 failed

61 tests total, 60 passed, 1 failed

==============================================================================

Output: /share/logs/0226_ete_helmlist/output.xml

Log: /share/logs/0226_ete_helmlist/log.html

Report: /share/logs/0226_ete_helmlist/report.html



Modeling Parser is a known issue.




Closed Loop Tests

This test uses the onap-ci job "Project windriver-longevity-vfwclosedloop".

The test uses the robot test script "demo-k8s.sh vfwclosedloop ". The script sets the number of streams on the vPacket Generator to 10 , waits for the change from 10 set sreams to 5 streams by the control loop then sets the stream to 1 and again waits for the 5 streams.

Success tests the loop from VNF through  DCAE, DMaaP, Policy, AAI , AAF and APPC.

The tests start with #1595 on October 12 at 4:00 PM EST

...

the stream to 1 and again waits for the 5 streams.

Success tests the loop from VNF through  DCAE, DMaaP, Policy, AAI , AAF and APPC.


The tests start with #1595 on October 12 at 4:00 PM EST


Test #Comment             Message                                                                               

Test Start    #1595   4 PM Oct 12

Test Status: #1610   7 AM Oct 13No issues. No failed tests

Test Status: #1615 12 PM Oct 13No issues. No failed tests

Test Status  #1620   5 PM Oct 13No issues. No failed tests

Test Status #1625  10 PM Oct 13No issues. No failed tests

Test Status #1635  8 AM  Oct 14No issues. No failed tests

Test Status #1640  1 PM  Oct 14No issues. No failed tests

Test Status #1644  5 PM Oct 14

No issues. No failed tests.

Comparing El Alto to Dublin we see that the average loop response time is shorter.

for El Alto (2:17 minutes) vs Dublin (3:19 minutes)

This is likely because the TCA polling interval was reduced in El Alto to speed up the loop in recognition that the VES reporter default configuration in the ONAP test VNFs was set aggressively to emit status every 10 seconds.


Test Status #1659  8 AM Oct 15No issues. No failed tests.





Summary

To be completed after the test run

...