...

Test #

Comment

Message

Test start #243 1 PM Oct 12

245

Validate vServer in testsuite HeatBridge needed to wait for the AAI index update. Wrapped this step in a Wait For Keyword Success

post response: {"requestError":{"serviceException":{"messageId":"SVC3001","text":"Resource not found for %1 using id %2 (msg=%3) (ec=%4)","variables":["POST Search","getNamedQueryResponse","Node Not Found:No Node of type vserver found for properties","ERR.5.4.6114"]}}}

260

Tooling or SO to Openstack interface transient

Received failure response from so {"request":{"requestId":"79264729-04ab-4738-a27d-29013c59218c","startTime":"Sun, 13 Oct 2019 09:38:20 GMT","finishTime":"Sun, 13 Oct 2019 09:39:14 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"VfwclVfwsnk0f6a8e47E64e..base_vfw..module-0","modelInvariantId":"e994097b-6285-49e1-a87c-76ba6e0371ab","modelType":"vfModule","modelName":"VfwclVfwsnk0f6a8e47E64e..base_vfw..module-0","modelVersion":"1","modelCustomizationUuid":"6ce786ef-31e8-4f00-bdb4-1c66f54eaffd","modelVersionId":"72f56293-fbf2-49fa-bb13-1df8f5f88548","modelCustomizationId":"6ce786ef-31e8-4f00-bdb4-1c66f54eaffd","modelUuid":"72f56293-fbf2-49fa-bb13-1df8f5f88548","modelInvariantUuid":"e994097b-6285-49e1-a87c-76ba6e0371ab","modelInstanceName":"VfwclVfwsnk0f6a8e47E64e..base_vfw..module-0"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vFWCLvFWSNK_031aaae1_0","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"fc4a3aac-e15e-4cf2-b85c-93eee3cdf3cc","modelInfo":{"modelInvariantId":"ed6ca1d8-cf38-455b-bb0a-75ae84d51715","modelType":"service","modelName":"vFWCL 2019-10-13 09:29:","modelVersion":"1.0","modelVersionId":"1c3dece0-945e-4f38-b5d2-f1d3fe7579e1","modelUuid":"1c3dece0-945e-4f38-b5d2-f1d3fe7579e1","modelInvariantUuid":"ed6ca1d8-cf38-455b-bb0a-75ae84d51715"}}},{"relatedInstance":{"instanceId":"d4cc80c3-367c-4de2-8dd2-52904466b60a","modelInfo":{"modelCustomizationName":"vFWCL_vFWSNK 0f6a8e47-e64e 0","modelInvariantId":"dcbe3ca3-b9c3-4042-a06f-5ad83f1be089","modelType":"vnf","modelName":"vFWCL_vFWSNK 0f6a8e47-e64e","modelVersion":"1.0","modelCustomizationUuid":"9eaff9be-ac20-4872-9804-7bd45515a351","modelVersionId":"2de4b9dd-b6d6-4822-92c0-670c9329557f","modelCustomizationId":"9eaff9be-ac20-4872-9804-7bd45515a351","modelUuid":"2de4b9dd-b6d6-4822-92c0-670c9329557f","modelInvariantUuid":"dcbe3ca3-b9c3-4042-a06f-5ad83f1be089","modelInstanceName":"vFWCL_vFWSNK 0f6a8e47-e64e 0"}}}],"cloudConfiguration":{"tenantId":"28481f6939614cfd83e6767a0e039bcc","cloudOwner":"CloudOwner","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true,"testApi":"VNF_API"}},"instanceReferences":{"serviceInstanceId":"fc4a3aac-e15e-4cf2-b85c-93eee3cdf3cc","vnfInstanceId":"d4cc80c3-367c-4de2-8dd2-52904466b60a","vfModuleInstanceName":"Vfmodule_Ete_vFWCLvFWSNK_031aaae1_0","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"STATUS: Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF org.onap.so.openstack.utils.StackCreationException: Stack Creation Failed Openstack Status: CREATE_FAILED Status Reason: Resource CREATE failed: Conflict: resources.vsn_0_onap_private_port_0: IP address 10.0.235.102 already allocated in subnet 4ed99c09-aed6-4eca-8f94-48357ab4e5d1\nNeutron server returns request_ids: ['req-f60a93ff-ecbf-4c5e-b149-8ebdf64e38f2'] , Rollback of Stack Creation completed with status: DELETE_COMPLETE Status Reason: Stack DELETE completed successfully' rolledBack='true'","percentProgress":100,"timestamp":"Sun, 13 Oct 2019 09:39:14 GMT"}}}

Test Status : #261 7 AM Oct 13

No left over VMs or Stacks from delete

Docker-data-nfs at 21% of available capacity

robot container: 10.0.0.4:/dockerdata-nfs/dev-robot/robot/logs 162420736 33509376 128894976 21% /share/logs.

17 keypairs under demo account

Environment Spot Check when tests are not running look okay.

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2240m 28% 12448Mi 78%
long-k8s-02 1367m 17% 13446Mi 84%
long-k8s-03 505m 6% 9422Mi 59%
long-k8s-04 677m 8% 12128Mi 76%
long-k8s-05 1733m 21% 12895Mi 80%
long-k8s-06 1300m 16% 14170Mi 88%
long-k8s-07 1171m 14% 12040Mi 75%
long-k8s-08 3019m 37% 13843Mi 86%
long-k8s-09 4046m 50% 11023Mi 69%
long-orch-1 174m 8% 1369Mi 35%
long-orch-2 169m 8% 1305Mi 33%
long-orch-3 284m 14% 1190Mi 30%


root@long-nfs:/home/ubuntu# kubectl -n onap top pods | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2961m 713Mi 
dev-uui-uui-server-6cf5b6bb7f-2g88z 1004m 236Mi 
dev-oof-music-tomcat-98645f744-bdvp7 782m 262Mi 
dev-oof-music-cassandra-0 565m 761Mi 
dev-consul-consul-7556c76b57-rh7rs 497m 29Mi 
dev-oof-oof-has-controller-755d446d46-7hzb4 449m 144Mi 
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 385m 1096Mi 
dev-contrib-awx-0 258m 1375Mi 
dev-oof-music-cassandra-2 221m 766Mi 
dev-portal-portal-cassandra-75479b4646-8srbs 213m 2787Mi

Test Status: #267 12:00 PM Oct 13

/dev/vda1 162420480 36636868 125767228 23% /

No left over VMs or Stacks from previous runs

RegionOne_ONAP-NF_20191013T150300143Z_olc-key_PlYL style keypairs added in the morning. Up to 27 keypairs

268

Same signature as #260

,"requestStatus":{"requestState":"FAILED","statusMessage":"STATUS: Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF org.onap.so.openstack.utils.StackCreationException: Stack Creation Failed Openstack Status: CREATE_FAILED Status Reason: Resource CREATE failed: Conflict: resources.vdns_0_onap_private_port_0: IP address 10.0.236.25 already allocated in subnet 4ed99c09-aed6-4eca-8f94-48357ab4e5d1\nNeutron server returns request_ids: ['req-2a9b2ed0-0502-4377-b209-59f36a27d8dd'] , Rollback of Stack Creation completed with status: DELETE_COMPLETE Status Reason: Stack DELETE completed successfully'

Test Status #271 4:00 PM Oct 13

/dev/vda1 162420480 40495064 121909032 25% /

No left over VMs or Stacks

Up to 33 keypairs

root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 
long-k8s-01 2388m 29% 12455Mi 78% 
long-k8s-02 1439m 17% 13371Mi 83% 
long-k8s-03 502m 6% 9342Mi 58% 
long-k8s-04 836m 10% 12083Mi 75% 
long-k8s-05 1508m 18% 12989Mi 81% 
long-k8s-06 1238m 15% 14234Mi 89% 
long-k8s-07 1264m 15% 12020Mi 75% 
long-k8s-08 1275m 15% 13968Mi 87% 
long-k8s-09 4309m 53% 11170Mi 70% 
long-orch-1 190m 9% 1384Mi 36% 
long-orch-2 167m 8% 1410Mi 36% 
long-orch-3 300m 15% 1224Mi 31% 



root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2976m 700Mi 
dev-uui-uui-server-6cf5b6bb7f-2g88z 1005m 237Mi 
dev-oof-music-tomcat-98645f744-bdvp7 728m 262Mi 
dev-consul-consul-7556c76b57-rh7rs 621m 27Mi 
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 578m 1118Mi 
dev-cassandra-cassandra-1 565m 1839Mi 
dev-oof-oof-has-controller-755d446d46-7hzb4 476m 145Mi 
dev-sdc-sdc-onboarding-be-75fbbb65c9-d95zh 342m 752Mi 
dev-contrib-awx-0 303m 1438Mi 
dev-cassandra-cassandra-2 286m 1809Mi

276

Same signature as #260

questParameters":{"usePreload":true,"testApi":"VNF_API"}},"instanceReferences":{"serviceInstanceId":"7374c399-e4af-4cc8-81b3-cb0ff810ac7c","vnfInstanceId":"06888576-bd1b-4b30-b27f-3b61a0898bee","vfModuleInstanceName":"Vfmodule_Ete_vFWCLvPKG_3cd57462_1","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"STATUS: Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF org.onap.so.openstack.utils.StackCreationException: Stack Creation Failed Openstack Status: CREATE_FAILED Status Reason: Resource CREATE failed: Conflict: resources.vpg_0_onap_private_port_0: IP address 10.0.158.103 already allocated in subnet 4ed99c09-aed6-4eca-8f94-48357ab4e5d1\nNeutron server returns request_ids: ['req-3e4fd376-8698-4211-95a1-eb1312a71c28'] , Rollback of Stack Creation completed with status: DELETE_COMPLETE Status Reason: Stack DELETE completed successfully' rolledBack='true'","percentProgress":100,"timestamp":"Mon, 14 Oct 2019 01:35:58 GMT"}}

Test Status #276 10:00 PM Oct 13

No stranded VMs or Stacks

robot log storage up to 28% usage

10.0.0.4:/dockerdata-nfs/dev-robot/robot/logs 162420736 44839936 117564416 28% /share/logs

root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2658m 33% 12480Mi 78%
long-k8s-02 296m 3% 13336Mi 83%
long-k8s-03 514m 6% 9328Mi 58%
long-k8s-04 694m 8% 12168Mi 76%
long-k8s-05 1436m 17% 13196Mi 82%
long-k8s-06 1154m 14% 14280Mi 89%
long-k8s-07 1288m 16% 12068Mi 75%
long-k8s-08 3080m 38% 14026Mi 87%
long-k8s-09 4068m 50% 11226Mi 70%
long-orch-1 197m 9% 1331Mi 34%
long-orch-2 163m 8% 1393Mi 36%
long-orch-3 432m 21% 1203Mi 31%
root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2968m 700Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1002m 236Mi
dev-robot-robot-5f67d595b6-cl6bc 830m 4327Mi
dev-oof-music-cassandra-0 734m 763Mi
dev-oof-music-tomcat-98645f744-bdvp7 717m 262Mi
dev-consul-consul-7556c76b57-rh7rs 673m 28Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 460m 146Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 317m 1139Mi
dev-contrib-awx-0 239m 1380Mi
dev-oof-music-cassandra-2 202m 770Mi

281

Same signature as #260

vFWCL Heat address 10.0.241.102 already allocated

282

Same signature as #260

vFWCL Heat vfmodule duplicate

285

Same signature as #260

vVWCL Duplicate IP address 10.0.187.102

286

Same signature as #260

Potential cause is conflict with vfwclosedloop vnf in preload / test tool data

10.0.251.101, 102, 103 are used by vfwclosed loop but not excluded from test data for instantiate tests.

Still need to look at ip address removal in openstack to see if ip address aging is affecting the tests

2019-10-14T10:35:21.610Z||org.onap.so.adapters.vnf.VnfAdapterRest - Create VfModule enter inside VnfAdapterRest: {"createVfModuleRequest":{"messageId":"94679b5b-a360-4f78-a9c3-d097e1b2ec25-1571049321389","skipAAI":true,"notificationUrl":"http://so-bpmn-infra.onap:8081/mso/WorkflowMessage/VNFAResponse/94679b5b-a360-4f78-a9c3-d097e1b2ec25-1571049321389",

....

--------------------------------------
2019-10-14T10:35:22.014Z|94679b5b-a360-4f78-a9c3-d097e1b2ec25|org.onap.so.openstack.utils.MsoCommonUtils - Config values RetryDelay:5 RetryCount:3 RetryCodes:504 ResponseCode:404
2019-10-14T10:35:22.015Z|94679b5b-a360-4f78-a9c3-d097e1b2ec25|org.onap.so.openstack.utils.MsoHeatUtils - Error in Query Stack
com.woorea.openstack.base.client.OpenStackResponseException: Not Found
at com.woorea.openstack.connector.HttpClientConnector.request(HttpClientConnector.java:186)
at com.woorea.openstack.base.client.OpenStackClient.request(OpenStackClient.java:71)
at com.woorea.openstack.base.client.OpenStackClient.execute(OpenStackClient.java:87)

...

2019-10-14T10:35:22.015Z|94679b5b-a360-4f78-a9c3-d097e1b2ec25|org.onap.so.openstack.utils.MsoHeatUtils - queryHeatStack - stack not found: Vfmodule_Ete_vFWCLvFWSNK_b0857107_0

2019-10-14T10:35:22.021Z||org.onap.so.adapters.vnf.VnfAdapterRest - Create VfModule enter inside VnfAdapterRest: {"createVfModuleRequest":{"messageId":"94679b5b-a360-4f78-a9c3-d097e1b2ec25-1571049321778","skipAAI":true,"notificationUrl":"http://so-bpmn-infra.onap:8081/mso/WorkflowMessage/VNFAResponse/94679b5b-a360-4f78-a9c3-d097e1b2ec25-1571049321778",

Looks like openstack does not respond with status correctly after a vfModule create.

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	SO-2447

vFWCL Duplicate IP address 10.0.251.103

Test Status: 8 AM Oct 14

/dev/vda1 162420480 52463904 109940192 33% /

root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2667m 33% 12582Mi 78%
long-k8s-02 1447m 18% 13325Mi 83%
long-k8s-03 554m 6% 9325Mi 58%
long-k8s-04 654m 8% 12090Mi 75%
long-k8s-05 1419m 17% 13221Mi 82%
long-k8s-06 1077m 13% 14284Mi 89%
long-k8s-07 1350m 16% 12302Mi 77%
long-k8s-08 2907m 36% 14036Mi 88%
long-k8s-09 4157m 51% 11142Mi 69%
long-orch-1 207m 10% 1424Mi 37%
long-orch-2 198m 9% 1449Mi 37%
long-orch-3 320m 16% 1253Mi 32%

root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2985m 673Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1005m 236Mi
dev-oof-music-tomcat-98645f744-bdvp7 712m 263Mi
dev-consul-consul-7556c76b57-rh7rs 659m 29Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 548m 145Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 467m 1141Mi
dev-oof-music-cassandra-0 412m 766Mi
dev-contrib-awx-0 322m 1378Mi
dev-oof-music-cassandra-2 258m 772Mi
dev-cassandra-cassandra-0 206m 1813Mi

288

Same problem as #260

vVG : Stack Vfmodule_Ete_vVG_b6aa2967_0 already exists in Tenant

duplicate stack name instead of duplicate ip address but same problem

289

Heatbridge Validation

AAI query on reverse heat bridge

Testsuite should wrap in Wait For Keyword Success instead of justh query - cassandra replication delay

290

Same problem as #260

vFWCL: vfmodule name duplicate

Test Status 1 PM Oct 14

/dev/vda1 162420480 56555456 105848640 35% /

No stranded VMs or Stacks

36 keypairs

(during active instantiate phase)

root@long-nfs:/tmp# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2823m 35% 12580Mi 78%
long-k8s-02 1540m 19% 13353Mi 83%
long-k8s-03 584m 7% 9364Mi 58%
long-k8s-04 653m 8% 12108Mi 75%
long-k8s-05 1781m 22% 13207Mi 82%
long-k8s-06 1471m 18% 14279Mi 89%
long-k8s-07 867m 10% 12362Mi 77%
long-k8s-08 3024m 37% 14075Mi 88%
long-k8s-09 4352m 54% 11064Mi 69%
long-orch-1 192m 9% 1359Mi 35%
long-orch-2 146m 7% 1386Mi 36%
long-orch-3 301m 15% 1159Mi 30%

root@long-nfs:/tmp# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2936m 668Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1004m 236Mi
dev-oof-music-tomcat-98645f744-bdvp7 670m 263Mi
dev-consul-consul-7556c76b57-rh7rs 542m 69Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 525m 145Mi
dev-oof-music-cassandra-0 443m 768Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 383m 1158Mi
dev-cassandra-cassandra-1 265m 1847Mi
dev-robot-robot-5f67d595b6-cl6bc 224m 4311Mi
dev-contrib-awx-0 207m 1378Mi

292

Same problem as #260

vFWCL : IP address 10.0.227.101 already allocated in subnet

Latest analysis indicates it may be a problem with shared mariadb-galera server.cnf that is not providing the right locking to Camunda under load.

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	OOM-2132

Test Status 5 PM Oct 14

/dev/vda1 162420480 59795972 102608124 37% /

No stranded VMs or Stacks

root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2902m 36% 12577Mi 78%
long-k8s-02 1508m 18% 13292Mi 83%
long-k8s-03 546m 6% 9337Mi 58%
long-k8s-04 702m 8% 12079Mi 75%
long-k8s-05 1621m 20% 13283Mi 83%
long-k8s-06 1239m 15% 14345Mi 90%
long-k8s-07 860m 10% 12420Mi 77%
long-k8s-08 3142m 39% 14058Mi 88%
long-k8s-09 4366m 54% 11121Mi 69%
long-orch-1 167m 8% 1453Mi 37%
long-orch-2 182m 9% 1476Mi 38%
long-orch-3 367m 18% 1258Mi 32%

root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2982m 699Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1008m 235Mi
dev-oof-music-tomcat-98645f744-bdvp7 762m 263Mi
dev-consul-consul-7556c76b57-rh7rs 671m 46Mi
dev-oof-music-cassandra-0 671m 770Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 546m 145Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 431m 1163Mi
dev-contrib-awx-0 198m 1378Mi
dev-oof-music-cassandra-2 193m 773Mi
dev-oof-music-cassandra-1 166m 778Mi

304

Same problem as #260

vFWCL : IP address 192.168.10.200 already allocated in subnet

Test Status 8 AM Oct 15

/dev/vda1 162420480 72287180 90116916 45% /

No stranded VMs or Stacks

root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 3023m 37% 12621Mi 79%
long-k8s-02 1578m 19% 13298Mi 83%
long-k8s-03 548m 6% 9344Mi 58%
long-k8s-04 643m 8% 12039Mi 75%
long-k8s-05 1891m 23% 13415Mi 84%
long-k8s-06 1739m 21% 14462Mi 90%
long-k8s-07 700m 8% 12654Mi 79%
long-k8s-08 3355m 41% 14055Mi 88%
long-k8s-09 4288m 53% 10840Mi 68%
long-orch-1 345m 17% 1486Mi 38%
long-orch-2 196m 9% 1476Mi 38%
long-orch-3 391m 19% 1327Mi 34%

root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2980m 606Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1010m 235Mi
dev-oof-music-tomcat-98645f744-bdvp7 696m 263Mi
dev-consul-consul-7556c76b57-rh7rs 680m 28Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 628m 145Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 558m 1193Mi
dev-oof-music-cassandra-0 413m 771Mi
dev-oof-music-cassandra-1 370m 880Mi
dev-contrib-awx-0 280m 1387Mi
dev-cassandra-cassandra-1 239m 1861Mi

312

vVVG

On Delete Phase

Closed Loop Tests

This test uses the onap-ci job "Project windriver-longevity-vfwclosedloop".

...

Versions Compared

Old Version 11

New Version 12

Key

Closed Loop Tests

Page Comparison

Versions Compared

Old Version 11

New Version 12

Key

Closed Loop Tests