Versions Compared

Old Version 4

changes.mady.by.user FREEMAN, BRIAN D

Saved on

compared with

New Version 5

changes.mady.by.user FREEMAN, BRIAN D

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Test #CommentMessage

Test start #243 1 PM Oct 12
245Validate vServer in testsuite HeatBridge needed to wait for the AAI index update. Wrapped this step in a  Wait For Keyword Successpost response: {"requestError":{"serviceException":{"messageId":"SVC3001","text":"Resource not found for %1 using id %2 (msg=%3) (ec=%4)","variables":["POST Search","getNamedQueryResponse","Node Not Found:No Node of type vserver found for properties","ERR.5.4.6114"]}}}
260


Tooling or SO to Openstack interface transient


Received failure response from so {"request":{"requestId":"79264729-04ab-4738-a27d-29013c59218c","startTime":"Sun, 13 Oct 2019 09:38:20 GMT","finishTime":"Sun, 13 Oct 2019 09:39:14 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"VfwclVfwsnk0f6a8e47E64e..base_vfw..module-0","modelInvariantId":"e994097b-6285-49e1-a87c-76ba6e0371ab","modelType":"vfModule","modelName":"VfwclVfwsnk0f6a8e47E64e..base_vfw..module-0","modelVersion":"1","modelCustomizationUuid":"6ce786ef-31e8-4f00-bdb4-1c66f54eaffd","modelVersionId":"72f56293-fbf2-49fa-bb13-1df8f5f88548","modelCustomizationId":"6ce786ef-31e8-4f00-bdb4-1c66f54eaffd","modelUuid":"72f56293-fbf2-49fa-bb13-1df8f5f88548","modelInvariantUuid":"e994097b-6285-49e1-a87c-76ba6e0371ab","modelInstanceName":"VfwclVfwsnk0f6a8e47E64e..base_vfw..module-0"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vFWCLvFWSNK_031aaae1_0","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"fc4a3aac-e15e-4cf2-b85c-93eee3cdf3cc","modelInfo":{"modelInvariantId":"ed6ca1d8-cf38-455b-bb0a-75ae84d51715","modelType":"service","modelName":"vFWCL 2019-10-13 09:29:","modelVersion":"1.0","modelVersionId":"1c3dece0-945e-4f38-b5d2-f1d3fe7579e1","modelUuid":"1c3dece0-945e-4f38-b5d2-f1d3fe7579e1","modelInvariantUuid":"ed6ca1d8-cf38-455b-bb0a-75ae84d51715"}}},{"relatedInstance":{"instanceId":"d4cc80c3-367c-4de2-8dd2-52904466b60a","modelInfo":{"modelCustomizationName":"vFWCL_vFWSNK 0f6a8e47-e64e 0","modelInvariantId":"dcbe3ca3-b9c3-4042-a06f-5ad83f1be089","modelType":"vnf","modelName":"vFWCL_vFWSNK 0f6a8e47-e64e","modelVersion":"1.0","modelCustomizationUuid":"9eaff9be-ac20-4872-9804-7bd45515a351","modelVersionId":"2de4b9dd-b6d6-4822-92c0-670c9329557f","modelCustomizationId":"9eaff9be-ac20-4872-9804-7bd45515a351","modelUuid":"2de4b9dd-b6d6-4822-92c0-670c9329557f","modelInvariantUuid":"dcbe3ca3-b9c3-4042-a06f-5ad83f1be089","modelInstanceName":"vFWCL_vFWSNK 0f6a8e47-e64e 0"}}}],"cloudConfiguration":{"tenantId":"28481f6939614cfd83e6767a0e039bcc","cloudOwner":"CloudOwner","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true,"testApi":"VNF_API"}},"instanceReferences":{"serviceInstanceId":"fc4a3aac-e15e-4cf2-b85c-93eee3cdf3cc","vnfInstanceId":"d4cc80c3-367c-4de2-8dd2-52904466b60a","vfModuleInstanceName":"Vfmodule_Ete_vFWCLvFWSNK_031aaae1_0","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"STATUS: Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF org.onap.so.openstack.utils.StackCreationException: Stack Creation Failed Openstack Status: CREATE_FAILED Status Reason: Resource CREATE failed: Conflict: resources.vsn_0_onap_private_port_0: IP address 10.0.235.102 already allocated in subnet 4ed99c09-aed6-4eca-8f94-48357ab4e5d1\nNeutron server returns request_ids: ['req-f60a93ff-ecbf-4c5e-b149-8ebdf64e38f2'] , Rollback of Stack Creation completed with status: DELETE_COMPLETE Status Reason: Stack DELETE completed successfully' rolledBack='true'","percentProgress":100,"timestamp":"Sun, 13 Oct 2019 09:39:14 GMT"}}}

Test Status : #261 7 AM Oct 13

No left over VMs or Stacks from delete

Docker-data-nfs at 21% of available capacity

robot container:   10.0.0.4:/dockerdata-nfs/dev-robot/robot/logs 162420736 33509376 128894976 21% /share/logs.


17 keypairs under demo account


Environment Spot Check when tests are not running look okay.

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2240m 28% 12448Mi 78%
long-k8s-02 1367m 17% 13446Mi 84%
long-k8s-03 505m 6% 9422Mi 59%
long-k8s-04 677m 8% 12128Mi 76%
long-k8s-05 1733m 21% 12895Mi 80%
long-k8s-06 1300m 16% 14170Mi 88%
long-k8s-07 1171m 14% 12040Mi 75%
long-k8s-08 3019m 37% 13843Mi 86%
long-k8s-09 4046m 50% 11023Mi 69%
long-orch-1 174m 8% 1369Mi 35%
long-orch-2 169m 8% 1305Mi 33%
long-orch-3 284m 14% 1190Mi 30%


root@long-nfs:/home/ubuntu# kubectl -n onap top pods | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2961m 713Mi 
dev-uui-uui-server-6cf5b6bb7f-2g88z 1004m 236Mi 
dev-oof-music-tomcat-98645f744-bdvp7 782m 262Mi 
dev-oof-music-cassandra-0 565m 761Mi 
dev-consul-consul-7556c76b57-rh7rs 497m 29Mi 
dev-oof-oof-has-controller-755d446d46-7hzb4 449m 144Mi 
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 385m 1096Mi 
dev-contrib-awx-0 258m 1375Mi 
dev-oof-music-cassandra-2 221m 766Mi 
dev-portal-portal-cassandra-75479b4646-8srbs 213m 2787Mi

Test Status: #267 12:00 PM Oct 13

/dev/vda1 162420480 36636868 125767228 23% /

No left over VMs or Stacks from previous runs

RegionOne_ONAP-NF_20191013T150300143Z_olc-key_PlYL style keypairs added in the morning. Up to 27 keypairs


268


Same signature as #260

,"requestStatus":{"requestState":"FAILED","statusMessage":"STATUS: Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF org.onap.so.openstack.utils.StackCreationException: Stack Creation Failed Openstack Status: CREATE_FAILED Status Reason: Resource CREATE failed: Conflict: resources.vdns_0_onap_private_port_0: IP address 10.0.236.25 already allocated in subnet 4ed99c09-aed6-4eca-8f94-48357ab4e5d1\nNeutron server returns request_ids: ['req-2a9b2ed0-0502-4377-b209-59f36a27d8dd'] , Rollback of Stack Creation completed with status: DELETE_COMPLETE Status Reason: Stack DELETE completed successfully'

Test Status #271  4:00 PM Oct 13

/dev/vda1 162420480 40495064 121909032 25% /


No left over VMs or Stacks

Up to 33 keypairs


root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 
long-k8s-01 2388m 29% 12455Mi 78% 
long-k8s-02 1439m 17% 13371Mi 83% 
long-k8s-03 502m 6% 9342Mi 58% 
long-k8s-04 836m 10% 12083Mi 75% 
long-k8s-05 1508m 18% 12989Mi 81% 
long-k8s-06 1238m 15% 14234Mi 89% 
long-k8s-07 1264m 15% 12020Mi 75% 
long-k8s-08 1275m 15% 13968Mi 87% 
long-k8s-09 4309m 53% 11170Mi 70% 
long-orch-1 190m 9% 1384Mi 36% 
long-orch-2 167m 8% 1410Mi 36% 
long-orch-3 300m 15% 1224Mi 31% 



root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2976m 700Mi 
dev-uui-uui-server-6cf5b6bb7f-2g88z 1005m 237Mi 
dev-oof-music-tomcat-98645f744-bdvp7 728m 262Mi 
dev-consul-consul-7556c76b57-rh7rs 621m 27Mi 
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 578m 1118Mi 
dev-cassandra-cassandra-1 565m 1839Mi 
dev-oof-oof-has-controller-755d446d46-7hzb4 476m 145Mi 
dev-sdc-sdc-onboarding-be-75fbbb65c9-d95zh 342m 752Mi 
dev-contrib-awx-0 303m 1438Mi 
dev-cassandra-cassandra-2 286m 1809Mi


276Same signature as #260questParameters":{"usePreload":true,"testApi":"VNF_API"}},"instanceReferences":{"serviceInstanceId":"7374c399-e4af-4cc8-81b3-cb0ff810ac7c","vnfInstanceId":"06888576-bd1b-4b30-b27f-3b61a0898bee","vfModuleInstanceName":"Vfmodule_Ete_vFWCLvPKG_3cd57462_1","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"STATUS: Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF org.onap.so.openstack.utils.StackCreationException: Stack Creation Failed Openstack Status: CREATE_FAILED Status Reason: Resource CREATE failed: Conflict: resources.vpg_0_onap_private_port_0: IP address 10.0.158.103 already allocated in subnet 4ed99c09-aed6-4eca-8f94-48357ab4e5d1\nNeutron server returns request_ids: ['req-3e4fd376-8698-4211-95a1-eb1312a71c28'] , Rollback of Stack Creation completed with status: DELETE_COMPLETE Status Reason: Stack DELETE completed successfully' rolledBack='true'","percentProgress":100,"timestamp":"Mon, 14 Oct 2019 01:35:58 GMT"}}

Test Status   #276  10:00 PM Oct 13

No stranded VMs or Stacks


robot  log storage up to 28% usage

10.0.0.4:/dockerdata-nfs/dev-robot/robot/logs 162420736 44839936 117564416 28% /share/logs

root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2658m 33% 12480Mi 78%
long-k8s-02 296m 3% 13336Mi 83%
long-k8s-03 514m 6% 9328Mi 58%
long-k8s-04 694m 8% 12168Mi 76%
long-k8s-05 1436m 17% 13196Mi 82%
long-k8s-06 1154m 14% 14280Mi 89%
long-k8s-07 1288m 16% 12068Mi 75%
long-k8s-08 3080m 38% 14026Mi 87%
long-k8s-09 4068m 50% 11226Mi 70%
long-orch-1 197m 9% 1331Mi 34%
long-orch-2 163m 8% 1393Mi 36%
long-orch-3 432m 21% 1203Mi 31%
root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2968m 700Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1002m 236Mi
dev-robot-robot-5f67d595b6-cl6bc 830m 4327Mi
dev-oof-music-cassandra-0 734m 763Mi
dev-oof-music-tomcat-98645f744-bdvp7 717m 262Mi
dev-consul-consul-7556c76b57-rh7rs 673m 28Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 460m 146Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 317m 1139Mi
dev-contrib-awx-0 239m 1380Mi
dev-oof-music-cassandra-2 202m 770Mi

281Same signature as #260
282Same signature as #260
285Same signature as #260
286Same signature as #260

Potential cause is conflict with vfwclosedloop vnf in preload / test tool data

10.0.251.101, 102, 103 are used by vfwclosed loop but not excluded from test data for instantiate tests.

Still need to look at ip address removal in openstack to see if ip address aging is affecting the tests


Test Status: 8 AM Oct 14

/dev/vda1 162420480 52463904 109940192 33% /


root@long-nfs:/home/ubuntu# kubectl -n onap top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
long-k8s-01 2667m 33% 12582Mi 78%
long-k8s-02 1447m 18% 13325Mi 83%
long-k8s-03 554m 6% 9325Mi 58%
long-k8s-04 654m 8% 12090Mi 75%
long-k8s-05 1419m 17% 13221Mi 82%
long-k8s-06 1077m 13% 14284Mi 89%
long-k8s-07 1350m 16% 12302Mi 77%
long-k8s-08 2907m 36% 14036Mi 88%
long-k8s-09 4157m 51% 11142Mi 69%
long-orch-1 207m 10% 1424Mi 37%
long-orch-2 198m 9% 1449Mi 37%
long-orch-3 320m 16% 1253Mi 32%


root@long-nfs:/home/ubuntu# kubectl -n onap top pod | sort -k2,2nr | head
dev-log-log-logstash-7d6c57f746-2b9gb 2985m 673Mi
dev-uui-uui-server-6cf5b6bb7f-2g88z 1005m 236Mi
dev-oof-music-tomcat-98645f744-bdvp7 712m 263Mi
dev-consul-consul-7556c76b57-rh7rs 659m 29Mi
dev-oof-oof-has-controller-755d446d46-7hzb4 548m 145Mi
dep-dcae-tca-analytics-5fdcc57cc9-78f9d 467m 1141Mi
dev-oof-music-cassandra-0 412m 766Mi
dev-contrib-awx-0 322m 1378Mi
dev-oof-music-cassandra-2 258m 772Mi
dev-cassandra-cassandra-0 206m 1813Mi




Closed Loop Tests

This test uses the onap-ci job "Project windriver-longevity-vfwclosedloop".

...

Test #Comment             Message                                                                               

Test Start    #1595   4 PM Oct 12

Test Status: #1610   7 AM Oct 13No issues. No failed tests

Test Status: #1615 12 PM Oct 13No issues. No failed tests

Test Status  #1620   5 PM Oct 13No issues. No failed tests

Test Status #1625  10 PM Oct 13No issues. No failed tests

Test Status #1635  8 AM  Oct 14No issues. No failed tests



...