Beijing Release Stability Testing Status (w/ OOM)

Monitoring shift, EDT

5/24/2018: 12:00 - 24:00 (Helen / Gary)

5/25/2018: 00:00 - 2:00 (Helen / Gary), 2:00 - 22:00 (Marco), 22:00 - 24:00 (Helen)

5/26/2018 00:00 - 2:00 (Helen), 2:00 - 20:00 (Yang), 20:00 - 24:00 (Helen)

5/27/2018 00:00 – 12:00 (Helen)

Stability Testing Starts at 5/24/2018,11:23 EDT, at Intel / Windriver Lab, Integration-SB-04

Health Test: windriver-SB04-oom-healthcheck #1004 May 24, 2018 at 9:08 AM windriver-SB04-oom-healthcheck #1292 May 27, 2018 at 9:08 AM

Time (EDT)	Report By	Health Check	Robot Test (Every 30 minutes)	Closed Loop	Manual Check	Notes

5/27/2018 8:00 - 12:00	Helen	41/43	22/24	Pass	Portal is accessible	Instantiate Virtual DNS \| FAIL 06:55:28 PDT Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: ValueError: Element locator 'xpath=//pre[@class= 'log ng-binding']' did not match any elements. Instantiate Virtual FirewallCL \| FAIL 07:17:16 PDT Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: ValueError: Element locator 'xpath=//pre[@class= 'log ng-binding']' did not match any elements.
5/27/2018 4:00 - 8:00	Helen	41/43	22/24	Pass	Portal is accessible	(The following timestamp is PDT time) Instantiate Virtual FirewallCL \| FAIL \| 02:19:22 Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: ValueError: Element locator 'xpath=//pre[@class= 'log ng-binding']' did not match any elements. 2. Instantiate Virtual FirewallCL \| FAIL \| 01:45:55 '{"request":{"requestId":"4d5794c7-09e2-47d4-8810-5e6b931d2f7e","startTime":"Sun, 27 May 2018 08:44:53 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"Ab48a65a54094eba90f9..base_vfw..module-0","modelInvariantId":"b3687b3d-07e7-4704-af5a-2fc0c8934dca","modelType":"vfModule","modelName":"Ab48a65a54094eba90f9..base_vfw..module-0","modelVersion":"1","modelVersionId":"e3bab184-da66-491b-b12a-f26c1e430fe9","modelCustomizationId":"62c2ec5b-6379-4190-b21e-f968d4425ffe"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vFWSNK_8c1e3b87-225c-403a-a7b5-e9c6fa7b19f7","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"2d391e2d-60ac-4f88-add5-1c8652214344","modelInfo":{"modelInvariantId":"20d2d633-7f49-452f-a4f9-436d97537030","modelType":"service","modelName":"vFWCL 2018-05-27 08:34:","modelVersion":"1.0","modelVersionId":"45df431f-4c8f-45ba-8cde-cc23d413829d"}}},{"relatedInstance":{"instanceId":"e55e8b8a-7013-4269-86d0-7f5417b41d63","modelInfo":{"modelCustomizationName":"ab48a65a-5409-4eba-90f9 0","modelInvariantId":"017a8bf8-4ec8-4571-bbc2-5960ebe944ed","modelType":"vnf","modelName":"ab48a65a-5409-4eba-90f9","modelVersion":"1.0","modelVersionId":"8da0f02c-fee4-429d-96db-93db0946db09","modelCustomizationId":"165fd673-7980-424f-93cb-9d745990c1e9"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"2d391e2d-60ac-4f88-add5-1c8652214344","vnfInstanceId":"e55e8b8a-7013-4269-86d0-7f5417b41d63","vfModuleInstanceName":"Vfmodule_Ete_vFWSNK_8c1e3b87-225c-403a-a7b5-e9c6fa7b19f7","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=StackValidationFailed, error.message=The request you have made requires authentication. (HTTP 401) (Request-ID: req-dfe91230-7fa8-498c-8de7-050f5a2e2f2c)' rolledBack='true'","percentProgress":100,"finishTime":"Sun, 27 May 2018 08:46:03 GMT"}}}' contains 'FAILED'
5/27/2018 00:00 - 4:00	Helen	41/43	21/24	Pass	SDC and related portal is fine.	(The following timestamp is PDT time) 3. Instantiate Virtual FirewallCL \| FAIL \| 00:10:56 '{"request":{"requestId":"67e2b8c7-90c9-4b82-84da-7d737c00e42e","startTime":"Sun, 27 May 2018 07:10:39 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"5bf286b8C64f4efb922f..base_vpkg..module-0","modelInvariantId":"28d6a639-ec31-4e59-a2ba-ed5373df1dc8","modelType":"vfModule","modelName":"5bf286b8C64f4efb922f..base_vpkg..module-0","modelVersion":"1","modelVersionId":"bf6d329a-443d-414a-8a05-58ce8ce734c3","modelCustomizationId":"6875bad2-7c49-45e5-be81-999b1091516e"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vPKG_bd34861d-e48f-445e-a57e-3e1ee4cb4039","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"212dbc25-3309-4a19-9fe6-11ea9f2f7d5a","modelInfo":{"modelInvariantId":"3c6d6aee-168b-41b5-81d2-ee83045d978d","modelType":"service","modelName":"vFWCL 2018-05-27 07:01:","modelVersion":"1.0","modelVersionId":"8f77b464-391c-4ede-baea-c5a5e96b8f9f"}}},{"relatedInstance":{"instanceId":"5af90fee-0094-45bc-b6d7-0156ea44ea4e","modelInfo":{"modelCustomizationName":"5bf286b8-c64f-4efb-922f 0","modelInvariantId":"b2f7c025-0e11-45c2-a171-b84e588bbda4","modelType":"vnf","modelName":"5bf286b8-c64f-4efb-922f","modelVersion":"1.0","modelVersionId":"e6190537-df94-4cb1-a8f0-8edf26312c5e","modelCustomizationId":"10e34793-fc21-425d-bcaf-c5be4cf5aa22"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"212dbc25-3309-4a19-9fe6-11ea9f2f7d5a","vnfInstanceId":"5af90fee-0094-45bc-b6d7-0156ea44ea4e","vfModuleInstanceName":"Vfmodule_Ete_vPKG_bd34861d-e48f-445e-a57e-3e1ee4cb4039","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received error from SDN-C: Encountered error while updating vf-module orchestration-status in AnAI","percentProgress":100,"finishTime":"Sun, 27 May 2018 07:11:02 GMT"}}}' contains 'FAILED' 2. Instantiate Virtual DNS \| FAIL \| 23:12:25 Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: ValueError: Element locator 'xpath=//pre[@class= 'log ng-binding']' did not match any elements. Instantiate Virtual Volume Group + retval=0 5/26/2018 23:04:31 ++ ssh -o StrictHostKeychecking=no -i /var/lib/jenkins/.ssh/onap_key ubuntu@10.12.5.187 'sudo su -c "kubectl --namespace onap get pods"' 23:04:32 Build step 'Execute shell' marked build as failure 23:04:32 Robot results publisher started... 23:04:32 -Parsing output xml: 23:04:32 Failed! 23:04:32 hudson.AbortException: No files found in path /var/lib/jenkins/workspace/windriver-SB04-oom-stability72hr/archives with configured filemask: output.xml 23:04:32 at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:77) 23:04:32 at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:54) 23:04:32 at hudson.FilePath.act(FilePath.java:997) 23:04:32 at hudson.FilePath.act(FilePath.java:975) 23:04:32 at hudson.plugins.robot.RobotParser.parse(RobotParser.java:49) 23:04:32 at hudson.plugins.robot.RobotPublisher.parse(RobotPublisher.java:217) 23:04:32 at hudson.plugins.robot.RobotPublisher.perform(RobotPublisher.java:239) 23:04:32 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) 23:04:32 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) 23:04:32 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) 23:04:32 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) 23:04:32 at hudson.model.Build$BuildExecution.post2(Build.java:186) 23:04:32 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) 23:04:32 at hudson.model.Run.execute(Run.java:1749) 23:04:32 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 23:04:32 at hudson.model.ResourceController.execute(ResourceController.java:97) 23:04:32 at hudson.model.Executor.run(Executor.java:429) 23:04:32 Finished: FAILURE (This is a serious error, which failed robot scripts)
5/26/2018 20:00 - 24:00	Helen	41/43	23/24 Pass	Pass		Health check failed AAF and VNFSDK (non-blocking). Instantiate Virtual Volume Group \| FAIL \| 5/26 20:48PM EDT Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: ValueError: Element locator 'xpath=//pre[@class= 'log ng-binding']' did not match any elements.
5/26/2018 2:00-20:00	Yang	41/43	??	Pass	SDC Portal works fine	Health check failed on AAF and VNFSDK (non-blocking) 16:31:44 Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: ValueError: Element locator 'xpath=//pre[@class= 'log ng-binding']' did not match any elements. Instantiate Virtual DNS \| FAIL \| 01:55:05 Instantiate Virtual DNS Failed 02:05:10 '{"request":{"requestId":"4de3c5c2-a60d-40dd-a138-8250176c9f71","startTime":"Sat, 26 May 2018 09:04:38 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"0e1b3b82Fe8344a18985..base_vlb..module-0","modelInvariantId":"11d2f7f6-685c-41e4-9be7-99ffd573be50","modelType":"vfModule","modelName":"0e1b3b82Fe8344a18985..base_vlb..module-0","modelVersion":"1","modelVersionId":"93d2f04e-19a6-4ffb-ac7b-353f14db4be7","modelCustomizationId":"ed8d3a29-2bba-454a-adf4-2992173a336c"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vLB_1f4d6841-9513-4260-8f5c-cca185fe78f0","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"55cf4f99-b5fe-424e-b788-242352e1494b","modelInfo":{"modelInvariantId":"3ebd8df4-4dde-40d5-bb0d-3690c48919b4","modelType":"service","modelName":"vLB 2018-05-26 08:55:17","modelVersion":"1.0","modelVersionId":"8d84e20e-3b25-4d24-a60a-3bdca076bb00"}}},{"relatedInstance":{"instanceId":"886b17d1-a49d-47e6-bf8b-89242beb30ee","modelInfo":{"modelCustomizationName":"0e1b3b82-fe83-44a1-8985 0","modelInvariantId":"b1f7e99c-61c1-419c-9943-d6ec2e547c01","modelType":"vnf","modelName":"0e1b3b82-fe83-44a1-8985","modelVersion":"1.0","modelVersionId":"5af87f7e-9ee7-427e-bad8-1423a7080a70","modelCustomizationId":"ff7ec5c6-9bd5-46c4-8a04-246ba71cb3d0"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"55cf4f99-b5fe-424e-b788-242352e1494b","vnfInstanceId":"886b17d1-a49d-47e6-bf8b-89242beb30ee","vfModuleInstanceName":"Vfmodule_Ete_vLB_1f4d6841-9513-4260-8f5c-cca185fe78f0","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received error from SDN-C: Encountered error while updating vf-module orchestration-status in AnAI","percentProgress":100,"finishTime":"Sat, 26 May 2018 09:05:11 GMT"}}}' contains 'FAILED' Instantiate Virtual DNS Failed
5/25/2018 22:00- 5/26/2018 2:00	Helen	41/43	22/24	Pass		Health check failed on AAF and VNFSDK (non-blocking) Instantiate Virtual DNS \| FAIL \| 21:04:01 PDT Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: Element 'xpath=//select[@prompt='Select Service Type']' did not appear in 2 minutes Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: ValueError: Element locator 'xpath=//pre[@class= 'log ng-binding']' did not match any elements.
5/25/2018 6:00 - 22:00	Marco	41/43	??	vFirewall CL still working	Portal is accessible	Health check failed AAF and VNFSDK (non-blocking). vFW instantiation Failures 1) Keyword 'Check Catalog Service Distributed' failed after retrying for 3 minutes. The last error was: SO Test 2) '{"request":{"requestId":"92576e97-095a-4e77-996a-8415b055443d","startTime":"Fri, 25 May 2018 00:50:37 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"C6709519942142d2A1de..base_vfw..module-0","modelInvariantId":"6273a5fd-578e-4ed0-bc6e-e5c00dd474af","modelType":"vfModule","modelName":"C6709519942142d2A1de..base_vfw..module-0","modelVersion":"1","modelVersionId":"65b45fa9-7031-43ec-9365-8d6d803cb73d","modelCustomizationId":"06689505-eae7-4a15-b9d0-a9d152aa68e0"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vFWSNK_41cafa88-f813-4eec-992e-30cecfa08d65","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"01fe980a-c9c9-4236-bc8b-291c99ae91e9","modelInfo":{"modelInvariantId":"ef354d84-8aa5-48d2-9f62-46d71f07ccb6","modelType":"service","modelName":"vFWCL 2018-05-25 00:46:","modelVersion":"1.0","modelVersionId":"5b0c9947-60ae-4eb7-91cf-7970cadfac5c"}}},{"relatedInstance":{"instanceId":"2eefeaf6-6ca4-421d-a77c-f03099a64797","modelInfo":{"modelCustomizationName":"c6709519-9421-42d2-a1de 0","modelInvariantId":"5146e5ae-a72b-4dd2-b08e-e667b9cd0021","modelType":"vnf","modelName":"c6709519-9421-42d2-a1de","modelVersion":"1.0","modelVersionId":"20d22b9d-ff94-4076-8cab-5c7227b7f95b","modelCustomizationId":"73a013d4-28b0-4e53-8b32-fdeb7a249617"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"01fe980a-c9c9-4236-bc8b-291c99ae91e9","vnfInstanceId":"2eefeaf6-6ca4-421d-a77c-f03099a64797","vfModuleInstanceName":"Vfmodule_Ete_vFWSNK_41cafa88-f813-4eec-992e-30cecfa08d65","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Fri, 25 May 2018 00:51:08 GMT"}}}' contains 'FAILED' vLoadBalancer Instantiation Failures 1) '{"request":{"requestId":"ad291713-d59b-47c3-bde4-07e8b3a98b26","startTime":"Fri, 25 May 2018 02:04:50 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"2a5d75b64e3347fa84d5..base_vlb..module-0","modelInvariantId":"7e8c9464-c89d-4e7c-b1ca-ff9cdb5c592b","modelType":"vfModule","modelName":"2a5d75b64e3347fa84d5..base_vlb..module-0","modelVersion":"1","modelVersionId":"ef5672d6-7fa6-4cd7-a31b-e600933b2896","modelCustomizationId":"0aa6c664-4f3d-4ee4-a408-bc3134d6be76"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vLB_9d9631ba-ae23-4c11-bbc8-8748d1548a88","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"ebe5d310-911c-477c-8c5e-e7f7b693e0cb","modelInfo":{"modelInvariantId":"0b666635-c454-474a-86e3-f74705de0460","modelType":"service","modelName":"vLB 2018-05-25 02:00:24","modelVersion":"1.0","modelVersionId":"5ac5f370-4a81-4d59-ac83-ccf3ce347d9c"}}},{"relatedInstance":{"instanceId":"1c671246-505b-4660-834b-631fe40b16f8","modelInfo":{"modelCustomizationName":"2a5d75b6-4e33-47fa-84d5 0","modelInvariantId":"c7af8dce-91fd-426b-a084-24766beb1317","modelType":"vnf","modelName":"2a5d75b6-4e33-47fa-84d5","modelVersion":"1.0","modelVersionId":"bc931325-5c60-48ab-a85f-5c1280e4c4d8","modelCustomizationId":"66876b00-91ab-4cfe-929f-5cc2ebf665be"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"ebe5d310-911c-477c-8c5e-e7f7b693e0cb","vnfInstanceId":"1c671246-505b-4660-834b-631fe40b16f8","vfModuleInstanceName":"Vfmodule_Ete_vLB_9d9631ba-ae23-4c11-bbc8-8748d1548a88","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=StackValidationFailed, error.message=Property error: : resources.vlb_private_1_port.properties.fixed_ips[0].subnet: : Error validating value 'c7c8198e-e0f9-4b0f-a27d-316f97d7be98': 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}\nNeutron server returns request_ids: ['req-2764e9a9-638b-4ad3-8ecc-c8bfe37396cc']' rolledBack='true'","percentProgress":100,"finishTime":"Fri, 25 May 2018 02:05:21 GMT"}}}' contains 'FAILED' Virtual Volume Group Failures 1) StaleElementReferenceException: Message: stale element reference: element is not attached to the page document 2) Keyword 'Check Catalog Service Distributed' failed after retrying for 3 minutes. The last error was: SO Test
5/25/2018 2:00 - 6:00	Marco / Helen	41/43	18/24	vFirewall CL still working		Health check failed AAF and VNFSDK (non-blocking). Instantiate Virtual FirewallCL \| FAIL \| 02:21:29 PDT Keyword 'Check Catalog Service Distributed' failed after retrying for 3 minutes. The last error was: SO Test 2. Instantiate Virtual DNS \| FAIL \| 01:35:01 '{"request":{"requestId":"c230759f-70fc-4b38-a4f0-7b8b6a78d733","startTime":"Fri, 25 May 2018 08:34:44 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"55cc2a82Eada4bccBabc..base_vlb..module-0","modelInvariantId":"01ae15e1-c6fd-45de-a4f5-713598005468","modelType":"vfModule","modelName":"55cc2a82Eada4bccBabc..base_vlb..module-0","modelVersion":"1","modelVersionId":"d02e48c8-5010-4ba1-8188-f07c6efe150b","modelCustomizationId":"eccfbcbf-c650-4c39-931e-b383c0b4767e"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vLB_54c4304a-2dde-4cdf-89cd-cb17d9c692db","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"62b6b006-da32-458c-8ef9-3d02ebf8027f","modelInfo":{"modelInvariantId":"448d662d-79b9-4f9e-aaa6-c41ba2793f48","modelType":"service","modelName":"vLB 2018-05-25 08:30:22","modelVersion":"1.0","modelVersionId":"09c27d3c-62c2-43ef-968e-5d5214b30f70"}}},{"relatedInstance":{"instanceId":"4cc5d411-fb09-46da-bf33-1c83663fa043","modelInfo":{"modelCustomizationName":"55cc2a82-eada-4bcc-babc 0","modelInvariantId":"4157dfc2-8eec-4f05-b845-2fdab6ff8266","modelType":"vnf","modelName":"55cc2a82-eada-4bcc-babc","modelVersion":"1.0","modelVersionId":"5c9903e9-76d3-494a-8a36-7bac97e141eb","modelCustomizationId":"6325ff9c-af3e-40b6-bd85-0e556d6b14f5"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"62b6b006-da32-458c-8ef9-3d02ebf8027f","vnfInstanceId":"4cc5d411-fb09-46da-bf33-1c83663fa043","vfModuleInstanceName":"Vfmodule_Ete_vLB_54c4304a-2dde-4cdf-89cd-cb17d9c692db","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=StackValidationFailed, error.message=The request you have made requires authentication. (HTTP 401)' rolledBack='true'","percentProgress":100,"finishTime":"Fri, 25 May 2018 08:35:00 GMT"}}}' contains 'FAILED' 3. Instantiate Virtual FirewallCL \| FAIL \| 01:44:34 PDT Keyword 'Check Catalog Service Distributed' failed after retrying for 3 minutes. The last error was: SO Test 4. Instantiate Virtual FirewallCL \| FAIL \| 01:23:39 PDT Keyword 'Check Catalog Service Distributed' failed after retrying for 3 minutes. The last error was: SO Test 5. Instantiate Virtual Volume Group \| FAIL \| 00:19:41 PDT Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: Element 'xpath=//select[@prompt='Select Service Type']' did not appear in 2 minutes 6. Instantiate Virtual FirewallCL \| FAIL \| 00:23:32 PDT Keyword 'Check Catalog Service Distributed' failed after retrying for 3 minutes. The last error was: SO Test
5/25/2018 0:00- 2:00	Helen	41/43	10/12	Pass	N/A	Health check failed AAF and VNFSDK (non-blocking). 1. 22:17:39 Instantiate Virtual FirewallCL \| FAIL \| 22:21:36 Keyword 'Check Catalog Service Distributed' failed after retrying for 3 minutes. The last error was: SO Test (No more error message) 2. 22:47:02 Instantiate Virtual FirewallCL + retval=0 22:56:31 ++ ssh -o StrictHostKeychecking=no -i /var/lib/jenkins/.ssh/onap_key ubuntu@10.12.5.187 'sudo su -c "kubectl --namespace onap get pods"' 22:56:31 ++ grep robot 22:56:31 ++ sed 's/ .//' 22:56:33* Error from server (InternalError): an error on the server ("Service unavailable") has prevented the request from succeeding (get pods) 22:56:33 + ROBOT_POD= 22:56:33 + '[' '' == '' ']' 22:56:33 + exit 1 22:56:33 Build step 'Execute shell' marked build as failure 22:56:33 Robot results publisher started... 22:56:33 -Parsing output xml: 22:56:33 Failed! 22:56:33 hudson.AbortException: No files found in path /var/lib/jenkins/workspace/windriver-SB04-oom-stability72hr/archives with configured filemask: output.xml 22:56:33 at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:77) 22:56:33 at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:54) 22:56:33 at hudson.FilePath.act(FilePath.java:997) 22:56:33 at hudson.FilePath.act(FilePath.java:975) 22:56:33 at hudson.plugins.robot.RobotParser.parse(RobotParser.java:49) 22:56:33 at hudson.plugins.robot.RobotPublisher.parse(RobotPublisher.java:217) 22:56:33 at hudson.plugins.robot.RobotPublisher.perform(RobotPublisher.java:239) 22:56:33 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) 22:56:33 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) 22:56:33 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) 22:56:33 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) 22:56:33 at hudson.model.Build$BuildExecution.post2(Build.java:186) 22:56:33 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) 22:56:33 at hudson.model.Run.execute(Run.java:1749) 22:56:33 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 22:56:33 at hudson.model.ResourceController.execute(ResourceController.java:97) 22:56:33 at hudson.model.Executor.run(Executor.java:429) 22:56:33 Finished: FAILURE
5/24/2018 20:00 - 24:00	Helen	41/43	21/24 Pass	Pass	N/A	Health check failed AAF and VNFSDK (non-blocking). 19:00:15 Instantiate Virtual DNS \| FAIL \| 19:05:24 '{"request":{"requestId":"ad291713-d59b-47c3-bde4-07e8b3a98b26","startTime":"Fri, 25 May 2018 02:04:50 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"2a5d75b64e3347fa84d5..base_vlb..module-0","modelInvariantId":"7e8c9464-c89d-4e7c-b1ca-ff9cdb5c592b","modelType":"vfModule","modelName":"2a5d75b64e3347fa84d5..base_vlb..module-0","modelVersion":"1","modelVersionId":"ef5672d6-7fa6-4cd7-a31b-e600933b2896","modelCustomizationId":"0aa6c664-4f3d-4ee4-a408-bc3134d6be76"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vLB_9d9631ba-ae23-4c11-bbc8-8748d1548a88","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"ebe5d310-911c-477c-8c5e-e7f7b693e0cb","modelInfo":{"modelInvariantId":"0b666635-c454-474a-86e3-f74705de0460","modelType":"service","modelName":"vLB 2018-05-25 02:00:24","modelVersion":"1.0","modelVersionId":"5ac5f370-4a81-4d59-ac83-ccf3ce347d9c"}}},{"relatedInstance":{"instanceId":"1c671246-505b-4660-834b-631fe40b16f8","modelInfo":{"modelCustomizationName":"2a5d75b6-4e33-47fa-84d5 0","modelInvariantId":"c7af8dce-91fd-426b-a084-24766beb1317","modelType":"vnf","modelName":"2a5d75b6-4e33-47fa-84d5","modelVersion":"1.0","modelVersionId":"bc931325-5c60-48ab-a85f-5c1280e4c4d8","modelCustomizationId":"66876b00-91ab-4cfe-929f-5cc2ebf665be"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"ebe5d310-911c-477c-8c5e-e7f7b693e0cb","vnfInstanceId":"1c671246-505b-4660-834b-631fe40b16f8","vfModuleInstanceName":"Vfmodule_Ete_vLB_9d9631ba-ae23-4c11-bbc8-8748d1548a88","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=StackValidationFailed, error.message=Property error: : resources.vlb_private_1_port.properties.fixed_ips[0].subnet: : Error validating value 'c7c8198e-e0f9-4b0f-a27d-316f97d7be98': 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}\nNeutron server returns request_ids: ['req-2764e9a9-638b-4ad3-8ecc-c8bfe37396cc']' rolledBack='true'","percentProgress":100,"finishTime":"Fri, 25 May 2018 02:05:21 GMT"}}}' contains 'FAILED' 19:05:24 2. 18:16:36 Instantiate Virtual FirewallCL \| FAIL \| 18:20:31 Keyword 'Check Catalog Service Distributed' failed after retrying for 3 minutes. The last error was: SO Test 3. 17:46:44 Instantiate Virtual FirewallCL \| FAIL \| 17:51:11 '{"request":{"requestId":"92576e97-095a-4e77-996a-8415b055443d","startTime":"Fri, 25 May 2018 00:50:37 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"C6709519942142d2A1de..base_vfw..module-0","modelInvariantId":"6273a5fd-578e-4ed0-bc6e-e5c00dd474af","modelType":"vfModule","modelName":"C6709519942142d2A1de..base_vfw..module-0","modelVersion":"1","modelVersionId":"65b45fa9-7031-43ec-9365-8d6d803cb73d","modelCustomizationId":"06689505-eae7-4a15-b9d0-a9d152aa68e0"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vFWSNK_41cafa88-f813-4eec-992e-30cecfa08d65","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"01fe980a-c9c9-4236-bc8b-291c99ae91e9","modelInfo":{"modelInvariantId":"ef354d84-8aa5-48d2-9f62-46d71f07ccb6","modelType":"service","modelName":"vFWCL 2018-05-25 00:46:","modelVersion":"1.0","modelVersionId":"5b0c9947-60ae-4eb7-91cf-7970cadfac5c"}}},{"relatedInstance":{"instanceId":"2eefeaf6-6ca4-421d-a77c-f03099a64797","modelInfo":{"modelCustomizationName":"c6709519-9421-42d2-a1de 0","modelInvariantId":"5146e5ae-a72b-4dd2-b08e-e667b9cd0021","modelType":"vnf","modelName":"c6709519-9421-42d2-a1de","modelVersion":"1.0","modelVersionId":"20d22b9d-ff94-4076-8cab-5c7227b7f95b","modelCustomizationId":"73a013d4-28b0-4e53-8b32-fdeb7a249617"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"01fe980a-c9c9-4236-bc8b-291c99ae91e9","vnfInstanceId":"2eefeaf6-6ca4-421d-a77c-f03099a64797","vfModuleInstanceName":"Vfmodule_Ete_vFWSNK_41cafa88-f813-4eec-992e-30cecfa08d65","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Fri, 25 May 2018 00:51:08 GMT"}}}' contains 'FAILED'
5/24/2018 16:00 - 20:00	Helen	41/43 Pass	21/24 Success	Pass	N/A	Health check failed AAF and VNFSDK (non-blocking). 13:30:12 OpenECOMP ETE.Robot.Testsuites.Vnf-Orchestration :: Executes the VNF Orches... 13:30:12 ============================================================================== 13:30:12 Instantiate Virtual DNS \| FAIL \| 13:35:22 '{"request":{"requestId":"18b9477d-2029-4f31-9f5c-84c657abed5e","startTime":"Thu, 24 May 2018 20:34:47 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"7e666c00E3e141a98c10..base_vlb..module-0","modelInvariantId":"a909be21-d4d4-4469-87ab-658ca3f27aad","modelType":"vfModule","modelName":"7e666c00E3e141a98c10..base_vlb..module- 13:41:42 Instantiate Virtual FirewallCL \| FAIL \| 2. 13:46:29 '{"request":{"requestId":"687869b9-686d-4453-b675-43a4712b3a6c","startTime":"Thu, 24 May 2018 20:45:41 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"590019f5Aa914919A328..base_vfw..module-0","modelInvariantId":"87fe21db-e3e8-49fc-a54d-4363c04cb562","modelType":"vfModule","modelName":"590019f5Aa914919A328..base_vfw..module-0","modelVersion":"1","modelVersionId":"f77386d5-5df0-4d87-a44d-0d422a020fdc","modelCustomizationId":"75a642b4-33df-4b31-bc04-9c1a77f12b38"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vFWSNK_90f95fb6-0660-41e9-8029-cf8c57ff3e8d","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"db09732b-4ea5-4d94-9129-bd7566afb140","modelInfo":{"modelInvariantId":"a798688f-ce2e-40cf-a90c-a1d6212d3a16","modelType":"service","modelName":"vFWCL 2018-05-24 20:41:","modelVersion":"1.0","modelVersionId":"23ebc0f6-1f87-4f23-bbf1-9ccae386144f"}}},{"relatedInstance":{"instanceId":"537d6c42-a4f3-4fde-885b-3ded41af874c","modelInfo":{"modelCustomizationName":"590019f5-aa91-4919-a328 0","modelInvariantId":"21b9f7ba-98df-47ff-8e33-c21002e89860","modelType":"vnf","modelName":"590019f5-aa91-4919-a328","modelVersion":"1.0","modelVersionId":"b680ce90-e870-4eab-8802-b4372c5d7dd3","modelCustomizationId":"5721b822-1ddd-4144-9f5b-17013e56f7a4"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"db09732b-4ea5-4d94-9129-bd7566afb140","vnfInstanceId":"537d6c42-a4f3-4fde-885b-3ded41af874c","vfModuleInstanceName":"Vfmodule_Ete_vFWSNK_90f95fb6-0660-41e9-8029-cf8c57ff3e8d","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 0 : Stack error (CREATE_FAILED): Resource CREATE failed: OverQuotaClient: resources.unprotected_private_network: Quota exceeded for resources: ['network'].\nNeutron server returns request_ids: ['req-73eebb1f-e968-412f-8de3-b57611079079'] - stack successfully deleted' rolledBack='true'","percentProgress":100,"finishTime":"Thu, 24 May 2018 20:46:24 GMT"}}}' contains 'FAILED' 13:46:29 Helen: There're two success creation in between. 3. 14:00:13 Instantiate Virtual DNS \| FAIL \| 14:05:34 '{"request":{"requestId":"00b099be-dfef-4820-ada6-937c4f8e310f","startTime":"Thu, 24 May 2018 21:04:46 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"0d0060be2ee9492082b3..base_vlb..module-0","modelInvariantId":"8919bd23-d3d5-4def-a25e-b1e962f95009","modelType":"vfModule","modelName":"0d0060be2ee9492082b3..base_vlb..module-0","modelVersion":"1","modelVersionId":"05a5cdb5-2388-415c-ac4b-b8b944fabbb4","modelCustomizationId":"44ea016f-608e-43a7-be3c-30a2cf1abac4"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vLB_5df5464a-8edd-453c-a76b-8d5eb0ec8a05","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"eddbcaeb-1105-470d-9a1e-150b7226cc64","modelInfo":{"modelInvariantId":"f23af2c1-8159-4ce0-b0fe-81bd719551d4","modelType":"service","modelName":"vLB 2018-05-24 21:00:22","modelVersion":"1.0","modelVersionId":"26f9791a-d79d-4cf7-ab26-52194d3d2f8f"}}},{"relatedInstance":{"instanceId":"ddec794b-e77b-4c5e-8497-b9229aba2034","modelInfo":{"modelCustomizationName":"0d0060be-2ee9-4920-82b3 0","modelInvariantId":"f20fdbad-56e4-4705-88f9-93ef119cdfab","modelType":"vnf","modelName":"0d0060be-2ee9-4920-82b3","modelVersion":"1.0","modelVersionId":"360b1ea0-b0e1-40c6-acd8-98cca629bbf1","modelCustomizationId":"b0911f68-67e0-418d-8890-852fcb83a0d7"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"eddbcaeb-1105-470d-9a1e-150b7226cc64","vnfInstanceId":"ddec794b-e77b-4c5e-8497-b9229aba2034","vfModuleInstanceName":"Vfmodule_Ete_vLB_5df5464a-8edd-453c-a76b-8d5eb0ec8a05","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 0 : Stack error (CREATE_FAILED): Resource CREATE failed: OverQuotaClient: resources.pktgen_private_network: Quota exceeded for resources: ['network'].\nNeutron server returns request_ids: ['req-84d2ebb7-5ea3-49d0-9744-36b25d13e1e1'] - stack successfully deleted' rolledBack='true'","percentProgress":100,"finishTime":"Thu, 24 May 2018 21:05:30 GMT"}}}' contains 'FAILED' (Helen: we checked manually and found that there're a few stacks were not deleted successfully, therefore, also leave some network not deleted)
5/24/2018 11:23 - 16:00	Helen	41/43	25/30 Pass	Pass	N/A	Health check failed AAF and VNFSDK (non-blocking). 5. Instantiate Virtual FirewallCL \| FAIL \| Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: Element 'xpath=//select[@prompt='Select Service Type']' did not appear in 2 minutes 4. Instantiate Virtual FirewallCL \| FAIL \| 11:23:53 '{"request":{"requestId":"b08c3088-dbf4-4342-88be-5e537db384a6","startTime":"Thu, 24 May 2018 18:22:49 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"1029613b2836445c9931..base_vfw..module-0","modelInvariantId":"6381fe76-761a-49a8-9cbb-0223070c9f19","modelType":"vfModule","modelName":"1029613b2836445c9931..base_vfw..module-0","modelVersion":"1","modelVersionId":"4009b688-90f2-4aa6-878e-192e4a6d79a3","modelCustomizationId":"b79a2a67-8fe9-41af-a3ce-ad328089acdd"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vFWSNK_69760123-ec17-419e-a6fa-231b841899a6","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"580e9598-8b20-4a8c-bf04-4fc78cd31b12","modelInfo":{"modelInvariantId":"65587d70-c566-4f24-9eff-5d33237bf440","modelType":"service","modelName":"vFWCL 2018-05-24 18:18:","modelVersion":"1.0","modelVersionId":"5bb83533-95fe-4b8c-be2f-4aa85e2ff8d5"}}},{"relatedInstance":{"instanceId":"0c0509f9-3af7-4feb-b6a9-e77be590c097","modelInfo":{"modelCustomizationName":"1029613b-2836-445c-9931 0","modelInvariantId":"3b697274-5252-44f7-8f6d-e0dcbac42056","modelType":"vnf","modelNam e":"1029613b-2836-445c-9931","modelVersion":"1.0","modelVersionId":"3d39e56c-00fb-4183-8750-9638dff90b46","modelCustomizationId":"11567b54-17ab-4b4b-9dd6-45c4b9f7eb96"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"580e9598-8b20-4a8c-bf04-4fc78cd31b12","vnfInstanceId":"0c0509f9-3af7-4feb-b6a9-e77be590c097","vfModuleInstanceName":"Vfmodule_Ete_vFWSNK_69760123-ec17-419e-a6fa-231b841899a6","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Thu, 24 May 2018 18:24:01 GMT"}}}' contains 'FAILED' 3. Instantiate Virtual Volume Group 10:46:44 Instantiate Virtual Volume Group + retval=0 10:55:02 ++ ssh -o StrictHostKeychecking=no -i /var/lib/jenkins/.ssh/onap_key ubuntu@10.12.5.187 'sudo su -c "kubectl --namespace onap get pods"' 10:55:02 ++ grep robot 10:55:02 ++ sed 's/ .//' 10:55:04 Error from server (InternalError): an error on the server ("Service unavailable") has prevented the request from succeeding (get pods) 10:55:04 + ROBOT_POD= 10:55:04 + '[' '' == '' ']' 10:55:04 + exit 1 10:55:04 Build step 'Execute shell' marked build as failure 10:55:04 Robot results publisher started... 10:55:04 -Parsing output xml: 10:55:04 Failed! 10:55:04 hudson.AbortException: No files found in path /var/lib/jenkins/workspace/windriver-SB04-oom-stability72hr/archives with configured filemask: output.xml 10:55:04 at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:77) 10:55:04 at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:54) 10:55:04 at hudson.FilePath.act(FilePath.java:997) 10:55:04 at hudson.FilePath.act(FilePath.java:975) 10:55:04 at hudson.plugins.robot.RobotParser.parse(RobotParser.java:49) 10:55:04 at hudson.plugins.robot.RobotPublisher.parse(RobotPublisher.java:217) 10:55:04 at hudson.plugins.robot.RobotPublisher.perform(RobotPublisher.java:239) 10:55:04 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) 10:55:04 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) 10:55:04 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) 10:55:04 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) 10:55:04 at hudson.model.Build$BuildExecution.post2(Build.java:186) 10:55:04 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) 10:55:04 at hudson.model.Run.execute(Run.java:1749) 10:55:04 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 10:55:04 at hudson.model.ResourceController.execute(ResourceController.java:97) 10:55:04 at hudson.model.Executor.run(Executor.java:429) 10:55:04 Finished: FAILURE 2. Failed Test case:Instantiate Virtual FirewallCL* Keyword: 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: Element 'xpath=//select[@prompt='Select Service Type']' did not appear in 2 minutes Failed: Instantiate Virtual Volume Group, Error Message: Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: Element 'xpath=//select[@prompt='Select Service Type']' did not appear in 2 minutes
5/15: 11:00 – 12:00	Team	Not started		Pass	N/A	Failed healthcheck of OOF-Homing, ~~AAF (SMS)~~, External API (non-blocking for vFW and vLB use case) SMS Health check is pass from latest build (Kiran)