Beijing Release Stability Testing Status (w/ OOM)

Monitoring shift, EDT

5/24/2018: 12:00 - 24:00 (Helen / Gary)

5/25/2018: 00:00 - 2:00 (Helen / Gary), 2:00 - 22:00 (Marco), 22:00 - 24:00 (Helen)

5/26/2018 00:00 - 2:00 (Helen), 2:00 - 20:00 (Yang), 20:00 - 24:00 (Helen)

5/27/2018 00:00 – 12:00 (Helen)

Stability Testing Starts at 5/24/2018,11:23 EDT, at Intel / Windriver Lab, Integration-SB-04

Time (EDT)	Report By	Health Check	Robot Test (Every 30 minutes)	Closed Loop	Manual Check	Notes
5/26/2018 2:00-20:00	Yang	41/43		Pass	SDC Portal works fine	Health check failed on AAF and VNFSDK (non-blocking)
5/25/2018 2:00 -						Health check failed AAF and VNFSDK (non-blocking).
5/25/2018 0:00- 2:00	Helen	41/43	10/12	Pass	N/A	Health check failed AAF and VNFSDK (non-blocking). 1. 22:17:39 Instantiate Virtual FirewallCL \| FAIL \| 22:21:36 Keyword 'Check Catalog Service Distributed' failed after retrying for 3 minutes. The last error was: SO Test (No more error message) 2. 22:47:02 Instantiate Virtual FirewallCL + retval=0 22:56:31 ++ ssh -o StrictHostKeychecking=no -i /var/lib/jenkins/.ssh/onap_key ubuntu@10.12.5.187 'sudo su -c "kubectl --namespace onap get pods"' 22:56:31 ++ grep robot 22:56:31 ++ sed 's/ .//' 22:56:33* Error from server (InternalError): an error on the server ("Service unavailable") has prevented the request from succeeding (get pods) 22:56:33 + ROBOT_POD= 22:56:33 + '[' '' == '' ']' 22:56:33 + exit 1 22:56:33 Build step 'Execute shell' marked build as failure 22:56:33 Robot results publisher started... 22:56:33 -Parsing output xml: 22:56:33 Failed! 22:56:33 hudson.AbortException: No files found in path /var/lib/jenkins/workspace/windriver-SB04-oom-stability72hr/archives with configured filemask: output.xml 22:56:33 at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:77) 22:56:33 at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:54) 22:56:33 at hudson.FilePath.act(FilePath.java:997) 22:56:33 at hudson.FilePath.act(FilePath.java:975) 22:56:33 at hudson.plugins.robot.RobotParser.parse(RobotParser.java:49) 22:56:33 at hudson.plugins.robot.RobotPublisher.parse(RobotPublisher.java:217) 22:56:33 at hudson.plugins.robot.RobotPublisher.perform(RobotPublisher.java:239) 22:56:33 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) 22:56:33 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) 22:56:33 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) 22:56:33 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) 22:56:33 at hudson.model.Build$BuildExecution.post2(Build.java:186) 22:56:33 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) 22:56:33 at hudson.model.Run.execute(Run.java:1749) 22:56:33 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 22:56:33 at hudson.model.ResourceController.execute(ResourceController.java:97) 22:56:33 at hudson.model.Executor.run(Executor.java:429) 22:56:33 Finished: FAILURE
5/24/2018 20:00 - 24:00	Helen	41/43	21/24 Pass	Pass	N/A	Health check failed AAF and VNFSDK (non-blocking). 19:00:15 Instantiate Virtual DNS \| FAIL \| 19:05:24 '{"request":{"requestId":"ad291713-d59b-47c3-bde4-07e8b3a98b26","startTime":"Fri, 25 May 2018 02:04:50 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"2a5d75b64e3347fa84d5..base_vlb..module-0","modelInvariantId":"7e8c9464-c89d-4e7c-b1ca-ff9cdb5c592b","modelType":"vfModule","modelName":"2a5d75b64e3347fa84d5..base_vlb..module-0","modelVersion":"1","modelVersionId":"ef5672d6-7fa6-4cd7-a31b-e600933b2896","modelCustomizationId":"0aa6c664-4f3d-4ee4-a408-bc3134d6be76"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vLB_9d9631ba-ae23-4c11-bbc8-8748d1548a88","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"ebe5d310-911c-477c-8c5e-e7f7b693e0cb","modelInfo":{"modelInvariantId":"0b666635-c454-474a-86e3-f74705de0460","modelType":"service","modelName":"vLB 2018-05-25 02:00:24","modelVersion":"1.0","modelVersionId":"5ac5f370-4a81-4d59-ac83-ccf3ce347d9c"}}},{"relatedInstance":{"instanceId":"1c671246-505b-4660-834b-631fe40b16f8","modelInfo":{"modelCustomizationName":"2a5d75b6-4e33-47fa-84d5 0","modelInvariantId":"c7af8dce-91fd-426b-a084-24766beb1317","modelType":"vnf","modelName":"2a5d75b6-4e33-47fa-84d5","modelVersion":"1.0","modelVersionId":"bc931325-5c60-48ab-a85f-5c1280e4c4d8","modelCustomizationId":"66876b00-91ab-4cfe-929f-5cc2ebf665be"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"ebe5d310-911c-477c-8c5e-e7f7b693e0cb","vnfInstanceId":"1c671246-505b-4660-834b-631fe40b16f8","vfModuleInstanceName":"Vfmodule_Ete_vLB_9d9631ba-ae23-4c11-bbc8-8748d1548a88","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=StackValidationFailed, error.message=Property error: : resources.vlb_private_1_port.properties.fixed_ips[0].subnet: : Error validating value 'c7c8198e-e0f9-4b0f-a27d-316f97d7be98': 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}\nNeutron server returns request_ids: ['req-2764e9a9-638b-4ad3-8ecc-c8bfe37396cc']' rolledBack='true'","percentProgress":100,"finishTime":"Fri, 25 May 2018 02:05:21 GMT"}}}' contains 'FAILED' 19:05:24 2. 18:16:36 Instantiate Virtual FirewallCL \| FAIL \| 18:20:31 Keyword 'Check Catalog Service Distributed' failed after retrying for 3 minutes. The last error was: SO Test 3. 17:46:44 Instantiate Virtual FirewallCL \| FAIL \| 17:51:11 '{"request":{"requestId":"92576e97-095a-4e77-996a-8415b055443d","startTime":"Fri, 25 May 2018 00:50:37 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"C6709519942142d2A1de..base_vfw..module-0","modelInvariantId":"6273a5fd-578e-4ed0-bc6e-e5c00dd474af","modelType":"vfModule","modelName":"C6709519942142d2A1de..base_vfw..module-0","modelVersion":"1","modelVersionId":"65b45fa9-7031-43ec-9365-8d6d803cb73d","modelCustomizationId":"06689505-eae7-4a15-b9d0-a9d152aa68e0"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vFWSNK_41cafa88-f813-4eec-992e-30cecfa08d65","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"01fe980a-c9c9-4236-bc8b-291c99ae91e9","modelInfo":{"modelInvariantId":"ef354d84-8aa5-48d2-9f62-46d71f07ccb6","modelType":"service","modelName":"vFWCL 2018-05-25 00:46:","modelVersion":"1.0","modelVersionId":"5b0c9947-60ae-4eb7-91cf-7970cadfac5c"}}},{"relatedInstance":{"instanceId":"2eefeaf6-6ca4-421d-a77c-f03099a64797","modelInfo":{"modelCustomizationName":"c6709519-9421-42d2-a1de 0","modelInvariantId":"5146e5ae-a72b-4dd2-b08e-e667b9cd0021","modelType":"vnf","modelName":"c6709519-9421-42d2-a1de","modelVersion":"1.0","modelVersionId":"20d22b9d-ff94-4076-8cab-5c7227b7f95b","modelCustomizationId":"73a013d4-28b0-4e53-8b32-fdeb7a249617"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"01fe980a-c9c9-4236-bc8b-291c99ae91e9","vnfInstanceId":"2eefeaf6-6ca4-421d-a77c-f03099a64797","vfModuleInstanceName":"Vfmodule_Ete_vFWSNK_41cafa88-f813-4eec-992e-30cecfa08d65","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Fri, 25 May 2018 00:51:08 GMT"}}}' contains 'FAILED'
5/24/2018 16:00 - 20:00	Helen	41/43 Pass	21/24 Success	Pass	N/A	Health check failed AAF and VNFSDK (non-blocking). 13:30:12 OpenECOMP ETE.Robot.Testsuites.Vnf-Orchestration :: Executes the VNF Orches... 13:30:12 ============================================================================== 13:30:12 Instantiate Virtual DNS \| FAIL \| 13:35:22 '{"request":{"requestId":"18b9477d-2029-4f31-9f5c-84c657abed5e","startTime":"Thu, 24 May 2018 20:34:47 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"7e666c00E3e141a98c10..base_vlb..module-0","modelInvariantId":"a909be21-d4d4-4469-87ab-658ca3f27aad","modelType":"vfModule","modelName":"7e666c00E3e141a98c10..base_vlb..module- 13:41:42 Instantiate Virtual FirewallCL \| FAIL \| 2. 13:46:29 '{"request":{"requestId":"687869b9-686d-4453-b675-43a4712b3a6c","startTime":"Thu, 24 May 2018 20:45:41 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"590019f5Aa914919A328..base_vfw..module-0","modelInvariantId":"87fe21db-e3e8-49fc-a54d-4363c04cb562","modelType":"vfModule","modelName":"590019f5Aa914919A328..base_vfw..module-0","modelVersion":"1","modelVersionId":"f77386d5-5df0-4d87-a44d-0d422a020fdc","modelCustomizationId":"75a642b4-33df-4b31-bc04-9c1a77f12b38"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vFWSNK_90f95fb6-0660-41e9-8029-cf8c57ff3e8d","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"db09732b-4ea5-4d94-9129-bd7566afb140","modelInfo":{"modelInvariantId":"a798688f-ce2e-40cf-a90c-a1d6212d3a16","modelType":"service","modelName":"vFWCL 2018-05-24 20:41:","modelVersion":"1.0","modelVersionId":"23ebc0f6-1f87-4f23-bbf1-9ccae386144f"}}},{"relatedInstance":{"instanceId":"537d6c42-a4f3-4fde-885b-3ded41af874c","modelInfo":{"modelCustomizationName":"590019f5-aa91-4919-a328 0","modelInvariantId":"21b9f7ba-98df-47ff-8e33-c21002e89860","modelType":"vnf","modelName":"590019f5-aa91-4919-a328","modelVersion":"1.0","modelVersionId":"b680ce90-e870-4eab-8802-b4372c5d7dd3","modelCustomizationId":"5721b822-1ddd-4144-9f5b-17013e56f7a4"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"db09732b-4ea5-4d94-9129-bd7566afb140","vnfInstanceId":"537d6c42-a4f3-4fde-885b-3ded41af874c","vfModuleInstanceName":"Vfmodule_Ete_vFWSNK_90f95fb6-0660-41e9-8029-cf8c57ff3e8d","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 0 : Stack error (CREATE_FAILED): Resource CREATE failed: OverQuotaClient: resources.unprotected_private_network: Quota exceeded for resources: ['network'].\nNeutron server returns request_ids: ['req-73eebb1f-e968-412f-8de3-b57611079079'] - stack successfully deleted' rolledBack='true'","percentProgress":100,"finishTime":"Thu, 24 May 2018 20:46:24 GMT"}}}' contains 'FAILED' 13:46:29 Helen: There're two success creation in between. 3. 14:00:13 Instantiate Virtual DNS \| FAIL \| 14:05:34 '{"request":{"requestId":"00b099be-dfef-4820-ada6-937c4f8e310f","startTime":"Thu, 24 May 2018 21:04:46 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"0d0060be2ee9492082b3..base_vlb..module-0","modelInvariantId":"8919bd23-d3d5-4def-a25e-b1e962f95009","modelType":"vfModule","modelName":"0d0060be2ee9492082b3..base_vlb..module-0","modelVersion":"1","modelVersionId":"05a5cdb5-2388-415c-ac4b-b8b944fabbb4","modelCustomizationId":"44ea016f-608e-43a7-be3c-30a2cf1abac4"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vLB_5df5464a-8edd-453c-a76b-8d5eb0ec8a05","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"eddbcaeb-1105-470d-9a1e-150b7226cc64","modelInfo":{"modelInvariantId":"f23af2c1-8159-4ce0-b0fe-81bd719551d4","modelType":"service","modelName":"vLB 2018-05-24 21:00:22","modelVersion":"1.0","modelVersionId":"26f9791a-d79d-4cf7-ab26-52194d3d2f8f"}}},{"relatedInstance":{"instanceId":"ddec794b-e77b-4c5e-8497-b9229aba2034","modelInfo":{"modelCustomizationName":"0d0060be-2ee9-4920-82b3 0","modelInvariantId":"f20fdbad-56e4-4705-88f9-93ef119cdfab","modelType":"vnf","modelName":"0d0060be-2ee9-4920-82b3","modelVersion":"1.0","modelVersionId":"360b1ea0-b0e1-40c6-acd8-98cca629bbf1","modelCustomizationId":"b0911f68-67e0-418d-8890-852fcb83a0d7"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"eddbcaeb-1105-470d-9a1e-150b7226cc64","vnfInstanceId":"ddec794b-e77b-4c5e-8497-b9229aba2034","vfModuleInstanceName":"Vfmodule_Ete_vLB_5df5464a-8edd-453c-a76b-8d5eb0ec8a05","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 0 : Stack error (CREATE_FAILED): Resource CREATE failed: OverQuotaClient: resources.pktgen_private_network: Quota exceeded for resources: ['network'].\nNeutron server returns request_ids: ['req-84d2ebb7-5ea3-49d0-9744-36b25d13e1e1'] - stack successfully deleted' rolledBack='true'","percentProgress":100,"finishTime":"Thu, 24 May 2018 21:05:30 GMT"}}}' contains 'FAILED' (Helen: we checked manually and found that there're a few stacks were not deleted successfully, therefore, also leave some network not deleted)
5/24/2018 11:23 - 16:00	Helen	41/43	25/30 Pass	Pass	N/A	Health check failed AAF and VNFSDK (non-blocking). 5. Instantiate Virtual FirewallCL \| FAIL \| Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: Element 'xpath=//select[@prompt='Select Service Type']' did not appear in 2 minutes 4. Instantiate Virtual FirewallCL \| FAIL \| 11:23:53 '{"request":{"requestId":"b08c3088-dbf4-4342-88be-5e537db384a6","startTime":"Thu, 24 May 2018 18:22:49 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":"1029613b2836445c9931..base_vfw..module-0","modelInvariantId":"6381fe76-761a-49a8-9cbb-0223070c9f19","modelType":"vfModule","modelName":"1029613b2836445c9931..base_vfw..module-0","modelVersion":"1","modelVersionId":"4009b688-90f2-4aa6-878e-192e4a6d79a3","modelCustomizationId":"b79a2a67-8fe9-41af-a3ce-ad328089acdd"},"requestInfo":{"source":"VID","instanceName":"Vfmodule_Ete_vFWSNK_69760123-ec17-419e-a6fa-231b841899a6","suppressRollback":false,"requestorId":"demo"},"relatedInstanceList":[{"relatedInstance":{"instanceId":"580e9598-8b20-4a8c-bf04-4fc78cd31b12","modelInfo":{"modelInvariantId":"65587d70-c566-4f24-9eff-5d33237bf440","modelType":"service","modelName":"vFWCL 2018-05-24 18:18:","modelVersion":"1.0","modelVersionId":"5bb83533-95fe-4b8c-be2f-4aa85e2ff8d5"}}},{"relatedInstance":{"instanceId":"0c0509f9-3af7-4feb-b6a9-e77be590c097","modelInfo":{"modelCustomizationName":"1029613b-2836-445c-9931 0","modelInvariantId":"3b697274-5252-44f7-8f6d-e0dcbac42056","modelType":"vnf","modelNam e":"1029613b-2836-445c-9931","modelVersion":"1.0","modelVersionId":"3d39e56c-00fb-4183-8750-9638dff90b46","modelCustomizationId":"11567b54-17ab-4b4b-9dd6-45c4b9f7eb96"}}}],"cloudConfiguration":{"tenantId":"d570c718cbc545029f40e50b75eb13df","lcpCloudRegionId":"RegionOne"},"requestParameters":{"usePreload":true}},"instanceReferences":{"serviceInstanceId":"580e9598-8b20-4a8c-bf04-4fc78cd31b12","vnfInstanceId":"0c0509f9-3af7-4feb-b6a9-e77be590c097","vfModuleInstanceName":"Vfmodule_Ete_vFWSNK_69760123-ec17-419e-a6fa-231b841899a6","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Thu, 24 May 2018 18:24:01 GMT"}}}' contains 'FAILED' 3. Instantiate Virtual Volume Group 10:46:44 Instantiate Virtual Volume Group + retval=0 10:55:02 ++ ssh -o StrictHostKeychecking=no -i /var/lib/jenkins/.ssh/onap_key ubuntu@10.12.5.187 'sudo su -c "kubectl --namespace onap get pods"' 10:55:02 ++ grep robot 10:55:02 ++ sed 's/ .//' 10:55:04 Error from server (InternalError): an error on the server ("Service unavailable") has prevented the request from succeeding (get pods) 10:55:04 + ROBOT_POD= 10:55:04 + '[' '' == '' ']' 10:55:04 + exit 1 10:55:04 Build step 'Execute shell' marked build as failure 10:55:04 Robot results publisher started... 10:55:04 -Parsing output xml: 10:55:04 Failed! 10:55:04 hudson.AbortException: No files found in path /var/lib/jenkins/workspace/windriver-SB04-oom-stability72hr/archives with configured filemask: output.xml 10:55:04 at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:77) 10:55:04 at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:54) 10:55:04 at hudson.FilePath.act(FilePath.java:997) 10:55:04 at hudson.FilePath.act(FilePath.java:975) 10:55:04 at hudson.plugins.robot.RobotParser.parse(RobotParser.java:49) 10:55:04 at hudson.plugins.robot.RobotPublisher.parse(RobotPublisher.java:217) 10:55:04 at hudson.plugins.robot.RobotPublisher.perform(RobotPublisher.java:239) 10:55:04 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) 10:55:04 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) 10:55:04 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) 10:55:04 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) 10:55:04 at hudson.model.Build$BuildExecution.post2(Build.java:186) 10:55:04 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) 10:55:04 at hudson.model.Run.execute(Run.java:1749) 10:55:04 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 10:55:04 at hudson.model.ResourceController.execute(ResourceController.java:97) 10:55:04 at hudson.model.Executor.run(Executor.java:429) 10:55:04 Finished: FAILURE 2. Failed Test case:Instantiate Virtual FirewallCL* Keyword: 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: Element 'xpath=//select[@prompt='Select Service Type']' did not appear in 2 minutes Failed: Instantiate Virtual Volume Group, Error Message: Keyword 'Create VID Service Instance' failed after retrying for 5 minutes. The last error was: Element 'xpath=//select[@prompt='Select Service Type']' did not appear in 2 minutes
5/15: 11:00 – 12:00	Team	Not started		Pass	N/A	Failed healthcheck of OOF-Homing, ~~AAF (SMS)~~, External API (non-blocking for vFW and vLB use case) SMS Health check is pass from latest build (Kiran)