Beijing Release Stability Testing Status (w/ HEAT)

5/16 20 - 24 EDT

Gary

Pass (39/40)

Pass

Pass

Portal is live

ete.sh stability72hr

Current status: 100% Success!

vFWCL continued soak test

Current status: Success

Health Checks

39/40 Passed

OOM-Homing failed

5/16

4PM-8PM EDT

Marco

Pass (39/40)

Pass

Pass

Automated testing Issues

Date: 05-15-2018 @4PM-8PM EDT

Affected component: OOF

Test: Health check

Error: 404 != 200

Comment: Non-blocking issue.  A fix is available in Gerrit: https://gerrit.onap.org/r/#/c/47509/

Manual testing

Test: vFirewall instantiation and closed loop

Status: Success

Test: vFirewall teardown

Status: Success

Ongoing testing

Test: vFirewall instantiation and closed loop - 72-hour soaking test

Running since: 05-13-2018 @ 10PM EDT

Current status: Success (I had to reboot the vPacketGen VM. It seems that after many hour of work the vPacketGen gets stuck on 5 flows, which doesn't activate closed loop)

5/16 12:00-16:00PM

Yang

Pass

(39/40)

Pass

Pass

No issue was observed, except the health check for OOF-Homing failure (known issue)

5/16 8:00 AM - 12:00 PM

Brian

Pass

Pass

Pass

Cleaned up a vVG from yesterday that we had left in place.

Control Loop running normal

Stability72hr tests normal

5/16

3:00am -8:00AM

Ran

Pass (39/40)

Pass (12/13)

Pass

NA

(Created VLM & VSP on the onboarding page)

Didn't observe any issue.

ete.sh stability72hr

Current status: Success; no ONAP errors observed during this period.

Health Test Fail: OOF-Homing(Known issue).

Knowen issue "authentication" observe during the run only 1 time:

error.message=HEAT-E99001 Service nova is not available for resource type OS::Nova::KeyPair, reason: The request you have made requires authentication. (HTTP 401).

Stacks - I was able to observe Vfmodule on Stacks that should have been deleted(only one during my Shift).

5/15 20 - 24 EDT

Gary

Pass (39/40)

Pass (22/24)

Pass

ete.sh stability72hr

Current status: Success; no ONAP errors observed during this period.

Continued intermittent failures of "The request you have made requires authentication" from OpenStack. Failed in 2 test cases out of 24 = 8.3% error rate from OpenStack.

Had to manually delete one vFWSNK stack orphaned by the OpenStack errors.

vFWCL continued soak test

Current status: Success

Health Checks

39/40 Passed

OOM-Homing failed

5/15 16 - 20 EDT

Helen

Pass (39/40)

Pass (22/24)

Pass

Health Test Fail: OOF-Homing

Test Case: Instantiate Virtual Volume Group

Status: FAIL (critical)

Message: '{"request":{"requestId":"f5afa2df-40ba-4492-9b42-9c05d4215240","startTime":"Tue, 15 May 2018 22:16:42 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"c75edfa0-49a1-4503-b673-bba740a50cd5","vnfInstanceId":"ea3c464a-8cf1-4071-9927-9995de2fa616","vfModuleInstanceName":"Vfmodule_Ete_vVG_73c32a00-f705-47ca-baa3-b032d116679e","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 22:16:51 GMT"}}}' contains 'FAILED'

5/15 12-16

Yang

Pass (39/40)

Pass (6/8)

Pass

Pass

Health Check Test

OOF-Homing failed with 404 != 200 error, same as reported earlier

Robot Test

2 out 8 failed with the authentication error in Robot, the time are 13:33:36 and 15:03:37.

The issue is the same as reported repeatedly.

Closed Loop Test

worked fine during the period

Manual Test

Tested VNF onboarding, service creation and distribution with VoLTE use case, it worked fine

5/15 8 AM - 12 PM

Brian/Marco

39 out of 40 Pass

Pass

8/9

Pass*

VNF issue on Packet Generator not ONAP related

Automated testing issues

 Date: 05-15-2018 @12PM-4PM EDT

Affected component: OOF

Test: Health check

Error: 404 != 200

Comment: Non-blocking issue. Team identified the issue. A fix has been submitted and available in Gerrit: https://gerrit.onap.org/r/#/c/47509/

Date: 05-15-2018 @8:30AM EDT

Affected component: vLB

Test: Instantiation

Error:'{"request":{"requestId":"bc9dee7f-b9e1-45b8-91ad-5289e62f2cfa","startTime":"Tue, 15 May 2018 12:37:31 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"f7b4da8f-cd86-4be0-a822-a1b3026686e0","vnfInstanceId":"7936c473-959c-400c-bb40-3bd6707f1766","vfModuleInstanceName":"Vfmodule_Ete_vLB_a63e39c5-89f8-4a01-9ebf-9f1182a8d3b7","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=ResourceTypeUnavailable, error.message=HEAT-E99001 Service nova is not available for resource type OS::Nova::KeyPair, reason: The request you have made requires authentication. (HTTP 401) (Request-ID: req-07c0b84a-0cdb-4abc-9afa-af8c90655cb5)' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 12:37:39 GMT"}}}' contains 'FAILED'

Comment: Intermittent failure

Manual testing

Test: vFirewall instantiation and closed loop

Status: Success

Test: vFirewall teardown

Status: Success

Ongoing testing 

Test: vFirewall instantiation and closed loop - 72-hour soaking test

Running since: 05-13-2018 @ 10PM EDT

Current status: Success (I had to reboot the vPacketGen VM due to some input from outside that may have confused the vPacketGen)

Brian's notes:

Deleted 1 vFW and 1 vVG from failed to delete activity

Closed loop is not looking correct as of 8:29 AM - back to normal ~9:30 after restarting packet generator VNF.

Open

Open

Ran

to 5/15/2018 2:00 EDT

Helen

Pass (39/40)

Pass (14/15)

Pass

Test Case: Instantiate Virtual DNS

Status: FAIL (critical)

Message:

'{"request":{"requestId":"fb3322e5-ef59-417c-8891-814f3dde76b9","startTime":"Tue, 15 May 2018 05:07:31 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"c8038ce6-695d-4f96-9079-07dffcbb7f01","vnfInstanceId":"2449ea65-7044-4ccc-ae1b-28bd8e47c80e","vfModuleInstanceName":"Vfmodule_Ete_vLB_ca5da6e1-b2e8-42e0-99b7-15283803b3ea","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=StackValidationFailed, error.message=Property error: : resources.vpg_0.properties.flavor: : The request you have made requires authentication. (HTTP 401)' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 05:07:41 GMT"}}}' contains 'FAILED'

to 5/14/2018 24:00 EDT

Helen

Pass (39/40)

Pass (22/24)

Pass

5/14/2018 23:30 EDT

Test Case: Instantiate Virtual FirewallCL

Status:

'{"request":{"requestId":"348d71db-7846-4a01-9496-9d058e993387","startTime":"Tue, 15 May 2018 03:51:36 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"b3a63233-8321-4535-af44-a6599ea40877","vnfInstanceId":"9ac48053-7e1d-43e3-8734-c09a15dfabca","vfModuleInstanceName":"Vfmodule_Ete_vFWSNK_151b24fd-d8ba-4079-a4e4-68b0dae8cc8e","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 400 Bad Request: The server could not comply with the request since it is either malformed or otherwise incorrect., error.type=ResourceTypeUnavailable, error.message=HEAT-E99001 Service nova is not available for resource type OS::Nova::KeyPair, reason: The request you have made requires authentication. (HTTP 401) (Request-ID: req-bd852abb-421b-4a77-8bcf-04c6d433b8b6)' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 03:51:44 GMT"}}}' contains 'FAILED'

======================================================

5/14/2018 21:30 EDT:

Test Case: Instantiate Virtual Volume Group

Status: FAIL (critical)

Message:

'{"request":{"requestId":"b0d0e687-14c2-4002-a359-92341e6cc465","startTime":"Tue, 15 May 2018 01:46:29 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"5bd36e34-8923-4068-9d63-c62e0dd53ff3","vnfInstanceId":"992422ac-5652-4a46-a513-4bde82347ccd","vfModuleInstanceName":"Vfmodule_Ete_vVG_51f740af-8319-4c9f-8c27-a9a7979e2333","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Tue, 15 May 2018 01:46:37 GMT"}}}' contains 'FAILED'

5/14 4 PM to 8 PM EDT

Gary

Pass (39/40)

Pass (22/24)

Pass

Pass (Incomplete)

Automated Testing

ete.sh stability72hr

Current status: Success; no ONAP errors observed during this period.

Continued intermittent failures of "The request you have made requires authentication" from OpenStack. Failed in 2 test cases out of 24 = 8.3% error rate from OpenStack.

Had to manually delete one set of vFWSNK and vPKG stacks orphaned by the OpenStack errors.

vFWCL continued soak test

Current status: Success

Manual Testing

Only got as far as service creation in VID for vFWCL. Did not get to complete SDNC Preload. Everything attempted in the UI so far worked.

5/14 12PM-4PM EDT

Marco

38 out of 40 Passed

21 out of 24 test Passed

Pass

Pass/Fail

Automated testing issues

 Date: 05-14-2018 @12PM-4PM EDT

Affected component: AAF

Test: Health check

Error: ConnectionError: HTTPSConnectionPool(host='aaf.api.simpledemo.onap.org', port=8100): Max retries exceeded with url: /authz/perms/user/demo@people.osaaf.org (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff9bf31b590>: Failed to establish a new connection: [Errno 111] Connection refused',))

Comment: Non-blocking issue, a fix has been submitted to Gerrit

Date: 05-14-2018 @12PM-4PM EDT

Affected component: OOF

Test: Health check

Error: 404 != 200

Comment: Non-blocking issue

Date: 05-14-2018 @1PM EDT

Affected component: vVG

Test: Instantiation

Error: '{"request":{"requestId":"cd6440cc-4a8a-499a-995a-9cd1cedb6be5","startTime":"Mon, 14 May 2018 17:16:37 GMT","requestScope":"vfModule","requestType":"createInstance","instanceReferences":{"serviceInstanceId":"cc267011-9954-4a65-8f8b-c65b981092de","vnfInstanceId":"e9491247-cdb4-41a3-9fd7-3033215b45c6","vfModuleInstanceName":"Vfmodule_Ete_vVG_a591c63a-77ca-40e1-93b5-b22af2b4cdfe","requestorId":"demo"},"requestStatus":{"requestState":"FAILED","statusMessage":"Received vfModuleException from VnfAdapter: category='INTERNAL' message='Exception during create VF 0 null: null, error.type=null, error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Mon, 14 May 2018 17:17:00 GMT"}}}' contains 'FAILED'

Comment: Intermittent failure

Manual testing

Test: vFirewall instantiation and closed loop

Status: Success

Test: vFirewall teardown

Status: Success

Test: vLoadBalancer/vDNS closed loop

Status: Failed

Affected component: SO

Error: Exception caught mapping requestBody to RequestDetails com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "requestDetails"

Jira ticket: https://jira.onap.org/browse/SO-621 A fix has been submitted to Gerrit

Test: vLB/vDNS (for manual scaling use case) design, distribution, and instantiation

Status: Success

Ongoing testing

Test: vFirewall instantiation and closed loop - 72-hour soaking test

Running since: 05-13-2018 @ 10PM EDT

Current status: Success

Brian

Just an FYI: Here are the messages coming out of TCA to Policy

another vVG stack not deleted - wonder if that is correlated to the Openstack Authentication Error ?

Vfmodule_Ete_vVG_a591c63a-77ca-40e1-93b5-b22af2b4cdfe @ 1:30 PM EDT test (or so)

Yes it is http://10.12.5.33:88/logs/ETE_0473_stability72hr/log.html is the same VNF

5/14 8:00 AM

Brian

Pass

2 OS fails in last 10 test.

3X8 + 2X2 =

Pass

Each cycle is 3 tests (vFW, vVG, vLB) - the Openstack Error throws out one of those when it happens.

3X8 + 2x2 = 28 tests PASS out of 30 total tests 28/30 ~93% success if we count the Openstack errors.

100% if we ignore the Openstack errors.

5/14/2018 1:00AM-8:00AM

Ran

Pass

2/11 had issues of Open-Stack NET Please Observe the Notes =>

Pass

N/A

(1)

Issues while running the VFW(Occured after passing 3 Tests Flows):

http://10.12.5.33:88/logs/ETE_0401_stability72hr/log.html

Problem : 401 Issue found.

Assumption - Open stack Issue of NET Causing Authorization failed:

error.message=The request you have made requires authentication.' rolledBack='true'","percentProgress":100,"finishTime":"Mon, 14 May 2018 05:21:35 GMT"}}}' contains 'FAILED'

The next flow of VFW Passed:

(2)

Another Failure has occurred after 2 Tests Flow Passed:

Instantiate Virtual DNS -

error.message=The request you have made requires authentication

http://10.12.5.33:88/logs/ETE_0410_stability72hr/log.html

Looks like the same NET Issue we observe...

Agin after this Fail mention above the next Test Passed, PSB!

Note : Another Issue regarding not deleting the VM, PSB:

5/13: 23:34 – 1:00AM

Team

Pass

Pass

Pass

N/A