ONAP APPC 72 Hour Stability Test Results

Owned by Scott Seabolt
Last updated: May 17, 2018
14 min read



Summary:

The APPC 72 Hour Stability Test was kicked off at 10:30 pm ET on Thursday March 22, 2018.  The APPC environment reamined up and processed requests during the entire test period without crashing.

Environment: 

Test Execution: 

  • The test was kicked off via cron set to run for 72 hours. 

  • The fully automated "closed-lopp" type test wherein Athe test is kicked off and waits for the transaction completion to be received before sending another request will be enabled in the testing once APPC-658 is resolved.

  • Command to execute the test:  ./jmeter.sh -n -t ../APPC-LCM-Action-V1.jmx -l ../APPC-LCM-Action-V1.jtl' > /dev/null 2>&1 &

Test Results:

  • The test was executed for 72 hours with the APPC environment remaining up and successfully processing transactions during that time.

  • The results from the test are logged in the APPC-LCM-Action-V1.jtl file on the Test Client Server

  • The APPC application logs logged the results on the APPC side:

    • karaf.log

    • Service Logic logs (DG logs)

  • Notes/Observations:

    • The performance monitor collecting statistics for the APPC VM CPU, Memory etc... crashed during the test.  A cronjob will be created going forward to check for the existance of the client running on APPC and will restart the process if it is down.  This is not APPC code but is part of the JMeter Performance monitoring plugin.



Sample Stop Request: from the JTL Log
1521776330123,18013,Stop Transaction Controller,,"Number of samples in transaction : 4, number of failing samples : 3",Stability-Test-VM2 LCM Group 2-1,,false,,2231,2180,1,3,18013,90014,3 1521776438150,3,Submit request via DMaaP,200,OK,Stability-Test-VM2 LCM Group 2-1,text,true,,332,1223,1,3,3,0,0



VM STOP from the karaf.log
2018-03-26 10:38:11,304 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5293 * Sending client request on thread appc-dispatcher-13420 5293 > GET http://10.12.25.2:8774/v2/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847 5293 > Accept: application/json 5293 > X-Auth-Token: 2dab36c8a9fb417a907cb7e05202008f 2018-03-26 10:38:11,736 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5293 * Client response received on thread appc-dispatcher-13420 5293 < 200 5293 < Content-Length: 2590 5293 < Content-Type: application/json 5293 < Date: Mon, 26 Mar 2018 10:38:19 GMT 5293 < X-Compute-Request-Id: req-84bb3a4f-9188-40cd-8091-ffb287fc7477 {"server": {"wrs-res:topology": "node:0, 2048MB, pgsize:2M, vcpus:0, pol:sha", "OS-EXT-STS:task_state": null, "addresses": {"appc-multicloud-integration": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:90:23:92", "version": 4, "addr": "10.10.5.14", "OS-EXT-IPS:type": "fixed"}], "external": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:35:03:47", "version": 4, "addr": "10.12.5.143", "OS-EXT-IPS:type": "fixed"}], "oam_onap_LH2Z": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:91:85:1d", "version": 4, "addr": "10.0.0.13", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://10.12.25.2:8774/v2/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847", "rel": "self"}, {"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847", "rel": "bookmark"}], "image": {"id": "f9166f14-c428-4080-895e-308f998f587e", "links": [{"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/images/f9166f14-c428-4080-895e-308f998f587e", "rel": "bookmark"}]}, "wrs-if:nics": [{"nic1": {"vif_model": "virtio", "network": "appc-multicloud-integration", "port_id": "ca84a4cb-f6cb-4fb4-a493-2924e6bed4a6", "mtu": 1500, "mac_address": "fa:16:3e:90:23:92", "vif_pci_address": ""}}, {"nic2": {"vif_model": "virtio", "network": "external", "port_id": "d7f388c7-97cf-459e-9081-1d08783c9557", "mtu": 1500, "mac_address": "fa:16:3e:35:03:47", "vif_pci_address": ""}}, {"nic3": {"vif_model": "virtio", "network": "oam_onap_LH2Z", "port_id": "ea4d9c9e-e480-4e3c-9137-34441f28bc60", "mtu": 1500, "mac_address": "fa:16:3e:91:85:1d", "vif_pci_address": ""}}], "wrs-sg:server_group": "", "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2018-03-14T14:59:52.000000", "flavor": {"id": "1a0d1e0a-9e6d-4dfc-a0d8-73aa2ef794e6", "links": [{"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/flavors/1a0d1e0a-9e6d-4dfc-a0d8-73aa2ef794e6", "rel": "bookmark"}]}, "id": "c0d9b8c7-9534-4a22-b464-d6aa2c930847", "user_id": "6a87d2aa015c4fc5b6a6f08a9a15b006", "OS-DCF:diskConfig": "AUTO", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ACTIVE", "updated": "2018-03-26T10:37:51Z", "hostId": "f51e1768f5a01d1f7dd48a7124a9b3ebc10e57449ef3650c71566402", "OS-SRV-USG:terminated_at": null, "wrs-res:pci_devices": "", "wrs-res:vcpus": [1, 1, 1], "key_name": "onap_key_NhWI", "name": "Stability-Test-VM2", "created": "2018-02-13T17:29:33Z", "tenant_id": "cf142fb96ce24770946c0f7117c0aeb3", "os-extended-volumes:volumes_attached": [], "config_drive": ""}} 2018-03-26 10:38:11,743 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5294 * Sending client request on thread appc-dispatcher-13420 5294 > POST http://10.12.25.2:8774/v2/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847/action 5294 > Accept: application/json 5294 > Content-Type: application/json 5294 > X-Auth-Token: 2dab36c8a9fb417a907cb7e05202008f { "os-stop" : { } } 2018-03-26 10:38:12,022 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5294 * Client response received on thread appc-dispatcher-13420 5294 < 202 5294 < Content-Length: 0 5294 < Content-Type: application/json 5294 < Date: Mon, 26 Mar 2018 10:38:20 GMT 5294 < Openstack-Api-Version: compute 2.1 5294 < Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version 5294 < X-Compute-Request-Id: req-b5317e37-1620-4089-8333-4bd05ca6f71a 5294 < X-Openstack-Nova-Api-Version: 2.1 2018-03-26 10:38:12,028 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5295 * Sending client request on thread appc-dispatcher-13420 5295 > GET http://10.12.25.2:8774/v2/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847 5295 > Accept: application/json 5295 > X-Auth-Token: 2dab36c8a9fb417a907cb7e05202008f 2018-03-26 10:38:32,540 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5296 * Client response received on thread appc-dispatcher-13420 5296 < 200 5296 < Content-Length: 2577 5296 < Content-Type: application/json 5296 < Date: Mon, 26 Mar 2018 10:38:40 GMT 5296 < X-Compute-Request-Id: req-a155b129-0c50-4b22-9db4-81633298052e {"server": {"wrs-res:topology": "node:0, 2048MB, pgsize:2M, vcpus:0, pol:sha", "OS-EXT-STS:task_state": null, "addresses": {"appc-multicloud-integration": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:90:23:92", "version": 4, "addr": "10.10.5.14", "OS-EXT-IPS:type": "fixed"}], "external": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:35:03:47", "version": 4, "addr": "10.12.5.143", "OS-EXT-IPS:type": "fixed"}], "oam_onap_LH2Z": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:91:85:1d", "version": 4, "addr": "10.0.0.13", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://10.12.25.2:8774/v2/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847", "rel": "self"}, {"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847", "rel": "bookmark"}], "image": {"id": "f9166f14-c428-4080-895e-308f998f587e", "links": [{"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/images/f9166f14-c428-4080-895e-308f998f587e", "rel": "bookmark"}]}, "wrs-if:nics": [{"nic1": {"vif_model": "virtio", "network": "appc-multicloud-integration", "port_id": "ca84a4cb-f6cb-4fb4-a493-2924e6bed4a6", "mtu": 1500, "mac_address": "fa:16:3e:90:23:92", "vif_pci_address": ""}}, {"nic2": {"vif_model": "virtio", "network": "external", "port_id": "d7f388c7-97cf-459e-9081-1d08783c9557", "mtu": 1500, "mac_address": "fa:16:3e:35:03:47", "vif_pci_address": ""}}, {"nic3": {"vif_model": "virtio", "network": "oam_onap_LH2Z", "port_id": "ea4d9c9e-e480-4e3c-9137-34441f28bc60", "mtu": 1500, "mac_address": "fa:16:3e:91:85:1d", "vif_pci_address": ""}}], "wrs-sg:server_group": "", "OS-EXT-STS:vm_state": "stopped", "OS-SRV-USG:launched_at": "2018-03-14T14:59:52.000000", "flavor": {"id": "1a0d1e0a-9e6d-4dfc-a0d8-73aa2ef794e6", "links": [{"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/flavors/1a0d1e0a-9e6d-4dfc-a0d8-73aa2ef794e6", "rel": "bookmark"}]}, "id": "c0d9b8c7-9534-4a22-b464-d6aa2c930847", "user_id": "6a87d2aa015c4fc5b6a6f08a9a15b006", "OS-DCF:diskConfig": "AUTO", "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 4, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "SHUTOFF", "updated": "2018-03-26T10:38:22Z", "hostId": "f51e1768f5a01d1f7dd48a7124a9b3ebc10e57449ef3650c71566402", "OS-SRV-USG:terminated_at": null, "wrs-res:pci_devices": "", "wrs-res:vcpus": [1, 1, 1], "key_name": "onap_key_NhWI", "name": "Stability-Test-VM2", "created": "2018-02-13T17:29:33Z", "tenant_id": "cf142fb96ce24770946c0f7117c0aeb3", "os-extended-volumes:volumes_attached": [], "config_drive": ""}} 2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | SvcLogicServiceImpl | 295 - org.onap.ccsdk.sli.core.sli-provider - 0.1.2 | About to execute node # 2 (switch) 2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | SvcLogicServiceImpl | 295 - org.onap.ccsdk.sli.core.sli-provider - 0.1.2 | About to execute node # 3 (block) 2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = STOP_STATUS -- SUCCESS 2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = VNFCTYPE -- 2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = input.payload -- {"vm-id":"http://10.12.25.2:8774/v2.1/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847", "identity-url": "http://10.12.25.2:5000/v2.0","skip-hypervisor-check":"true"} 2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = vnf-name -- Stability-Test-VM2 2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = action-identifier -- vserver-id 2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = vnf-host-ip-address -- 2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = success-message -- APPC0136I Operation StopServer succeed for http://10.12.25.2:8774/v2.1/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847 2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = request-action -- Stop 2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = vnf-id -- Stability-Test-VM2 2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = tmp.vnfInfo.vnf.vm-count -- 0 2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = tmp.vnfInfo.status -- success
Sample Restart from the JTL Log
1521947136559,18013,Restart Transaction Controller,,"Number of samples in transaction : 4, number of failing samples : 3",Stability-Test-VM1 LCM Group 1-1,,false,,2238,2192,1,3,18013,9019,1 1521947163591,5,Submit request via DMaaP,200,OK,Stability-Test-VM1 LCM Group 1-1,text,true,,333,1232,1,3,5,0,0
Sample Rebuild from the JTL Log



Screenshot for Rebuild (Shows a Stop, Rebuild and Start command in the karaf.log):



From the karaf.log Rebuild (Shows Stop, Rebuild and Start)