Summary:
The APPC 72 Hour Stability Test was kicked off at 10:30 pm ET on Thursday March 22, 2018. The APPC environment reamined up and processed requests during the entire test period without crashing.
Environment:
- WindRiver Lab, APPC Project
- Due to a defect that is blocking sending and receiving DMaaP messages in the Beijing code the Amsterdam release code was used during the APPC 72 Hour Stability test.
- The Final Completion notification for each transaction is not returned to create a final completion result for this test. The final completion will be integrated and tested with the Beijing code once defect APPC-658 is corrected.
Test Execution:
- The test was kicked off via cron set to run for 72 hours.
- Command to execute the test: ./jmeter.sh -n -t ../APPC-LCM-Action-V1.jmx -l ../APPC-LCM-Action-V1.jtl' > /dev/null 2>&1 &
Test Results:
- The test was executed for 72 hours with the APPC environment remaining up and successfully processing transactions during that time.
- The results from the test are logged in the APPC-LCM-Action-V1.jtl file on the Test Client Server
- The APPC application logs logged the results on the APPC side:
- karaf.log
- Service Logic logs (DG logs)
- Sample Stop Request:
From the JTL Log: 1521776330123,18013,Stop Transaction Controller,,"Number of samples in transaction : 4, number of failing samples : 3",Stability-Test-VM2 LCM Group 2-1,,false,,2231,2180,1,3,18013,90014,3 1521776438150,3,Submit request via DMaaP,200,OK,Stability-Test-VM2 LCM Group 2-1,text,true,,332,1223,1,3,3,0,0
VM STOP fromt he karaf.log:
2018-03-26 10:38:11,304 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5293 * Sending client request on thread appc-dispatcher-13420
5293 > GET http://10.12.25.2:8774/v2/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847
5293 > Accept: application/json
5293 > X-Auth-Token: 2dab36c8a9fb417a907cb7e05202008f
2018-03-26 10:38:11,736 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5293 * Client response received on thread appc-dispatcher-13420
5293 < 200
5293 < Content-Length: 2590
5293 < Content-Type: application/json
5293 < Date: Mon, 26 Mar 2018 10:38:19 GMT
5293 < X-Compute-Request-Id: req-84bb3a4f-9188-40cd-8091-ffb287fc7477
{"server": {"wrs-res:topology": "node:0, 2048MB, pgsize:2M, vcpus:0, pol:sha", "OS-EXT-STS:task_state": null, "addresses": {"appc-multicloud-integration": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:90:23:92", "version": 4, "addr": "10.10.5.14", "OS-EXT-IPS:type": "fixed"}], "external": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:35:03:47", "version": 4, "addr": "10.12.5.143", "OS-EXT-IPS:type": "fixed"}], "oam_onap_LH2Z": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:91:85:1d", "version": 4, "addr": "10.0.0.13", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://10.12.25.2:8774/v2/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847", "rel": "self"}, {"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847", "rel": "bookmark"}], "image": {"id": "f9166f14-c428-4080-895e-308f998f587e", "links": [{"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/images/f9166f14-c428-4080-895e-308f998f587e", "rel": "bookmark"}]}, "wrs-if:nics": [{"nic1": {"vif_model": "virtio", "network": "appc-multicloud-integration", "port_id": "ca84a4cb-f6cb-4fb4-a493-2924e6bed4a6", "mtu": 1500, "mac_address": "fa:16:3e:90:23:92", "vif_pci_address": ""}}, {"nic2": {"vif_model": "virtio", "network": "external", "port_id": "d7f388c7-97cf-459e-9081-1d08783c9557", "mtu": 1500, "mac_address": "fa:16:3e:35:03:47", "vif_pci_address": ""}}, {"nic3": {"vif_model": "virtio", "network": "oam_onap_LH2Z", "port_id": "ea4d9c9e-e480-4e3c-9137-34441f28bc60", "mtu": 1500, "mac_address": "fa:16:3e:91:85:1d", "vif_pci_address": ""}}], "wrs-sg:server_group": "", "OS-EXT-STS:vm_state": "active", "OS-SRV-USG:launched_at": "2018-03-14T14:59:52.000000", "flavor": {"id": "1a0d1e0a-9e6d-4dfc-a0d8-73aa2ef794e6", "links": [{"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/flavors/1a0d1e0a-9e6d-4dfc-a0d8-73aa2ef794e6", "rel": "bookmark"}]}, "id": "c0d9b8c7-9534-4a22-b464-d6aa2c930847", "user_id": "6a87d2aa015c4fc5b6a6f08a9a15b006", "OS-DCF:diskConfig": "AUTO", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "ACTIVE", "updated": "2018-03-26T10:37:51Z", "hostId": "f51e1768f5a01d1f7dd48a7124a9b3ebc10e57449ef3650c71566402", "OS-SRV-USG:terminated_at": null, "wrs-res:pci_devices": "", "wrs-res:vcpus": [1, 1, 1], "key_name": "onap_key_NhWI", "name": "Stability-Test-VM2", "created": "2018-02-13T17:29:33Z", "tenant_id": "cf142fb96ce24770946c0f7117c0aeb3", "os-extended-volumes:volumes_attached": [], "config_drive": ""}}
2018-03-26 10:38:11,743 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5294 * Sending client request on thread appc-dispatcher-13420
5294 > POST http://10.12.25.2:8774/v2/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847/action
5294 > Accept: application/json
5294 > Content-Type: application/json
5294 > X-Auth-Token: 2dab36c8a9fb417a907cb7e05202008f
{
"os-stop" : { }
}
2018-03-26 10:38:12,022 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5294 * Client response received on thread appc-dispatcher-13420
5294 < 202
5294 < Content-Length: 0
5294 < Content-Type: application/json
5294 < Date: Mon, 26 Mar 2018 10:38:20 GMT
5294 < Openstack-Api-Version: compute 2.1
5294 < Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version
5294 < X-Compute-Request-Id: req-b5317e37-1620-4089-8333-4bd05ca6f71a
5294 < X-Openstack-Nova-Api-Version: 2.1
2018-03-26 10:38:12,028 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5295 * Sending client request on thread appc-dispatcher-13420
5295 > GET http://10.12.25.2:8774/v2/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847
5295 > Accept: application/json
5295 > X-Auth-Token: 2dab36c8a9fb417a907cb7e05202008f
2018-03-26 10:38:32,540 | INFO | dispatcher-13420 | Keystone | 401 - appc-iaas-adapter - 1.2.0 | 5296 * Client response received on thread appc-dispatcher-13420
5296 < 200
5296 < Content-Length: 2577
5296 < Content-Type: application/json
5296 < Date: Mon, 26 Mar 2018 10:38:40 GMT
5296 < X-Compute-Request-Id: req-a155b129-0c50-4b22-9db4-81633298052e
{"server": {"wrs-res:topology": "node:0, 2048MB, pgsize:2M, vcpus:0, pol:sha", "OS-EXT-STS:task_state": null, "addresses": {"appc-multicloud-integration": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:90:23:92", "version": 4, "addr": "10.10.5.14", "OS-EXT-IPS:type": "fixed"}], "external": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:35:03:47", "version": 4, "addr": "10.12.5.143", "OS-EXT-IPS:type": "fixed"}], "oam_onap_LH2Z": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:91:85:1d", "version": 4, "addr": "10.0.0.13", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://10.12.25.2:8774/v2/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847", "rel": "self"}, {"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847", "rel": "bookmark"}], "image": {"id": "f9166f14-c428-4080-895e-308f998f587e", "links": [{"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/images/f9166f14-c428-4080-895e-308f998f587e", "rel": "bookmark"}]}, "wrs-if:nics": [{"nic1": {"vif_model": "virtio", "network": "appc-multicloud-integration", "port_id": "ca84a4cb-f6cb-4fb4-a493-2924e6bed4a6", "mtu": 1500, "mac_address": "fa:16:3e:90:23:92", "vif_pci_address": ""}}, {"nic2": {"vif_model": "virtio", "network": "external", "port_id": "d7f388c7-97cf-459e-9081-1d08783c9557", "mtu": 1500, "mac_address": "fa:16:3e:35:03:47", "vif_pci_address": ""}}, {"nic3": {"vif_model": "virtio", "network": "oam_onap_LH2Z", "port_id": "ea4d9c9e-e480-4e3c-9137-34441f28bc60", "mtu": 1500, "mac_address": "fa:16:3e:91:85:1d", "vif_pci_address": ""}}], "wrs-sg:server_group": "", "OS-EXT-STS:vm_state": "stopped", "OS-SRV-USG:launched_at": "2018-03-14T14:59:52.000000", "flavor": {"id": "1a0d1e0a-9e6d-4dfc-a0d8-73aa2ef794e6", "links": [{"href": "http://10.12.25.2:8774/cf142fb96ce24770946c0f7117c0aeb3/flavors/1a0d1e0a-9e6d-4dfc-a0d8-73aa2ef794e6", "rel": "bookmark"}]}, "id": "c0d9b8c7-9534-4a22-b464-d6aa2c930847", "user_id": "6a87d2aa015c4fc5b6a6f08a9a15b006", "OS-DCF:diskConfig": "AUTO", "accessIPv4": "", "accessIPv6": "", "OS-EXT-STS:power_state": 4, "OS-EXT-AZ:availability_zone": "nova", "metadata": {}, "status": "SHUTOFF", "updated": "2018-03-26T10:38:22Z", "hostId": "f51e1768f5a01d1f7dd48a7124a9b3ebc10e57449ef3650c71566402", "OS-SRV-USG:terminated_at": null, "wrs-res:pci_devices": "", "wrs-res:vcpus": [1, 1, 1], "key_name": "onap_key_NhWI", "name": "Stability-Test-VM2", "created": "2018-02-13T17:29:33Z", "tenant_id": "cf142fb96ce24770946c0f7117c0aeb3", "os-extended-volumes:volumes_attached": [], "config_drive": ""}}
2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | SvcLogicServiceImpl | 295 - org.onap.ccsdk.sli.core.sli-provider - 0.1.2 | About to execute node # 2 (switch)
2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | SvcLogicServiceImpl | 295 - org.onap.ccsdk.sli.core.sli-provider - 0.1.2 | About to execute node # 3 (block)
2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = STOP_STATUS -- SUCCESS
2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = VNFCTYPE --
2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = input.payload -- {"vm-id":"http://10.12.25.2:8774/v2.1/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847", "identity-url": "http://10.12.25.2:5000/v2.0","skip-hypervisor-check":"true"}
2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = vnf-name -- Stability-Test-VM2
2018-03-26 10:38:32,541 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = action-identifier -- vserver-id
2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = vnf-host-ip-address --
2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = success-message -- APPC0136I Operation StopServer succeed for http://10.12.25.2:8774/v2.1/cf142fb96ce24770946c0f7117c0aeb3/servers/c0d9b8c7-9534-4a22-b464-d6aa2c930847
2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = request-action -- Stop
2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = vnf-id -- Stability-Test-VM2
2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = tmp.vnfInfo.vnf.vm-count -- 0
2018-03-26 10:38:32,542 | INFO | dispatcher-13420 | GraphExecutor | 360 - appc-common - 1.2.0 | NEW KEY = tmp.vnfInfo.status -- success