...
Completed
We will use vFW use case as the baseline to test this:
Pre-requirement: Instantiate a vFW with closed loop running.
VNF Onboarding and Distribution: 97.8% (44/45) PASS
| Pod Deleted | Start Time (PST) | Recovery Time (MM:SS) | Pass / Fail | Logs | Notes | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | dev-aai-aai-babel-7cb986869d-9zcxf | 18:37:07 | 00:35 | PASS | ||||||||||
| 2 | dev-aai-aai-c9bc74cd9-szq8l | 18:47:58 | 01:12 | PASS | ||||||||||
| 3 | dev-aai-aai-cassandra-0 | 18:54:05 | 01:14 | PASS | ||||||||||
| 4 | dev-aai-aai-champ-6b6d4779d8-snrdj | 19:02:03 | 04:11 | PASS | ||||||||||
| 5 | dev-aai-aai-data-router-6b4776bfc4-kfrs4 | 00:23:49 | 01:10 | PASS | ||||||||||
| 6 | dev-aai-aai-elasticsearch-669f4664c-5rqll | 19:12:48 | 01:05 | PASS | ||||||||||
| 7 | dev-aai-aai-gizmo-9df94f898-7vbh5 | 19:19:27 | 01:15 | PASS | ||||||||||
| 8 | dev-aai-aai-graphadmin-588cd58c76-vrm84 | 19:24:12 | 01:40 | PASS | ||||||||||
| 9 | dev-aai-aai-modelloader-54f7444dcd-mbqgq | 19:31:09 | 01:04 | PASS | ||||||||||
| 10 | dev-aai-aai-resources-6b9c446d7-rnm56 | 19:38:15 | 02:13 | PASS | ||||||||||
| 11 | dev-aai-aai-search-data-695b878655-b2z7n | 19:44:18 | 01:04 | PASS | ||||||||||
| 12 | dev-aai-aai-sparky-be-9bcc99756-f7796 | 19:51:22 | 03:54 | PASS | ||||||||||
| 13 | dev-aai-aai-spike-59c6db6574-phz9h | 19:59:18 | 01:06 | PASS | ||||||||||
| 14 | dev-aai-aai-traversal-7bd85969cf-mcg24 | 20:05:07 | 01:46 | PASS | ||||||||||
| 15 | dev-sdc-sdc-be-6d8f57d5c4-g6wg5 | 20:11:09 | 03:09 | PASS | ||||||||||
| 16 | dev-sdc-sdc-cs-7768fcf944-fs4ph | 00:28:46 | 35:07 | FAIL | After deletion of sdc-cs, sdc-be pod falls into 1/2 Running status and never recovered. sdc-be required manual pod deletion to recover. Logged | |||||||||
| 17 | dev-sdc-sdc-dcae-be-5bb587d95c-465l9 | 20:45:34 | 23:03 | PASS | ||||||||||
| 18 | dev-sdc-sdc-dcae-dt-68d8d77578-5fn69 | 21:12:47 | 01:14 | PASS | ||||||||||
| 19 | dev-sdc-sdc-dcae-fe-79675b4bc4-lgh65 | 21:18:46 | 01:14 | PASS | ||||||||||
| 20 | dev-sdc-sdc-dcae-tosca-lab-6d9f59cbf6-lsljh | 21:24:35 | 01:08 | PASS | ||||||||||
| 21 | dev-sdc-sdc-es-6d9566f787-j47fj | 21:29:39 | 03:48 | PASS | ||||||||||
| 22 | dev-sdc-sdc-fe-7c7c4577d9-jrxtg | 21:38:31 | 01:45 | PASS | ||||||||||
| 23 | dev-sdc-sdc-onboarding-be-576864dfbc-89zxt | 21:45:40 | 02:43 | PASS | ||||||||||
| 24 | dev-sdc-sdc-wfd-be-7b84b4bf97-hsgkd | 21:54:35 | 02:09 | PASS | ||||||||||
| 25 | dev-sdc-sdc-wfd-fe-86597867d6-4jtzh | 22:00:41 | 01:35 | PASS | ||||||||||
| 26 | dev-sdnc-controller-blueprints-684f7d865f-krmhc | 22:07:41 | 01:12 | PASS | ||||||||||
| 27 | dev-sdnc-controller-blueprints-db-0 | 22:14:35 | 01:46 | PASS | ||||||||||
| 28 | dev-sdnc-nengdb-0 | 22:21:36 | 00:37 | PASS | ||||||||||
| 29 | dev-sdnc-network-name-gen-7f54b5649c-b2h7n | 22:27:39 | 01:02 | PASS | ||||||||||
| 30 | dev-sdnc-sdnc-0 | 22:32:49 | 02:12 | PASS | ||||||||||
| 31 | dev-sdnc-sdnc-db-0 | 22:40:02 | 03:47 | PASS | ||||||||||
| 32 | dev-sdnc-sdnc-dgbuilder-77866dbfb5-fb5fd | 22:47:31 | 02:09 | PASS | ||||||||||
| 33 | dev-sdnc-sdnc-dmaap-listener-6f67f97dd9-nttl7 | 22:53:49 | 00:45 | PASS | ||||||||||
| 34 | dev-sdnc-sdnc-portal-79f65cfcf8-7q7h6 | 22:59:32 | 05:15 | PASS | ||||||||||
| 35 | dev-sdnc-sdnc-ueb-listener-557c57577c-th896 | 23:08:44 | 01:04 | PASS | ||||||||||
| 36 | dev-so-so-6fcf596b6b-w2dsj | 23:16:44 | 01:06 | PASS | ||||||||||
| 37 | dev-so-so-bpmn-infra-6979cdb87d-lpm5g | 23:22:36 | 01:03 | PASS | ||||||||||
| 38 | dev-so-so-catalog-db-adapter-566fdc66c-vtvp5 | 23:27:39 | 01:04 | PASS | ||||||||||
| 39 | dev-so-so-mariadb-576ccb887f-nf8j6 | 23:32:48 | 09:05 | PASS | ||||||||||
| 40 | dev-so-so-monitoring-6c5b768dc4-c975x | 23:46:50 | 01:03 | PASS | ||||||||||
| 41 | dev-so-so-openstack-adapter-8465dc7d79-7llmh | 23:53:43 | 01:09 | PASS | ||||||||||
| 42 | dev-so-so-request-db-adapter-86d4477b55-wxdqv | 00:00:48 | 01:12 | PASS | ||||||||||
| 43 | dev-so-so-sdc-controller-86d59b9f4f-7x5gz | 00:07:47 | 01:13 | PASS | ||||||||||
| 44 | dev-so-so-sdnc-adapter-6b7cdc9b-gnkzw | 00:12:41 | 01:11 | PASS | ||||||||||
| 45 | dev-so-so-vfc-adapter-85d6f88ff6-4xk6b | 00:18:30 | 01:34 | PASS |
VNF Instantiation: 96.3% (52/54) PASS
| Pod Deleted | Start Time (PST) | Recovery Time (MM:SS) | Pass / Fail | Logs | Notes | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | dev-aai-aai-babel-7cb986869d-zkf8r | 21:58:32 | 01:11 | PASS | ||||||||||
| 2 | dev-aai-aai-cassandra-0 | 22:14:59 | 01:46 | PASS | ||||||||||
| 3 | dev-aai-aai-data-router-6b4776bfc4-lssst | 22:26:45 | 01:12 | PASS | ||||||||||
| 4 | dev-aai-aai-elasticsearch-669f4664c-h5fh4 | 22:40:00 | 13:30 | PASS | ||||||||||
| 5 | dev-aai-aai-ff69fc688-r5hc9 | 23:04:30 | 01:12 | PASS | ||||||||||
| 6 | dev-aai-aai-gizmo-9df94f898-64j9t | 23:18:32 | 13:26 | PASS | ||||||||||
| 7 | dev-aai-aai-graphadmin-5d6d4d86dc-7wvgs | 23:45:45 | 06:51 | PASS | ||||||||||
| 8 | dev-aai-aai-modelloader-54f7444dcd-xw54c | 00:06:17 | 01:04 | PASS | ||||||||||
| 9 | dev-aai-aai-resources-8579748958-jqg9x | 00:20:41 | 01:48 | PASS | ||||||||||
| 10 | dev-aai-aai-search-data-695b878655-wfr5l | 00:36:00 | 01:09 | PASS | ||||||||||
| 11 | dev-aai-aai-sparky-be-9bcc99756-c8lvk879q9 | 1620:2729:3345 | 0205:4446 | PASS | 5610/ | |||||||||
| 12 | dev-aai-aai-spike-59c6db6574-nj465 | 00:58:25 | 01:07 | PASS | ||||||||||
| 13 | dev-aai-aai-traversal-7f5669d56-xwvlr | 01:12:58 | 02:09 | PASS | ||||||||||
| 14 | dev-multicloud-multicloud-7cd96f75fb-vzmq7 | 01:28:35 | 01:09 | PASS | ||||||||||
| 15 | dev-multicloud-multicloud-azure-85548b6469-d6kz9 | 01:41:41 | 01:03 | PASS | ||||||||||
| 16 | dev-multicloud-multicloud-ocata-85d796889b-ctvd9 | 18:44:00 | 01:04 | PASS | ||||||||||
| 17 | dev-multicloud-multicloud-pike-78fb779f6c-vn789 | 02:04:48 | 01:06 | PASS | ||||||||||
| 18 | dev-multicloud-multicloud-vio-794f8b7d85-hwnjg | 02:18:16 | 04:16 | PASS | ||||||||||
| 19 | dev-multicloud-multicloud-windriver-6ffb4dcd-dhnlf | 02:35:08 | 01:04 | PASS | ||||||||||
| 20 | dev-sdc-sdc-be-6d8f57d5c4-dqs8w | 02:49:07 | 03:16 | PASS | ||||||||||
| 21 | dev-sdc-sdc-cs-7768fcf944-cg75x | 03:05:30 | 26:25 | PASS | ||||||||||
| 22 | dev-sdc-sdc-dcae-be-5bb587d95c-dgtwj | 03:44:55 | 05:18 | PASS | ||||||||||
| 23 | dev-sdc-sdc-dcae-dt-68d8d77578-plvjn | 04:02:51 | 01:17 | PASS | ||||||||||
| 24 | dev-sdc-sdc-dcae-fe-79675b4bc4-78cgd | 04:17:54 | 02:14 | PASS | ||||||||||
| 25 | dev-sdc-sdc-dcae-tosca-lab-6d9f59cbf6-fj7jp | 04:33:40 | 02:19 | PASS | ||||||||||
| 26 | dev-sdc-sdc-es-6d9566f787-dvgx6 | 04:50:34 | 03:08 | PASS | ||||||||||
| 27 | dev-sdc-sdc-fe-7c7c4577d9-bpdhh | 05:08:04 | 01:38 | PASS | ||||||||||
| 28 | dev-sdc-sdc-kb-85f8df694-ghdvmkns5t | 1720:4919:4148 | 01:3811 | PASS | 619/ | Failed due to other prior failures. Re-test pending. | ||||||||
| 29 | dev-sdc-sdc-onboarding-be-576864dfbc-x8pt8 | 05:40:05 | 09:03 | PASS | ||||||||||
| 30 | dev-sdc-sdc-wfd-be-7b84b4bf97-stlxs | 06:04:46 | 02:11 | PASS | ||||||||||
| 31 | dev-sdc-sdc-wfd-fe-86597867d6-hw725 | 06:22:13 | 01:41 | PASS | ||||||||||
| 32 | dev-sdnc-controller-blueprints-684f7d865f-4pglc | 06:39:13 | 01:08 | PASS | ||||||||||
| 33 | dev-sdnc-controller-blueprints-db-0 | 06:55:57 | 01:48 | PASS | ||||||||||
| 34 | dev-sdnc-nengdb-0 | 07:12:23 | 01:42 | PASS | ||||||||||
| 35 | dev-sdnc-network-name-gen-7f54b5649c-w4j8x | 07:28:47 | 02:37 | PASS | ||||||||||
| 36 | dev-sdnc-sdnc-0 | 07:46:43 | 03:45 | PASS | ||||||||||
| 37 | dev-sdnc-sdnc-ansible-server-8b58d86f5-d6hz2 | 08:05:46 | 03:16 | PASS | ||||||||||
| 38 | dev-sdnc-sdnc-db-0 | 08:24:10 | 03:52 | FAIL | After sdnc-sdnc-db was deleted, the SDNC portal continues to fail due to inability to connect to the DB until sdnc-portal was manually restarted. Logged | |||||||||
| 39 | dev-sdnc-sdnc-dgbuilder-77866dbfb5-44jdswk4zj | 20:08:3209 | 02:4309:09 | PASS | 408/ | |||||||||
| 40 | dev-sdnc-sdnc-dmaap-listener-6f67f97dd9-dv5r7crfxr | 0819:4658:4125 | 01:0605 | PASS | 417/ | Failed due to other prior failures. Re-test pending.|||||||||
| 41 | dev-sdnc-sdnc-portal-79f65cfcf8-4jh8bl8nq8 | 0819:5246:3336 | 03:4107 | PASS | 426/ | |||||||||
| 42 | dev-sdnc-sdnc-ueb-listener-557c57577c-knsskhzt2v | 0919:35:08 | 02:254705:20 | PASS | 435/ | Failed due to other prior failures. Re-test pending.|||||||||
| 43 | dev-so-so-575dcc7fc5-8g626stw5p | 0919:1225:3216 | 01:0506 | PASS | 444/ | |||||||||
| 44 | dev-so-so-bpmn-infra-74875c7c-l6pl96k4lz | 0919:1815:2700 | 01:1411 | PASS | 453/ | Failed due to other prior failures. Re-test pending.|||||||||
| 45 | dev-so-so-catalog-db-adapter-67878b4fc8-mzwg2 | 09:24:36 | 03:53 | PASS | ||||||||||
| 46 | dev-so-so-mariadb-576ccb887f-vh2f2 | 09:45:07 | 08:27 | PASS | ||||||||||
| 47 | dev-so-so-monitoring-7b4fdbf895-szgtp | 10:11:45 | 01:40 | PASS | ||||||||||
| 48 | dev-so-so-openstack-adapter-85576544b6-xmd62 | 10:28:14 | 01:35 | PASS | ||||||||||
| 49 | dev-so-so-request-db-adapter-658db4b75d-nk7kpgfn6g | 1719:3105:5703 | 01:1611 | PASS | 592/ | |||||||||
| 50 | dev-so-so-sdc-controller-679f5dff6d-rxgf5 | 14:50:51 | 01:17 | PASS | ||||||||||
| 51 | dev-so-so-sdnc-adapter-54d99774c9-qgp6w | 15:10:15 | 01:05 | PASS | ||||||||||
| 52 | dev-so-so-vfc-adapter-5bb49bc97f-7rvzj | 15:31:05 | 01:03 | PASS | ||||||||||
| 53 | dev-vid-vid-54f8c6d78-jj8vp | 15:49:13 | 04:48 | PASS | ||||||||||
| 54 | dev-vid-vid-mariadb-galera-0 | 16:21:42 | 01:09 | FAIL | After vid-mariadb-galera was deleted, the instance creation via VID continue to fail even after the vid pod |
...
was also restarted. Logged
|
Closed Loop: 96.7% (59/61) PASS
| Pod Deleted | Start Time (PST) | Recovery Time (MM:SS) | Pass / Fail | Logs | Notes | |
|---|---|---|---|---|---|---|
| 1 | dep-config-binding-service-596688f976-tdrpz | 05:29:55 | 02:08 | PASS | ||
| 2 | dep-dcae-datafile-collector-65df56bc6b-4cj5p | 05:35:36 | 01:14 | PASS | ||
| 3 | dep-dcae-hv-ves-collector-9df8d946-nz85f | 05:39:33 | 00:45 | PASS | ||
| 4 | dep-dcae-prh-6bfbb97fdc-n2pvb | 05:43:18 | 01:09 | PASS | ||
| 5 | dep-dcae-snmptrap-collector-58bbcb6644-g9tkw | 05:47:10 | 02:36 | PASS | ||
| 6 | dep-dcae-tca-analytics-7dbddb9697-d2h8m | 05:51:14 | 03:11 | PASS | ||
| 7 | dep-dcae-ves-collector-749996c5cb-5lf6g | 05:57:39 | 01:35 | PASS | ||
| 8 | dep-deployment-handler-69dd5c4d56-5w4vc | 06:01:12 | 06:47 | PASS | ||
| 9 | dep-holmes-engine-mgmt-7bdd595854-m7d4v | 06:10:27 | 01:03 | PASS | ||
| 10 | dep-holmes-rule-mgmt-b99586579-znrhl | 06:13:14 | 01:03 | PASS | ||
| 11 | dep-inventory-9585c9d6-9jtxv | 06:16:15 | 01:14 | PASS | ||
| 12 | dep-policy-handler-775c84f5d4-tf27n | 06:19:12 | 02:37 | PASS | ||
| 13 | dep-pstg-write-f67b57947-7fmzq | 06:24:17 | 01:42 | PASS | ||
| 14 | dep-service-change-handler-b8cf87787-dzwcq | 06:28:12 | 01:35 | PASS | ||
| 15 | dev-aai-aai-64dd9545fc-tlj5s | 06:31:15 | 01:04 | PASS | ||
| 16 | dev-aai-aai-babel-7cb986869d-4qfxs | 06:34:17 | 01:36 | PASS | ||
| 17 | dev-aai-aai-cassandra-0 | 06:37:37 | 01:14 | PASS | ||
| 18 | dev-aai-aai-champ-6b6d4779d8-9drmz | 06:40:35 | 04:14 | PASS | ||
| 19 | dev-aai-aai-data-router-6b4776bfc4-8hb4k | 06:46:18 | 09:01 | PASS | ||
| 20 | dev-aai-aai-elasticsearch-669f4664c-wcclz | 06:57:17 | 01:13 | PASS | ||
| 21 | dev-aai-aai-gizmo-9df94f898-dg79k | 07:01:14 | 04:50 | PASS | ||
| 22 | dev-aai-aai-graphadmin-5d6d4d86dc-6shv6 | 07:08:18 | 02:50 | PASS | ||
| 23 | dev-aai-aai-modelloader-54f7444dcd-qnftb | 07:14:22 | 01:06 | PASS | ||
| 24 | dev-aai-aai-resources-8579748958-wwf5m | 07:16:26 | 03:14 | PASS | ||
| 25 | dev-aai-aai-search-data-695b878655-tff7s | 07:22:08 | 01:06 | PASS |
| 26 | dev-aai-aai-sparky-be-9bcc99756-ztfcg | 07:26:14 | 17:05 | PASS | ||||||||||
| 27 | dev-aai-aai-spike-59c6db6574-v6dfj | 07:46:19 | 03:51 | PASS | ||||||||||
| 28 | dev-aai-aai-traversal-7f5669d56-55rnr | 07:51:39 | 01:41 | PASS | ||||||||||
| 29 | dev-appc-appc-0 | 07:56:18 | 58:38 | FAIL | APPC took 75 minutes to recover. The testing was done against a single node APPC due to
| |||||||||
| 30 | dev-appc-appc-ansible-server-c4678946c-qncqx | 08:55:11 | 32:10 | PASS | ||||||||||
| 31 | dev-appc-appc-cdt-868f9896b-82mlj | 09:31:39 | 00:40 | PASS | ||||||||||
| 32 | dev-appc-appc-db-0 | 09:35:17 | 01:41 | PASS | ||||||||||
| 33 | dev-appc-appc-dgbuilder-5d5cbfdd45-q7nbv | 09:39:26 | 02:39 | PASS | ||||||||||
| 34 | dev-dcaegen2-dcae-bootstrap-7c5c8d7d76-ng999 | 09:45:20 | 01:08 | PASS | ||||||||||
| 35 | dev-dcaegen2-dcae-cloudify-manager-67b576dd66-86kzx | 09:49:12 | 01:46 | PASS | ||||||||||
| 36 | dev-dcaegen2-dcae-db-0 | 09:53:12 | 03:10 | PASS | ||||||||||
| 37 | dev-dcaegen2-dcae-healthcheck-65c6548689-r6qbq | 09:59:07 | 02:09 | PASS | ||||||||||
| 38 | dev-dcaegen2-dcae-pgpool-78fbd864cd-hhzcv | 10:04:15 | 01:05 | PASS | ||||||||||
| 39 | dev-dcaegen2-dcae-pgpool-78fbd864cd-k8lgx | 10:06:34 | 02:46 | PASS | ||||||||||
| 40 | dev-dcaegen2-dcae-redis-0 | 10:11:32 | 02:17 | PASS | ||||||||||
| 41 | dev-dcaegen2-dcae-redis-3 | 10:16:34 | 01:35 | PASS | ||||||||||
| 42 | dev-dcaegen2-dcae-redis-4 | 10:21:09 | 01:43 | PASS | ||||||||||
| 43 | dev-dcaegen2-dcae-redis-5 | 10:25:36 | 01:35 | PASS | ||||||||||
| 44 | dev-dmaap-dbc-pg-0 | 10:30:41 | 02:07 | PASS | ||||||||||
| 45 | dev-dmaap-dbc-pgpool-7b748d5894-7sfcw | 10:35:32 | 01:04 | PASS | ||||||||||
| 46 | dev-dmaap-dbc-pgpool-7b748d5894-ngxxj | 10:38:34 | 00:44 | PASS | ||||||||||
| 47 | dev-dmaap-dmaap-bus-controller-765495b674-642jf | 10:41:34 | 01:03 | PASS | ||||||||||
| 48 | dev-dmaap-dmaap-dr-db-56b956df8d-rwffp | 10:45:05 | 01:13 | PASS | ||||||||||
| 49 | dev-dmaap-dmaap-dr-node-64864d5cc-r8ck8 | 10:49:18 | 01:10 | PASS | ||||||||||
| 50 | dev-dmaap-dmaap-dr-prov-69bd7c6665-676cf | 10:51:27 | 01:03 | PASS | ||||||||||
| 51 | dev-dmaap-message-router-647bbfc54d-h588k | 10:55:14 | 03:08 | PASS | ||||||||||
| 52 | dev-dmaap-message-router-kafka-6f54ccf867-5mkbg | 11:01:37 | 02:43 | PASS | ||||||||||
| 53 | dev-dmaap-message-router-zookeeper-847656487f-mngbc | 11:07:34 | 01:45 | FAIL | After dmaap-message-router-zookeeper was deleted, vFWCL continued to fail until dmaap-message-router was subsequently restarted manually. Logged
| |||||||||
| 54 | dev-policy-brmsgw-78db5d4db4-n6b9j | 17:15:06 | 02:15 | PASS | ||||||||||
| 55 | dev-policy-drools-0 | 17:20:05 | 02:17 | PASS | ||||||||||
| 56 | dev-policy-nexus-7d5f84d8b6-xzmwd | 17:28:28 | 02:15 | PASS | ||||||||||
| 57 | dev-policy-pap-9cb89bf74-5cfmt | 17:33:14 | 02:39 | PASS | ||||||||||
| 58 | dev-policy-pdp-0 | 17:39:23 | 01:38 | PASS | ||||||||||
| 59 | dev-policy-policy-apex-pdp-0 | 17:44:00 | 02:48 | PASS | ||||||||||
| 60 | dev-policy-policydb-74d46689cb-fmrc6 | 17:55:16 | 02:44 | PASS | ||||||||||
| 61 | dev-policy-policy-distribution-75b76b55f7-drknk | 17:50:01 | 03:15 | PASS |
Requirement
Area | Priority | Min. Level | Stretch Goal | Level Descriptions (Abbreviated) |
Resiliency | High | Level 2 – run-time projects | Level 3 – run-time projects | •1 – manual failure and recovery (< 30 minutes) |