...
Completed
We will use vFW use case as the baseline to test this:
...
Pod Deleted | Start Time (PST) | Recovery Time (MM:SS) | Pass / Fail | Logs | Notes | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
dev-aai-aai-babel-7cb986869d-zkf8r | 21:58:32 | 01:11 | PASS | ||||||||||
dev-aai-aai-cassandra-0 | 22:14:59 | 01:46 | PASS | ||||||||||
dev-aai-aai-data-router-6b4776bfc4-lssst | 22:26:45 | 01:12 | PASS | ||||||||||
dev-aai-aai-elasticsearch-669f4664c-h5fh4 | 22:40:00 | 13:30 | PASS | ||||||||||
dev-aai-aai-ff69fc688-r5hc9 | 23:04:30 | 01:12 | PASS | ||||||||||
dev-aai-aai-gizmo-9df94f898-64j9t | 23:18:32 | 13:26 | PASS | ||||||||||
dev-aai-aai-graphadmin-5d6d4d86dc-7wvgs | 23:45:45 | 06:51 | PASS | ||||||||||
dev-aai-aai-modelloader-54f7444dcd-xw54c | 00:06:17 | 01:04 | PASS | ||||||||||
dev-aai-aai-resources-8579748958-jqg9x | 00:20:41 | 01:48 | PASS | ||||||||||
dev-aai-aai-search-data-695b878655-wfr5l | 00:36:00 | 01:09 | PASS | ||||||||||
dev-aai-aai-sparky-be-9bcc99756-c8lvk879q9 | 1620:2729:3345 | 0205:4446 | PASS | 5610/ | Failed due to other prior failures. Re-test pending. | ||||||||
dev-aai-aai-spike-59c6db6574-nj465 | 00:58:25 | 01:07 | PASS | ||||||||||
dev-aai-aai-traversal-7f5669d56-xwvlr | 01:12:58 | 02:09 | PASS | ||||||||||
dev-multicloud-multicloud-7cd96f75fb-vzmq7 | 01:28:35 | 01:09 | PASS | ||||||||||
dev-multicloud-multicloud-azure-85548b6469-d6kz9 | 01:41:41 | 01:03 | PASS | ||||||||||
dev-multicloud-multicloud-ocata-85d796889b-ctvd9 | 18:44:00 | 01:04 | PASS | ||||||||||
dev-multicloud-multicloud-pike-78fb779f6c-vn789 | 02:04:48 | 01:06 | PASS | ||||||||||
dev-multicloud-multicloud-vio-794f8b7d85-hwnjg | 02:18:16 | 04:16 | PASS | ||||||||||
dev-multicloud-multicloud-windriver-6ffb4dcd-dhnlf | 02:35:08 | 01:04 | PASS | ||||||||||
dev-sdc-sdc-be-6d8f57d5c4-dqs8w | 02:49:07 | 03:16 | PASS | ||||||||||
dev-sdc-sdc-cs-7768fcf944-cg75x | 03:05:30 | 26:25 | PASS | ||||||||||
dev-sdc-sdc-dcae-be-5bb587d95c-dgtwj | 03:44:55 | 05:18 | PASS | ||||||||||
dev-sdc-sdc-dcae-dt-68d8d77578-plvjn | 04:02:51 | 01:17 | PASS | ||||||||||
dev-sdc-sdc-dcae-fe-79675b4bc4-78cgd | 04:17:54 | 02:14 | PASS | ||||||||||
dev-sdc-sdc-dcae-tosca-lab-6d9f59cbf6-fj7jp | 04:33:40 | 02:19 | PASS | ||||||||||
dev-sdc-sdc-es-6d9566f787-dvgx6 | 04:50:34 | 03:08 | PASS | ||||||||||
dev-sdc-sdc-fe-7c7c4577d9-bpdhh | 05:08:04 | 01:38 | PASS | ||||||||||
dev-sdc-sdc-kb-85f8df694-ghdvmkns5t | 1720:4919:4148 | 01:3811 | PASS | 619/ | Failed due to other prior failures. Re-test pending. | ||||||||
dev-sdc-sdc-onboarding-be-576864dfbc-x8pt8 | 05:40:05 | 09:03 | PASS | ||||||||||
dev-sdc-sdc-wfd-be-7b84b4bf97-stlxs | 06:04:46 | 02:11 | PASS | ||||||||||
dev-sdc-sdc-wfd-fe-86597867d6-hw725 | 06:22:13 | 01:41 | PASS | ||||||||||
dev-sdnc-controller-blueprints-684f7d865f-4pglc | 06:39:13 | 01:08 | PASS | ||||||||||
dev-sdnc-controller-blueprints-db-0 | 06:55:57 | 01:48 | PASS | ||||||||||
dev-sdnc-nengdb-0 | 07:12:23 | 01:42 | PASS | ||||||||||
dev-sdnc-network-name-gen-7f54b5649c-w4j8x | 07:28:47 | 02:37 | PASS | ||||||||||
dev-sdnc-sdnc-0 | 07:46:43 | 03:45 | PASS | ||||||||||
dev-sdnc-sdnc-ansible-server-8b58d86f5-d6hz2 | 08:05:46 | 03:16 | PASS | ||||||||||
dev-sdnc-sdnc-db-0 | 08:24:10 | 03:52 | FAIL | After sdnc-sdnc-db was deleted, the SDNC portal continues to fail due to inability to connect to the DB until sdnc-portal was manually restarted. Logged | |||||||||
dev-sdnc-sdnc-dgbuilder-77866dbfb5-44jdswk4zj | 20:08:3209 | 02:43 | 09:09 | Failed due to other prior failures. Re-test pending. | PASS | ||||||||
dev-sdnc-sdnc-dmaap-listener-6f67f97dd9-dv5r7crfxr | 0819:4658:4125 | 01:0605 | PASS | 417/ | Failed due to other prior failures. Re-test pending. | ||||||||
dev-sdnc-sdnc-portal-79f65cfcf8-4jh8bl8nq8 | 0819:5246:3336 | 03:4107 | PASS | 426/ | Failed due to other prior failures. Re-test pending. | ||||||||
dev-sdnc-sdnc-ueb-listener-557c57577c-knsskhzt2v | 0919:35:08 | 02:254705:20 | PASS | Failed due to other prior failures. Re-test pending. | |||||||||
dev-so-so-575dcc7fc5-8g626stw5p | 0919:1225:3216 | 01:0506 | PASS | Failed due to other prior failures. Re-test pending. | |||||||||
dev-so-so-bpmn-infra-74875c7c-l6pl96k4lz | 0919:1815:2700 | 01:1411 | PASS | Failed due to other prior failures. Re-test pending. | |||||||||
dev-so-so-catalog-db-adapter-67878b4fc8-mzwg2 | 09:24:36 | 03:53 | PASS | ||||||||||
dev-so-so-mariadb-576ccb887f-vh2f2 | 09:45:07 | 08:27 | PASS | ||||||||||
dev-so-so-monitoring-7b4fdbf895-szgtp | 10:11:45 | 01:40 | PASS | ||||||||||
dev-so-so-openstack-adapter-85576544b6-xmd62 | 10:28:14 | 01:35 | PASS | ||||||||||
dev-so-so-request-db-adapter-658db4b75d-nk7kpgfn6g | 1719:3105:5703 | 01:1611 | PASS | 592/ | Failed due to other prior failures. Re-test pending. | ||||||||
dev-so-so-sdc-controller-679f5dff6d-rxgf5 | 14:50:51 | 01:17 | PASS | ||||||||||
dev-so-so-sdnc-adapter-54d99774c9-qgp6w | 15:10:15 | 01:05 | PASS | ||||||||||
dev-so-so-vfc-adapter-5bb49bc97f-7rvzj | 15:31:05 | 01:03 | PASS | ||||||||||
dev-vid-vid-54f8c6d78-jj8vp | 15:49:13 | 04:48 | PASS | ||||||||||
dev-vid-vid-mariadb-galera-0 | 16:21:42 | 01:09 | FAIL | After vid-mariadb-galera was deleted, the instance creation via VID continue to fail even after the vid pod was also restarted. Logged
|
Closed Loop
Pod Deleted | Start Time (PST) | Recovery Time (MM:SS) | Pass / Fail | Logs | Notes | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
dep-config-binding-service-596688f976-tdrpz | 05:29:55 | 02:08 | PASS | ||||||||||
dep-dcae-datafile-collector-65df56bc6b-4cj5p | 05:35:36 | 01:14 | PASS | ||||||||||
dep-dcae-hv-ves-collector-9df8d946-nz85f | 05:39:33 | 00:45 | PASS | ||||||||||
dep-dcae-prh-6bfbb97fdc-n2pvb | 05:43:18 | 01:09 | PASS | ||||||||||
dep-dcae-snmptrap-collector-58bbcb6644-g9tkw | 05:47:10 | 02:36 | PASS | ||||||||||
dep-dcae-tca-analytics-7dbddb9697-d2h8m | 05:51:14 | 03:11 | PASS | ||||||||||
dep-dcae-ves-collector-749996c5cb-5lf6g | 05:57:39 | 01:35 | PASS | ||||||||||
dep-deployment-handler-69dd5c4d56-5w4vc | 06:01:12 | 06:47 | PASS | ||||||||||
dep-holmes-engine-mgmt-7bdd595854-m7d4v | 06:10:27 | 01:03 | PASS | ||||||||||
dep-holmes-rule-mgmt-b99586579-znrhl | 06:13:14 | 01:03 | PASS | ||||||||||
dep-inventory-9585c9d6-9jtxv | 06:16:15 | 01:14 | PASS | ||||||||||
dep-policy-handler-775c84f5d4-tf27n | 06:19:12 | 02:37 | PASS | ||||||||||
dep-pstg-write-f67b57947-7fmzq | 06:24:17 | 01:42 | PASS | ||||||||||
dep-service-change-handler-b8cf87787-dzwcq | 06:28:12 | 01:35 | PASS | ||||||||||
dev-aai-aai-64dd9545fc-tlj5s | 06:31:15 | 01:04 | PASS | ||||||||||
dev-aai-aai-babel-7cb986869d-4qfxs | 06:34:17 | 01:36 | PASS | ||||||||||
dev-aai-aai-cassandra-0 | 06:37:37 | 01:14 | PASS | ||||||||||
dev-aai-aai-champ-6b6d4779d8-9drmz | 06:40:35 | 04:14 | PASS | ||||||||||
dev-aai-aai-data-router-6b4776bfc4-8hb4k | 06:46:18 | 09:01 | PASS | ||||||||||
dev-aai-aai-elasticsearch-669f4664c-wcclz | 06:57:17 | 01:13 | PASS | ||||||||||
dev-aai-aai-gizmo-9df94f898-dg79k | 07:01:14 | 04:50 | PASS | ||||||||||
dev-aai-aai-graphadmin-5d6d4d86dc-6shv6 | 07:08:18 | 02:50 | PASS | ||||||||||
dev-aai-aai-modelloader-54f7444dcd-qnftb | 07:14:22 | 01:06 | PASS | ||||||||||
dev-aai-aai-resources-8579748958-wwf5m | 07:16:26 | 03:14 | PASS | ||||||||||
dev-aai-aai-search-data-695b878655-tff7s | 07:22:08 | 01:06 | PASS | ||||||||||
dev-aai-aai-sparky-be-9bcc99756-ztfcg | 07:26:14 | 17:05 | PASS | ||||||||||
dev-aai-aai-spike-59c6db6574-v6dfj | 07:46:19 | 03:51 | PASS | ||||||||||
dev-aai-aai-traversal-7f5669d56-55rnr | 07:51:39 | 01:41 | PASS | ||||||||||
dev-appc-appc-0 | 07:56:18 | 58:38 | FAIL | APPC took 75 minutes to recover. The testing was done against a single node APPC due to
| |||||||||
dev-appc-appc-ansible-server-c4678946c-qncqx | 08:55:11 | 32:10 | PASS | ||||||||||
dev-appc-appc-cdt-868f9896b-82mlj | 09:31:39 | 00:40 | PASS | ||||||||||
dev-appc-appc-db-0 | 09:35:17 | 01:41 | PASS | ||||||||||
dev-appc-appc-dgbuilder-5d5cbfdd45-q7nbv | 09:39:26 | 02:39 | PASS | ||||||||||
dev-dcaegen2-dcae-bootstrap-7c5c8d7d76-ng999 | 09:45:20 | 01:08 | PASS | ||||||||||
dev-dcaegen2-dcae-cloudify-manager-67b576dd66-86kzx | 09:49:12 | 01:46 | PASS | ||||||||||
dev-dcaegen2-dcae-db-0 | 09:53:12 | 03:10 | PASS | ||||||||||
dev-dcaegen2-dcae-healthcheck-65c6548689-r6qbq | 09:59:07 | 02:09 | PASS | ||||||||||
dev-dcaegen2-dcae-pgpool-78fbd864cd-hhzcv | 10:04:15 | 01:05 | PASS | ||||||||||
dev-dcaegen2-dcae-pgpool-78fbd864cd-k8lgx | 10:06:34 | 02:46 | PASS | ||||||||||
dev-dcaegen2-dcae-redis-0 | 10:11:32 | 02:17 | PASS | ||||||||||
dev-dcaegen2-dcae-redis-3 | 10:16:34 | 01:35 | PASS | ||||||||||
dev-dcaegen2-dcae-redis-4 | 10:21:09 | 01:43 | PASS | ||||||||||
dev-dcaegen2-dcae-redis-5 | 10:25:36 | 01:35 | PASS | ||||||||||
dev-dmaap-dbc-pg-0 | 10:30:41 | 02:07 | PASS | ||||||||||
dev-dmaap-dbc-pgpool-7b748d5894-7sfcw | 10:35:32 | 01:04 | PASS | ||||||||||
dev-dmaap-dbc-pgpool-7b748d5894-ngxxj | 10:38:34 | 00:44 | PASS | ||||||||||
dev-dmaap-dmaap-bus-controller-765495b674-642jf | 10:41:34 | 01:03 | PASS | ||||||||||
dev-dmaap-dmaap-dr-db-56b956df8d-rwffp | 10:45:05 | 01:13 | PASS | ||||||||||
dev-dmaap-dmaap-dr-node-64864d5cc-r8ck8 | 10:49:18 | 01:10 | PASS | ||||||||||
dev-dmaap-dmaap-dr-prov-69bd7c6665-676cf | 10:51:27 | 01:03 | PASS | ||||||||||
dev-dmaap-message-router-647bbfc54d-h588k | 10:55:14 | 03:08 | PASS | ||||||||||
dev-dmaap-message-router-kafka-6f54ccf867-5mkbg | 11:01:37 | 02:43 | PASS | ||||||||||
dev-dmaap-message-router-zookeeper-847656487f-mngbc | 11:07:34 | 01:45 | FAIL | After dmaap-message-router-zookeeper was deleted, vFWCL continued to fail until dmaap-message-router was subsequently restarted manually. Logged
| |||||||||
dev-policy-brmsgw-78db5d4db4-n6b9j | 17:15:06 | 02:15 | PASS | ||||||||||
dev-policy-drools-0 | 17:20:05 | 02:17 | PASS | ||||||||||
dev-policy-nexus-7d5f84d8b6-xzmwd | 17:28:28 | 02:15 | PASS | ||||||||||
dev-policy-pap-9cb89bf74-5cfmt | 17:33:14 | 02:39 | PASS | ||||||||||
dev-policy-pdp-0 | 17:39:23 | 01:38 | PASS | ||||||||||
dev-policy-policy-apex-pdp-0 | 17:44:00 | 02:48 | PASS | ||||||||||
dev-policy-policydb-74d46689cb-fmrc6 | 17:55:16 | 02:44 | PASS | ||||||||||
dev-policy-policy-distribution-75b76b55f7-drknk | 17:50:01 | 03:15 | PASS |
Closed Loop
Requirement
Area | Priority | Min. Level | Stretch Goal | Level Descriptions (Abbreviated) |
Resiliency | High | Level 2 – run-time projects | Level 3 – run-time projects | •1 – manual failure and recovery (< 30 minutes) |