...
Completed
We will use vFW use case as the baseline to test this:
Pre-requirement: Instantiate a vFW with closed loop running.
VNF Onboarding and Distribution: 97.8% (44/45) PASS
| Pod Deleted | Start Time (PST) | Recovery Time (MM:SS) | Pass / Fail |
|---|
| Logs | Notes | |||
|---|---|---|---|---|
| 1 | dev-aai-aai-babel-7cb986869d-9zcxf | 18:37:07 | 00:35 | PASS |
| 2 | dev-aai-aai-c9bc74cd9-szq8l | 18:47:58 | 01:12 | PASS |
| 3 | dev-aai-aai-cassandra-0 | 18:54:05 | 01:14 | PASS |
| 4 | dev-aai-aai-champ-6b6d4779d8-snrdj | 19:02:03 | 04:11 | PASS |
| 5 | dev-aai-aai-data-router-6b4776bfc4-kfrs4 | 00:23:49 | 01:10 | PASS |
http://onapci.org/jenkins/job/windriver-sb01-oom-resilience-healthdist/47/
| 6 | dev-aai-aai-elasticsearch-669f4664c-5rqll | 19:12:48 | 01:05 | PASS |
| 7 | dev-aai-aai-gizmo-9df94f898-7vbh5 | 19:19:27 | 01:15 | PASS |
| 8 | dev-aai-aai-graphadmin-588cd58c76-vrm84 | 19:24:12 | 01:40 | PASS |
| 9 | dev-aai-aai-modelloader-54f7444dcd-mbqgq | 19:31:09 | 01:04 | PASS |
| 10 | dev-aai-aai-resources-6b9c446d7-rnm56 | 19:38:15 | 02:13 | PASS |
http://onapci.org/jenkins/job/windriver-sb01-oom-resilience-healthdist/11/
| 11 | dev-aai-aai-search-data-695b878655-b2z7n | 19:44:18 | 01:04 | PASS |
| 12 | dev-aai-aai-sparky-be-9bcc99756-f7796 | 19:51:22 | 03:54 | PASS |
| 13 | dev-aai-aai-spike-59c6db6574-phz9h | 19:59:18 | 01:06 | PASS |
| 14 | dev-aai-aai-traversal-7bd85969cf-mcg24 | 20:05:07 | 01:46 | PASS |
| 15 | dev-sdc-sdc-be-6d8f57d5c4-g6wg5 | 20:11:09 | 03:09 | PASS |
http://onapci.org/jenkins/job/windriver-sb01-oom-resilience-healthdist/16/
| 16 | dev-sdc-sdc-cs-7768fcf944-fs4ph | 00:28:46 | 35:07 | FAIL |
After deletion of sdc-cs, sdc-be pod falls into 1/2 Running status and never recovered. sdc-be required manual pod deletion to recover. Logged | |||||||||
| 17 | dev-sdc-sdc-dcae-be-5bb587d95c-465l9 | 20:45:34 | 23:03 | PASS |
| 18 | dev-sdc-sdc-dcae-dt-68d8d77578-5fn69 | 21:12:47 | 01:14 | PASS |
| 19 | dev-sdc-sdc-dcae-fe-79675b4bc4-lgh65 | 21:18:46 | 01:14 | PASS |
http://onapci.org/jenkins/job/windriver-sb01-oom-resilience-healthdist/20/
| 20 | dev-sdc-sdc-dcae-tosca-lab-6d9f59cbf6-lsljh | 21:24:35 | 01:08 | PASS |
| 21 | dev-sdc-sdc-es-6d9566f787-j47fj | 21:29:39 | 03:48 | PASS |
| 22 | dev-sdc-sdc-fe-7c7c4577d9-jrxtg | 21:38:31 | 01:45 | PASS |
| 23 | dev-sdc-sdc-onboarding-be-576864dfbc-89zxt | 21:45:40 | 02:43 | PASS |
| 24 | dev-sdc-sdc-wfd-be-7b84b4bf97-hsgkd | 21:54:35 | 02:09 | PASS |
http://onapci.org/jenkins/job/windriver-sb01-oom-resilience-healthdist/25/
| 25 | dev-sdc-sdc-wfd-fe-86597867d6-4jtzh | 22:00:41 | 01:35 | PASS |
| 26 | dev-sdnc-controller-blueprints-684f7d865f-krmhc | 22:07:41 | 01:12 | PASS |
| 27 | dev-sdnc-controller-blueprints-db-0 | 22:14:35 | 01:46 | PASS |
| 28 | dev-sdnc-nengdb-0 | 22:21:36 | 00:37 | PASS |
| 29 | dev-sdnc-network-name-gen-7f54b5649c-b2h7n | 22:27:39 | 01:02 | PASS |
http://onapci.org/jenkins/job/windriver-sb01-oom-resilience-healthdist/30/
dev-sdnc-sdnc-0
| 30 | dev-sdnc-sdnc-0 | 22:32:49 | 02:12 | PASS |
| 31 | dev-sdnc-sdnc-db-0 | 22:40:02 | 03:47 | PASS |
| 32 | dev-sdnc-sdnc-dgbuilder-77866dbfb5-fb5fd | 22:47:31 | 02:09 | PASS |
| 33 | dev-sdnc-sdnc-dmaap-listener-6f67f97dd9-nttl7 | 22:53:49 | 00:45 | PASS |
| 34 | dev-sdnc-sdnc-portal-79f65cfcf8-7q7h6 | 22:59:32 | 05:15 | PASS |
http://onapci.org/jenkins/job/windriver-sb01-oom-resilience-healthdist/35/
| 35 | dev-sdnc-sdnc-ueb-listener-557c57577c-th896 | 23:08:44 | 01:04 | PASS |
| 36 | dev-so-so-6fcf596b6b-w2dsj | 23:16:44 | 01:06 | PASS |
| 37 | dev-so-so-bpmn-infra-6979cdb87d-lpm5g | 23:22:36 | 01:03 | PASS |
| 38 | dev-so-so-catalog-db-adapter-566fdc66c-vtvp5 | 23:27:39 | 01:04 | PASS |
| 39 | dev-so-so-mariadb-576ccb887f-nf8j6 | 23:32:48 | 09:05 | PASS |
http://onapci.org/jenkins/job/windriver-sb01-oom-resilience-healthdist/40/
| 40 | dev-so-so-monitoring-6c5b768dc4-c975x | 23:46:50 | 01:03 | PASS |
| 41 | dev-so-so-openstack-adapter-8465dc7d79-7llmh | 23:53:43 | 01:09 | PASS |
| 42 | dev-so-so-request-db-adapter-86d4477b55-wxdqv | 00:00:48 | 01:12 | PASS |
| 43 | dev-so-so-sdc-controller-86d59b9f4f-7x5gz | 00:07:47 | 01:13 | PASS |
| 44 | dev-so-so-sdnc-adapter-6b7cdc9b-gnkzw | 00:12:41 | 01:11 | PASS |
http://onapci.org/jenkins/job/windriver-sb01-oom-resilience-healthdist/45/
| 45 | dev-so-so-vfc-adapter-85d6f88ff6-4xk6b | 00:18:30 | 01:34 | PASS |
Sub-Categories
(In Error Mode Component)
Closed Loop
Policy
Requirement
...
Area
...
Priority
...
Min. Level
...
Stretch Goal
...
Level Descriptions (Abbreviated)
...
Resiliency
...
High
...
Level 2 – run-time projects
Level 1 – remaining projects
...
Level 3 – run-time projects
Level 2 – remaining projects
...
VNF Instantiation: 96.3% (52/54) PASS
| Pod Deleted | Start Time (PST) | Recovery Time (MM:SS) | Pass / Fail | Logs | Notes | |
|---|---|---|---|---|---|---|
| 1 | dev-aai-aai-babel-7cb986869d-zkf8r | 21:58:32 | 01:11 | PASS | ||
| 2 | dev-aai-aai-cassandra-0 | 22:14:59 | 01:46 | PASS | ||
| 3 | dev-aai-aai-data-router-6b4776bfc4-lssst | 22:26:45 | 01:12 | PASS | ||
| 4 | dev-aai-aai-elasticsearch-669f4664c-h5fh4 | 22:40:00 | 13:30 | PASS | ||
| 5 | dev-aai-aai-ff69fc688-r5hc9 | 23:04:30 | 01:12 | PASS | ||
| 6 | dev-aai-aai- |
...
gizmo- |
...
9df94f898-64j9t | 23:18: |
...
32 | 13:26 | PASS | ||
| 7 | dev-aai-aai-graphadmin- |
...
5d6d4d86dc-7wvgs | 23:45:45 | 06:51 | PASS | ||
| 8 | dev-aai-aai-modelloader- |
...
54f7444dcd-xw54c | 00:06:17 | 01:04 | PASS | ||
| 9 | dev-aai-aai- |
...
resources- |
...
8579748958-jqg9x | 00:20:41 | 01:48 | PASS | ||
| 10 | dev-aai-aai-search-data- |
...
695b878655-wfr5l | 00: |
...
36:00 | 01:09 | PASS | ||
| 11 | dev-aai-aai-sparky- |
...
be- |
...
9bcc99756-879q9 | 20:29:45 | 05:46 | PASS | ||
| 12 | dev-aai-aai- |
...
spike- |
...
59c6db6574-nj465 | 00:58:25 | 01:07 | PASS | ||
| 13 | dev-aai-aai- |
...
traversal- |
...
7f5669d56-xwvlr | 01:12:58 | 02:09 | PASS | |||
| 14 | dev-multicloud-multicloud-7cd96f75fb-vzmq7 | 01:28:35 | 01:09 | PASS | ||
| 15 | dev-multicloud-multicloud-azure-85548b6469-d6kz9 | 01:41:41 | 01:03 | PASS | ||
| 16 | dev-multicloud-multicloud-ocata-85d796889b-ctvd9 | 18:44:00 | 01:04 | PASS | ||
| 17 | dev-multicloud-multicloud-pike-78fb779f6c-vn789 | 02:04:48 | 01:06 | PASS | ||
| 18 | dev-multicloud-multicloud-vio-794f8b7d85-hwnjg | 02:18:16 | 04:16 | PASS | ||
| 19 | dev-multicloud-multicloud-windriver-6ffb4dcd-dhnlf | 02:35:08 | 01:04 | PASS | ||
| 20 | dev-sdc-sdc-be-6d8f57d5c4- |
...
dqs8w | 02:49:07 | 03:16 | PASS | ||
| 21 | dev-sdc-sdc-cs-7768fcf944- |
...
cg75x | 03:05:30 | 26:25 | PASS | ||
| 22 | dev-sdc-sdc-dcae-be-5bb587d95c- |
...
dgtwj | 03:44:55 | 05:18 | PASS | ||
| 23 | dev-sdc-sdc-dcae-dt-68d8d77578- |
...
plvjn | 04:02:51 | 01:17 | PASS | ||
| 24 | dev-sdc-sdc-dcae-fe-79675b4bc4- |
...
78cgd | 04:17:54 | 02:14 | PASS | ||
| 25 | dev-sdc-sdc-dcae-tosca-lab-6d9f59cbf6- |
...
fj7jp | 04:33:40 | 02:19 | PASS | ||
| 26 | dev-sdc-sdc-es-6d9566f787- |
...
dvgx6 | 04:50:34 | 03:08 | PASS | ||
| 27 | dev-sdc-sdc-fe-7c7c4577d9- |
...
bpdhh | 05:08:04 | 01:38 | PASS | ||
| 28 | dev-sdc-sdc- |
...
kb- |
...
85f8df694-kns5t | 20:19:48 | 01:11 | PASS | ||
| 29 | dev-sdc-sdc- |
...
onboarding-be- |
...
576864dfbc-x8pt8 | 05:40:05 | 09:03 | PASS | ||
| 30 | dev-sdc-sdc-wfd- |
...
be- |
...
7b84b4bf97-stlxs | 06:04:46 | 02:11 | PASS | |||
| 31 | dev-sdc-sdc-wfd-fe-86597867d6-hw725 | 06:22:13 | 01:41 | PASS | ||
| 32 | dev-sdnc-controller-blueprints- |
...
684f7d865f-4pglc | 06:39:13 | 01:08 | PASS | |||
| 33 | dev-sdnc-controller-blueprints-db-0 | 06:55:57 | 01:48 | PASS | ||
| 34 | dev-sdnc-nengdb-0 | 07:12:23 | 01:42 | PASS | ||
| 35 | dev-sdnc-network-name-gen-7f54b5649c- |
...
w4j8x | 07:28:47 | 02:37 | PASS | ||
| 36 | dev-sdnc-sdnc-0 |
...
07: |
...
46:43 | 03:45 | PASS | ||
| 37 | dev-sdnc-sdnc- |
...
ansible-server-8b58d86f5-d6hz2 | 08:05:46 | 03:16 | PASS | ||
| 38 | dev-sdnc-sdnc- |
...
db-0 | 08:24:10 | 03:52 | FAIL | After sdnc-sdnc-db was deleted, the SDNC portal |
...
continues to fail due to inability to connect to the DB until sdnc-portal was manually restarted. Logged | ||||||||
| 39 | dev-sdnc-sdnc- |
...
dgbuilder- |
...
77866dbfb5-wk4zj | 20:08: |
...
09 | 02:43 | PASS | ||||||||||||
| 40 | dev-sdnc-sdnc-dmaap-listener-6f67f97dd9-crfxr | 19:58:25 | 01:05 | PASS | ||||||||||
| 41 | dev-sdnc-sdnc-portal-79f65cfcf8-l8nq8 | 19:46:36 | 03:07 | PASS | ||||||||||
| 42 | dev-sdnc-sdnc-ueb-listener-557c57577c-hzt2v | 19:35:08 | 02:47 | PASS | ||||||||||
| 43 | dev-so-so-575dcc7fc5-stw5p | 19:25:16 | 01:06 | PASS | ||||||||||
| 44 | dev-so-so-bpmn-infra-74875c7c-6k4lz | 19:15:00 | 01:11 | PASS | ||||||||||
| 45 | dev-so-so-catalog-db-adapter-67878b4fc8-mzwg2 | 09:24:36 | 03:53 | PASS | ||||||||||
| 46 | dev-so-so-mariadb-576ccb887f-vh2f2 | 09:45:07 | 08:27 | PASS | ||||||||||
| 47 | dev-so-so-monitoring-7b4fdbf895-szgtp | 10:11:45 | 01:40 | PASS | ||||||||||
| 48 | dev-so-so-openstack-adapter-85576544b6-xmd62 | 10:28:14 | 01:35 | PASS | ||||||||||
| 49 | dev-so-so-request-db-adapter-658db4b75d-gfn6g | 19:05:03 | 01:11 | PASS | ||||||||||
| 50 | dev-so-so-sdc-controller-679f5dff6d-rxgf5 | 14:50:51 | 01:17 | PASS | ||||||||||
| 51 | dev-so-so-sdnc-adapter-54d99774c9-qgp6w | 15:10:15 | 01:05 | PASS | ||||||||||
| 52 | dev-so-so-vfc-adapter-5bb49bc97f-7rvzj | 15:31:05 | 01:03 | PASS | ||||||||||
| 53 | dev-vid-vid-54f8c6d78-jj8vp | 15:49:13 | 04:48 | PASS | ||||||||||
| 54 | dev-vid-vid-mariadb-galera-0 | 16:21:42 | 01:09 | FAIL | After vid-mariadb-galera was deleted, the instance creation via VID continue to fail even after the vid pod was also restarted. Logged
|
Closed Loop: 96.7% (59/61) PASS
| Pod Deleted | Start Time (PST) | Recovery Time (MM:SS) | Pass / Fail | Logs | Notes | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | dep-config-binding-service-596688f976-tdrpz | 05:29:55 | 02:08 | PASS | ||||||||||
| 2 | dep-dcae-datafile-collector-65df56bc6b-4cj5p | 05:35:36 | 01:14 | PASS | ||||||||||
| 3 | dep-dcae-hv-ves-collector-9df8d946-nz85f | 05:39:33 | 00:45 | PASS | ||||||||||
| 4 | dep-dcae-prh-6bfbb97fdc-n2pvb | 05:43:18 | 01:09 | PASS | ||||||||||
| 5 | dep-dcae-snmptrap-collector-58bbcb6644-g9tkw | 05:47:10 | 02:36 | PASS | ||||||||||
| 6 | dep-dcae-tca-analytics-7dbddb9697-d2h8m | 05:51:14 | 03:11 | PASS | ||||||||||
| 7 | dep-dcae-ves-collector-749996c5cb-5lf6g | 05:57:39 | 01:35 | PASS | ||||||||||
| 8 | dep-deployment-handler-69dd5c4d56-5w4vc | 06:01:12 | 06:47 | PASS | ||||||||||
| 9 | dep-holmes-engine-mgmt-7bdd595854-m7d4v | 06:10:27 | 01:03 | PASS | ||||||||||
| 10 | dep-holmes-rule-mgmt-b99586579-znrhl | 06:13:14 | 01:03 | PASS | ||||||||||
| 11 | dep-inventory-9585c9d6-9jtxv | 06:16:15 | 01:14 | PASS | ||||||||||
| 12 | dep-policy-handler-775c84f5d4-tf27n | 06:19:12 | 02:37 | PASS | ||||||||||
| 13 | dep-pstg-write-f67b57947-7fmzq | 06:24:17 | 01:42 | PASS | ||||||||||
| 14 | dep-service-change-handler-b8cf87787-dzwcq | 06:28:12 | 01:35 | PASS | ||||||||||
| 15 | dev-aai-aai-64dd9545fc-tlj5s | 06:31:15 | 01:04 | PASS | ||||||||||
| 16 | dev-aai-aai-babel-7cb986869d-4qfxs | 06:34:17 | 01:36 | PASS | ||||||||||
| 17 | dev-aai-aai-cassandra-0 | 06:37:37 | 01:14 | PASS | ||||||||||
| 18 | dev-aai-aai-champ-6b6d4779d8-9drmz | 06:40:35 | 04:14 | PASS | ||||||||||
| 19 | dev-aai-aai-data-router-6b4776bfc4-8hb4k | 06:46:18 | 09:01 | PASS | ||||||||||
| 20 | dev-aai-aai-elasticsearch-669f4664c-wcclz | 06:57:17 | 01:13 | PASS | ||||||||||
| 21 | dev-aai-aai-gizmo-9df94f898-dg79k | 07:01:14 | 04:50 | PASS | ||||||||||
| 22 | dev-aai-aai-graphadmin-5d6d4d86dc-6shv6 | 07:08:18 | 02:50 | PASS | ||||||||||
| 23 | dev-aai-aai-modelloader-54f7444dcd-qnftb | 07:14:22 | 01:06 | PASS | ||||||||||
| 24 | dev-aai-aai-resources-8579748958-wwf5m | 07:16:26 | 03:14 | PASS | ||||||||||
| 25 | dev-aai-aai-search-data-695b878655-tff7s | 07:22:08 | 01:06 | PASS | ||||||||||
| 26 | dev-aai-aai-sparky-be-9bcc99756-ztfcg | 07:26:14 | 17:05 | PASS | ||||||||||
| 27 | dev-aai-aai-spike-59c6db6574-v6dfj | 07:46:19 | 03:51 | PASS | ||||||||||
| 28 | dev-aai-aai-traversal-7f5669d56-55rnr | 07:51:39 | 01:41 | PASS | ||||||||||
| 29 | dev-appc-appc-0 | 07:56:18 | 58:38 | FAIL | APPC took 75 minutes to recover. The testing was done against a single node APPC due to
| |||||||||
| 30 | dev-appc-appc-ansible-server-c4678946c-qncqx | 08:55:11 | 32:10 | PASS | ||||||||||
| 31 | dev-appc-appc-cdt-868f9896b-82mlj | 09:31:39 | 00:40 | PASS | ||||||||||
| 32 | dev-appc-appc-db-0 | 09:35:17 | 01:41 | PASS | ||||||||||
| 33 | dev-appc-appc-dgbuilder-5d5cbfdd45-q7nbv | 09:39:26 | 02:39 | PASS | ||||||||||
| 34 | dev-dcaegen2-dcae-bootstrap-7c5c8d7d76-ng999 | 09:45:20 | 01:08 | PASS | ||||||||||
| 35 | dev-dcaegen2-dcae-cloudify-manager-67b576dd66-86kzx | 09:49:12 | 01:46 | PASS | ||||||||||
| 36 | dev-dcaegen2-dcae-db-0 | 09:53:12 | 03:10 | PASS | ||||||||||
| 37 | dev-dcaegen2-dcae-healthcheck-65c6548689-r6qbq | 09:59:07 | 02:09 | PASS | ||||||||||
| 38 | dev-dcaegen2-dcae-pgpool-78fbd864cd-hhzcv | 10:04:15 | 01:05 | PASS | ||||||||||
| 39 | dev-dcaegen2-dcae-pgpool-78fbd864cd-k8lgx | 10:06:34 | 02:46 | PASS | ||||||||||
| 40 | dev-dcaegen2-dcae-redis-0 | 10:11:32 | 02:17 | PASS | ||||||||||
| 41 | dev-dcaegen2-dcae-redis-3 | 10:16:34 | 01:35 | PASS | ||||||||||
| 42 | dev-dcaegen2-dcae-redis-4 | 10:21:09 | 01:43 | PASS | ||||||||||
| 43 | dev-dcaegen2-dcae-redis-5 | 10:25:36 | 01:35 | PASS | ||||||||||
| 44 | dev-dmaap-dbc-pg-0 | 10:30:41 | 02:07 | PASS | ||||||||||
| 45 | dev-dmaap-dbc-pgpool-7b748d5894-7sfcw | 10:35:32 | 01:04 | PASS | ||||||||||
| 46 | dev-dmaap-dbc-pgpool-7b748d5894-ngxxj | 10:38:34 | 00:44 | PASS | ||||||||||
| 47 | dev-dmaap-dmaap-bus-controller-765495b674-642jf | 10:41:34 | 01:03 | PASS | ||||||||||
| 48 | dev-dmaap-dmaap-dr-db-56b956df8d-rwffp | 10:45:05 | 01:13 | PASS | ||||||||||
| 49 | dev-dmaap-dmaap-dr-node-64864d5cc-r8ck8 | 10:49:18 | 01:10 | PASS | ||||||||||
| 50 | dev-dmaap-dmaap-dr-prov-69bd7c6665-676cf | 10:51:27 | 01:03 | PASS | ||||||||||
| 51 | dev-dmaap-message-router-647bbfc54d-h588k | 10:55:14 | 03:08 | PASS | ||||||||||
| 52 | dev-dmaap-message-router-kafka-6f54ccf867-5mkbg | 11:01:37 | 02:43 | PASS | ||||||||||
| 53 | dev-dmaap-message-router-zookeeper-847656487f-mngbc | 11:07:34 | 01:45 | FAIL | After dmaap-message-router-zookeeper was deleted, vFWCL continued to fail until dmaap-message-router was subsequently restarted manually. Logged
| |||||||||
| 54 | dev-policy-brmsgw-78db5d4db4-n6b9j | 17:15:06 | 02:15 | PASS | ||||||||||
| 55 | dev-policy-drools-0 | 17:20:05 | 02:17 | PASS | ||||||||||
| 56 | dev-policy-nexus-7d5f84d8b6-xzmwd | 17:28:28 | 02:15 | PASS | ||||||||||
| 57 | dev-policy-pap-9cb89bf74-5cfmt | 17:33:14 | 02:39 | PASS | ||||||||||
| 58 | dev-policy-pdp-0 | 17:39:23 | 01:38 | PASS | ||||||||||
| 59 | dev-policy-policy-apex-pdp-0 | 17:44:00 | 02:48 | PASS | ||||||||||
| 60 | dev-policy-policydb-74d46689cb-fmrc6 | 17:55:16 | 02:44 | PASS | ||||||||||
| 61 | dev-policy-policy-distribution-75b76b55f7-drknk | 17:50:01 | 03:15 | PASS |
Requirement
Area | Priority | Min. Level | Stretch Goal | Level Descriptions (Abbreviated) |
Resiliency | High | Level 2 – run-time projects | Level 3 – run-time projects | •1 – manual failure and recovery (< 30 minutes) |