Pod Deleted	Start Time (PST)	Recovery Time (MM:SS)	Pass / Fail	Logs	Notes
dev-aai-aai-babel-7cb986869d-9zcxf	18:37:07	00:35	PASS	2/
dev-aai-aai-c9bc74cd9-szq8l	18:47:58	01:12	PASS	3/
dev-aai-aai-cassandra-0	18:54:05	01:14	PASS	4/
dev-aai-aai-champ-6b6d4779d8-snrdj	19:02:03	04:11	PASS	5/
dev-aai-aai-data-router-6b4776bfc4-kfrs4	00:23:49	01:10	PASS	47/
dev-aai-aai-elasticsearch-669f4664c-5rqll	19:12:48	01:05	PASS	7/
dev-aai-aai-gizmo-9df94f898-7vbh5	19:19:27	01:15	PASS	8/
dev-aai-aai-graphadmin-588cd58c76-vrm84	19:24:12	01:40	PASS	9/
dev-aai-aai-modelloader-54f7444dcd-mbqgq	19:31:09	01:04	PASS	10/
dev-aai-aai-resources-6b9c446d7-rnm56	19:38:15	02:13	PASS	11/
dev-aai-aai-search-data-695b878655-b2z7n	19:44:18	01:04	PASS	12/
dev-aai-aai-sparky-be-9bcc99756-f7796	19:51:22	03:54	PASS	13/
dev-aai-aai-spike-59c6db6574-phz9h	19:59:18	01:06	PASS	14/
dev-aai-aai-traversal-7bd85969cf-mcg24	20:05:07	01:46	PASS	15/
dev-sdc-sdc-be-6d8f57d5c4-g6wg5	20:11:09	03:09	PASS	16/
dev-sdc-sdc-cs-7768fcf944-fs4ph	00:28:46	35:07	FAIL	48/	After deletion of sdc-cs, sdc-be pod falls into 1/2 Running status and never recovered. sdc-be required manual pod deletion to recover.
dev-sdc-sdc-dcae-be-5bb587d95c-465l9	20:45:34	23:03	PASS	18/
dev-sdc-sdc-dcae-dt-68d8d77578-5fn69	21:12:47	01:14	PASS	19/
dev-sdc-sdc-dcae-fe-79675b4bc4-lgh65	21:18:46	01:14	PASS	20/
dev-sdc-sdc-dcae-tosca-lab-6d9f59cbf6-lsljh	21:24:35	01:08	PASS	21/
dev-sdc-sdc-es-6d9566f787-j47fj	21:29:39	03:48	PASS	22/
dev-sdc-sdc-fe-7c7c4577d9-jrxtg	21:38:31	01:45	PASS	23/
dev-sdc-sdc-onboarding-be-576864dfbc-89zxt	21:45:40	02:43	PASS	24/
dev-sdc-sdc-wfd-be-7b84b4bf97-hsgkd	21:54:35	02:09	PASS	25/
dev-sdc-sdc-wfd-fe-86597867d6-4jtzh	22:00:41	01:35	PASS	26/
dev-sdnc-controller-blueprints-684f7d865f-krmhc	22:07:41	01:12	PASS	27/
dev-sdnc-controller-blueprints-db-0	22:14:35	01:46	PASS	28/
dev-sdnc-nengdb-0	22:21:36	00:37	PASS	29/
dev-sdnc-network-name-gen-7f54b5649c-b2h7n	22:27:39	01:02	PASS	30/
dev-sdnc-sdnc-0	22:32:49	02:12	PASS	31/
dev-sdnc-sdnc-db-0	22:40:02	03:47	PASS	32/
dev-sdnc-sdnc-dgbuilder-77866dbfb5-fb5fd	22:47:31	02:09	PASS	33/
dev-sdnc-sdnc-dmaap-listener-6f67f97dd9-nttl7	22:53:49	00:45	PASS	34/
dev-sdnc-sdnc-portal-79f65cfcf8-7q7h6	22:59:32	05:15	PASS	35/
dev-sdnc-sdnc-ueb-listener-557c57577c-th896	23:08:44	01:04	PASS	36/
dev-so-so-6fcf596b6b-w2dsj	23:16:44	01:06	PASS	37/
dev-so-so-bpmn-infra-6979cdb87d-lpm5g	23:22:36	01:03	PASS	38/
dev-so-so-catalog-db-adapter-566fdc66c-vtvp5	23:27:39	01:04	PASS	39/
dev-so-so-mariadb-576ccb887f-nf8j6	23:32:48	09:05	PASS	40/
dev-so-so-monitoring-6c5b768dc4-c975x	23:46:50	01:03	PASS	41/
dev-so-so-openstack-adapter-8465dc7d79-7llmh	23:53:43	01:09	PASS	42/
dev-so-so-request-db-adapter-86d4477b55-wxdqv	00:00:48	01:12	PASS	43/
dev-so-so-sdc-controller-86d59b9f4f-7x5gz	00:07:47	01:13	PASS	44/
dev-so-so-sdnc-adapter-6b7cdc9b-gnkzw	00:12:41	01:11	PASS	45/
dev-so-so-vfc-adapter-85d6f88ff6-4xk6b	00:18:30	01:34	PASS	46/

Closed Loop

Pod Deleted

Start Time

(PST)

Recovery Time

(MM:SS)

Pass / Fail

Logs

Notes

dep-config-binding-service-596688f976-tdrpz

05:29:55

02:08

PASS

1/

dep-dcae-datafile-collector-65df56bc6b-4cj5p

05:35:36

01:14

PASS

2/

dep-dcae-hv-ves-collector-9df8d946-nz85f

05:39:33

00:45

PASS

3/

dep-dcae-prh-6bfbb97fdc-n2pvb

05:43:18

01:09

PASS

4/

dep-dcae-snmptrap-collector-58bbcb6644-g9tkw

05:47:10

02:36

PASS

5/

dep-dcae-tca-analytics-7dbddb9697-d2h8m

05:51:14

03:11

PASS

6/

dep-dcae-ves-collector-749996c5cb-5lf6g

05:57:39

01:35

PASS

7/

dep-deployment-handler-69dd5c4d56-5w4vc

06:01:12

06:47

PASS

8/

dep-holmes-engine-mgmt-7bdd595854-m7d4v

06:10:27

01:03

PASS

9/

dep-holmes-rule-mgmt-b99586579-znrhl

06:13:14

01:03

PASS

10/

dep-inventory-9585c9d6-9jtxv

06:16:15

01:14

PASS

11/

dep-policy-handler-775c84f5d4-tf27n

06:19:12

02:37

PASS

12/

dep-pstg-write-f67b57947-7fmzq

06:24:17

01:42

PASS

13/

dep-service-change-handler-b8cf87787-dzwcq

06:28:12

01:35

PASS

14/

dev-aai-aai-64dd9545fc-tlj5s

06:31:15

01:04

PASS

15/

dev-aai-aai-babel-7cb986869d-4qfxs

06:34:17

01:36

PASS

16/

dev-aai-aai-cassandra-0

06:37:37

01:14

PASS

17/

dev-aai-aai-champ-6b6d4779d8-9drmz

06:40:35

04:14

PASS

18/

dev-aai-aai-data-router-6b4776bfc4-8hb4k

06:46:18

09:01

PASS

19/

dev-aai-aai-elasticsearch-669f4664c-wcclz

06:57:17

01:13

PASS

20/

dev-aai-aai-gizmo-9df94f898-dg79k

07:01:14

04:50

PASS

21/

dev-aai-aai-graphadmin-5d6d4d86dc-6shv6

07:08:18

02:50

PASS

22/

dev-aai-aai-modelloader-54f7444dcd-qnftb

07:14:22

01:06

PASS

23/

dev-aai-aai-resources-8579748958-wwf5m

07:16:26

03:14

PASS

24/

dev-aai-aai-search-data-695b878655-tff7s

07:22:08

01:06

PASS

25/

dev-aai-aai-sparky-be-9bcc99756-ztfcg

07:26:14

17:05

PASS

26/

dev-aai-aai-spike-59c6db6574-v6dfj

07:46:19

03:51

PASS

27/

dev-aai-aai-traversal-7f5669d56-55rnr

07:51:39

01:41

PASS

28/

dev-appc-appc-0

07:56:18

58:38

FAIL

29/

APPC took 75 minutes to recover. The testing was done against a single node APPC due to

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	APPC-1242

. Also, the Netconf mount was lost when appc pod was dropped and had to be re-created.

dev-appc-appc-ansible-server-c4678946c-qncqx

08:55:11

32:10

PASS

30/

dev-appc-appc-cdt-868f9896b-82mlj

09:31:39

00:40

PASS

31/

dev-appc-appc-db-0

09:35:17

01:41

PASS

32/

dev-appc-appc-dgbuilder-5d5cbfdd45-q7nbv

09:39:26

02:39

PASS

33/

dev-dcaegen2-dcae-bootstrap-7c5c8d7d76-ng999

09:45:20

01:08

PASS

34/

dev-dcaegen2-dcae-cloudify-manager-67b576dd66-86kzx

09:49:12

01:46

PASS

35/

dev-dcaegen2-dcae-db-0

09:53:12

03:10

PASS

36/

dev-dcaegen2-dcae-healthcheck-65c6548689-r6qbq

09:59:07

02:09

PASS

37/

dev-dcaegen2-dcae-pgpool-78fbd864cd-hhzcv

10:04:15

01:05

PASS

38/

dev-dcaegen2-dcae-pgpool-78fbd864cd-k8lgx

10:06:34

02:46

PASS

39/

dev-dcaegen2-dcae-redis-0

10:11:32

02:17

PASS

40/

dev-dcaegen2-dcae-redis-3

10:16:34

01:35

PASS

41/

dev-dcaegen2-dcae-redis-4

10:21:09

01:43

PASS

42/

dev-dcaegen2-dcae-redis-5

10:25:36

01:35

PASS

43/

dev-dmaap-dbc-pg-0

10:30:41

02:07

PASS

44/

dev-dmaap-dbc-pgpool-7b748d5894-7sfcw

10:35:32

01:04

PASS

45/

dev-dmaap-dbc-pgpool-7b748d5894-ngxxj

10:38:34

00:44

PASS

46/

dev-dmaap-dmaap-bus-controller-765495b674-642jf

10:41:34

01:03

PASS

47/

dev-dmaap-dmaap-dr-db-56b956df8d-rwffp

10:45:05

01:13

PASS

48/

dev-dmaap-dmaap-dr-node-64864d5cc-r8ck8

10:49:18

01:10

PASS

49/

dev-dmaap-dmaap-dr-prov-69bd7c6665-676cf

10:51:27

01:03

PASS

50/

dev-dmaap-message-router-647bbfc54d-h588k

10:55:14

03:08

PASS

51/

dev-dmaap-message-router-kafka-6f54ccf867-5mkbg

11:01:37

02:43

PASS

52/

dev-dmaap-message-router-zookeeper-847656487f-mngbc

11:07:34

01:45

FAIL

53/

After dmaap-message-router-zookeeper was deleted, vFWCL continued to fail until dmaap-message-router was subsequently restarted manually.

dev-policy-brmsgw-78db5d4db4-n6b9j

17:15:06

02:15

PASS

63/

dev-policy-drools-0

17:20:05

02:17

PASS

64/

dev-policy-nexus-7d5f84d8b6-xzmwd

17:28:28

02:15

PASS

65/

dev-policy-pap-9cb89bf74-5cfmt

17:33:14

02:39

PASS

66/

dev-policy-pdp-0

17:39:23

01:38

PASS

67/

dev-policy-policy-apex-pdp-0

17:44:00

02:48

PASS

68/

dev-policy-policydb-74d46689cb-fmrc6

17:55:16

02:44

PASS

70/

dev-policy-policy-distribution-75b76b55f7-drknk

17:50:01

03:15

PASS

69/

Time (EDT)	Categories	Sub-Categories (In Error Mode Component)	Time to Detect Failure and Repair	Pass?	Notes
	VNF Onboarding and Distribution	SDC
		SO
		A&AI
		SDNC
		SDNC
	VNF Instantiation	SDC
		VID
		SO
		A&AI
		SDNC
		MultiVIM
	Closed Loop	DCAE
		DMaaP
		Policy
		A&AI
		APPC (3-node cluster)

...

Versions Compared

Old Version 7

New Version 8

Key

Closed Loop

Page Comparison

Versions Compared

Old Version 7

New Version 8

Key

Closed Loop