Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Completed

We will use vFW use case as the baseline to test this:

...

Failed due to other prior failures. Re-test pending.Failed due to other prior failures. Re-test pending.Failed due to other prior failures. Re-test pending.
Pod Deleted

Start Time

(PST)

Recovery Time

(MM:SS)

Pass / FailLogsNotes

dev-aai-aai-babel-7cb986869d-zkf8r

21:58:32

01:11

PASS

2/


dev-aai-aai-cassandra-0

22:14:59

01:46

PASS

3/


dev-aai-aai-data-router-6b4776bfc4-lssst

22:26:45

01:12

PASS

4/


dev-aai-aai-elasticsearch-669f4664c-h5fh4

22:40:00

13:30

PASS

5/


dev-aai-aai-ff69fc688-r5hc9

23:04:30

01:12

PASS

6/


dev-aai-aai-gizmo-9df94f898-64j9t

23:18:32

13:26

PASS

7/


dev-aai-aai-graphadmin-5d6d4d86dc-7wvgs

23:45:45

06:51

PASS

8/


dev-aai-aai-modelloader-54f7444dcd-xw54c

00:06:17

01:04

PASS

9/


dev-aai-aai-resources-8579748958-jqg9x

00:20:41

01:48

PASS

10/


dev-aai-aai-search-data-695b878655-wfr5l

00:36:00

01:09

PASS

11/


dev-aai-aai-sparky-be-9bcc99756-c8lvk879q9

1620:2729:3345

0205:4446

PASS

5610/

Failed due to other prior failures. Re-test pending.

dev-aai-aai-spike-59c6db6574-nj465

00:58:25

01:07

PASS

13/


dev-aai-aai-traversal-7f5669d56-xwvlr

01:12:58

02:09

PASS

14/


dev-multicloud-multicloud-7cd96f75fb-vzmq7

01:28:35

01:09

PASS

15/


dev-multicloud-multicloud-azure-85548b6469-d6kz9

01:41:41

01:03

PASS

16/


dev-multicloud-multicloud-ocata-85d796889b-ctvd9

18:44:00

01:04

PASS

1/


dev-multicloud-multicloud-pike-78fb779f6c-vn789

02:04:48

01:06

PASS

18/


dev-multicloud-multicloud-vio-794f8b7d85-hwnjg

02:18:16

04:16

PASS

19/


dev-multicloud-multicloud-windriver-6ffb4dcd-dhnlf

02:35:08

01:04

PASS

20/


dev-sdc-sdc-be-6d8f57d5c4-dqs8w

02:49:07

03:16

PASS

21/


dev-sdc-sdc-cs-7768fcf944-cg75x

03:05:30

26:25

PASS

22/


dev-sdc-sdc-dcae-be-5bb587d95c-dgtwj

03:44:55

05:18

PASS

23/


dev-sdc-sdc-dcae-dt-68d8d77578-plvjn

04:02:51

01:17

PASS

24/


dev-sdc-sdc-dcae-fe-79675b4bc4-78cgd

04:17:54

02:14

PASS

25/


dev-sdc-sdc-dcae-tosca-lab-6d9f59cbf6-fj7jp

04:33:40

02:19

PASS

26/


dev-sdc-sdc-es-6d9566f787-dvgx6

04:50:34

03:08

PASS

27/


dev-sdc-sdc-fe-7c7c4577d9-bpdhh

05:08:04

01:38

PASS

28/


dev-sdc-sdc-kb-85f8df694-ghdvmkns5t

1720:4919:4148

01:3811

PASS

619/

Failed due to other prior failures. Re-test pending.

dev-sdc-sdc-onboarding-be-576864dfbc-x8pt8

05:40:05

09:03

PASS

30/


dev-sdc-sdc-wfd-be-7b84b4bf97-stlxs

06:04:46

02:11

PASS

31/


dev-sdc-sdc-wfd-fe-86597867d6-hw725

06:22:13

01:41

PASS

32/


dev-sdnc-controller-blueprints-684f7d865f-4pglc

06:39:13

01:08

PASS

33/


dev-sdnc-controller-blueprints-db-0

06:55:57

01:48

PASS

34/


dev-sdnc-nengdb-0

07:12:23

01:42

PASS

35/


dev-sdnc-network-name-gen-7f54b5649c-w4j8x

07:28:47

02:37

PASS

36/


dev-sdnc-sdnc-0

07:46:43

03:45

PASS

37/


dev-sdnc-sdnc-ansible-server-8b58d86f5-d6hz2

08:05:46

03:16

PASS

38/


dev-sdnc-sdnc-db-0

08:24:10

03:52

FAIL

39/

After sdnc-sdnc-db was deleted, the SDNC portal continues to fail due to inability to connect to the DB until sdnc-portal was manually restarted. Logged

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keySDNC-535

dev-sdnc-sdnc-dgbuilder-77866dbfb5-44jdswk4zj

20:08:3209

02:43

09:09

40/

Failed due to other prior failures. Re-test pending.

PASS

8/


dev-sdnc-sdnc-dmaap-listener-6f67f97dd9-dv5r7crfxr

0819:4658:4125

01:0605

PASS

417/

Failed due to other prior failures. Re-test pending.

dev-sdnc-sdnc-portal-79f65cfcf8-4jh8bl8nq8

0819:5246:3336

03:4107

PASS

426/

Failed due to other prior failures. Re-test pending.

dev-sdnc-sdnc-ueb-listener-557c57577c-knsskhzt2v

0919:35:08

02:254705:20

PASS

43/

5/


dev-so-so-575dcc7fc5-8g626stw5p

0919:1225:3216

01:0506

PASS

44/

4/


dev-so-so-bpmn-infra-74875c7c-l6pl96k4lz

0919:1815:2700

01:1411

PASS

45/

3/


dev-so-so-catalog-db-adapter-67878b4fc8-mzwg2

09:24:36

03:53

PASS

46/


dev-so-so-mariadb-576ccb887f-vh2f2

09:45:07

08:27

PASS

47/


dev-so-so-monitoring-7b4fdbf895-szgtp

10:11:45

01:40

PASS

48/


dev-so-so-openstack-adapter-85576544b6-xmd62

10:28:14

01:35

PASS

49/


dev-so-so-request-db-adapter-658db4b75d-nk7kpgfn6g

1719:3105:5703

01:1611

PASS

592/

Failed due to other prior failures. Re-test pending.

dev-so-so-sdc-controller-679f5dff6d-rxgf5

14:50:51

01:17

PASS

51/


dev-so-so-sdnc-adapter-54d99774c9-qgp6w

15:10:15

01:05

PASS

52/


dev-so-so-vfc-adapter-5bb49bc97f-7rvzj

15:31:05

01:03

PASS

53/


dev-vid-vid-54f8c6d78-jj8vp

15:49:13

04:48

PASS

54/


dev-vid-vid-mariadb-galera-0

16:21:42

01:09

FAIL

55/

After vid-mariadb-galera was deleted, the instance creation via VID continue to fail even after the vid pod was also restarted. Logged

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyVID-358


Closed Loop

Pod Deleted

Start Time

(PST)

Recovery Time

(MM:SS)

Pass / FailLogsNotes

dep-config-binding-service-596688f976-tdrpz

05:29:55

02:08

PASS

1/


dep-dcae-datafile-collector-65df56bc6b-4cj5p

05:35:36

01:14

PASS

2/


dep-dcae-hv-ves-collector-9df8d946-nz85f

05:39:33

00:45

PASS

3/


dep-dcae-prh-6bfbb97fdc-n2pvb

05:43:18

01:09

PASS

4/


dep-dcae-snmptrap-collector-58bbcb6644-g9tkw

05:47:10

02:36

PASS

5/


dep-dcae-tca-analytics-7dbddb9697-d2h8m

05:51:14

03:11

PASS

6/


dep-dcae-ves-collector-749996c5cb-5lf6g

05:57:39

01:35

PASS

7/


dep-deployment-handler-69dd5c4d56-5w4vc

06:01:12

06:47

PASS

8/


dep-holmes-engine-mgmt-7bdd595854-m7d4v

06:10:27

01:03

PASS

9/


dep-holmes-rule-mgmt-b99586579-znrhl

06:13:14

01:03

PASS

10/


dep-inventory-9585c9d6-9jtxv

06:16:15

01:14

PASS

11/


dep-policy-handler-775c84f5d4-tf27n

06:19:12

02:37

PASS

12/


dep-pstg-write-f67b57947-7fmzq

06:24:17

01:42

PASS

13/


dep-service-change-handler-b8cf87787-dzwcq

06:28:12

01:35

PASS

14/


dev-aai-aai-64dd9545fc-tlj5s

06:31:15

01:04

PASS

15/


dev-aai-aai-babel-7cb986869d-4qfxs

06:34:17

01:36

PASS

16/


dev-aai-aai-cassandra-0

06:37:37

01:14

PASS

17/


dev-aai-aai-champ-6b6d4779d8-9drmz

06:40:35

04:14

PASS

18/


dev-aai-aai-data-router-6b4776bfc4-8hb4k

06:46:18

09:01

PASS

19/


dev-aai-aai-elasticsearch-669f4664c-wcclz

06:57:17

01:13

PASS

20/


dev-aai-aai-gizmo-9df94f898-dg79k

07:01:14

04:50

PASS

21/


dev-aai-aai-graphadmin-5d6d4d86dc-6shv6

07:08:18

02:50

PASS

22/


dev-aai-aai-modelloader-54f7444dcd-qnftb

07:14:22

01:06

PASS

23/


dev-aai-aai-resources-8579748958-wwf5m

07:16:26

03:14

PASS

24/


dev-aai-aai-search-data-695b878655-tff7s

07:22:08

01:06

PASS

25/


dev-aai-aai-sparky-be-9bcc99756-ztfcg

07:26:14

17:05

PASS

26/


dev-aai-aai-spike-59c6db6574-v6dfj

07:46:19

03:51

PASS

27/


dev-aai-aai-traversal-7f5669d56-55rnr

07:51:39

01:41

PASS

28/


dev-appc-appc-0

07:56:18

58:38

FAIL

29/

APPC took 75 minutes to recover. The testing was done against a single node APPC due to

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyAPPC-1242
. Also, the Netconf mount was lost when appc pod was dropped and had to be re-created.

dev-appc-appc-ansible-server-c4678946c-qncqx

08:55:11

32:10

PASS

30/


dev-appc-appc-cdt-868f9896b-82mlj

09:31:39

00:40

PASS

31/


dev-appc-appc-db-0

09:35:17

01:41

PASS

32/


dev-appc-appc-dgbuilder-5d5cbfdd45-q7nbv

09:39:26

02:39

PASS

33/


dev-dcaegen2-dcae-bootstrap-7c5c8d7d76-ng999

09:45:20

01:08

PASS

34/


dev-dcaegen2-dcae-cloudify-manager-67b576dd66-86kzx

09:49:12

01:46

PASS

35/


dev-dcaegen2-dcae-db-0

09:53:12

03:10

PASS

36/


dev-dcaegen2-dcae-healthcheck-65c6548689-r6qbq

09:59:07

02:09

PASS

37/


dev-dcaegen2-dcae-pgpool-78fbd864cd-hhzcv

10:04:15

01:05

PASS

38/


dev-dcaegen2-dcae-pgpool-78fbd864cd-k8lgx

10:06:34

02:46

PASS

39/


dev-dcaegen2-dcae-redis-0

10:11:32

02:17

PASS

40/


dev-dcaegen2-dcae-redis-3

10:16:34

01:35

PASS

41/


dev-dcaegen2-dcae-redis-4

10:21:09

01:43

PASS

42/


dev-dcaegen2-dcae-redis-5

10:25:36

01:35

PASS

43/


dev-dmaap-dbc-pg-0

10:30:41

02:07

PASS

44/


dev-dmaap-dbc-pgpool-7b748d5894-7sfcw

10:35:32

01:04

PASS

45/


dev-dmaap-dbc-pgpool-7b748d5894-ngxxj

10:38:34

00:44

PASS

46/


dev-dmaap-dmaap-bus-controller-765495b674-642jf

10:41:34

01:03

PASS

47/


dev-dmaap-dmaap-dr-db-56b956df8d-rwffp

10:45:05

01:13

PASS

48/


dev-dmaap-dmaap-dr-node-64864d5cc-r8ck8

10:49:18

01:10

PASS

49/


dev-dmaap-dmaap-dr-prov-69bd7c6665-676cf

10:51:27

01:03

PASS

50/


dev-dmaap-message-router-647bbfc54d-h588k

10:55:14

03:08

PASS

51/


dev-dmaap-message-router-kafka-6f54ccf867-5mkbg

11:01:37

02:43

PASS

52/


dev-dmaap-message-router-zookeeper-847656487f-mngbc

11:07:34

01:45

FAIL

53/

After dmaap-message-router-zookeeper was deleted, vFWCL continued to fail until dmaap-message-router was subsequently restarted manually. Logged

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyDMAAP-893

dev-policy-brmsgw-78db5d4db4-n6b9j

17:15:06

02:15

PASS

63/


dev-policy-drools-0

17:20:05

02:17

PASS

64/


dev-policy-nexus-7d5f84d8b6-xzmwd

17:28:28

02:15

PASS

65/


dev-policy-pap-9cb89bf74-5cfmt

17:33:14

02:39

PASS

66/


dev-policy-pdp-0

17:39:23

01:38

PASS

67/


dev-policy-policy-apex-pdp-0

17:44:00

02:48

PASS

68/


dev-policy-policydb-74d46689cb-fmrc6

17:55:16

02:44

PASS

70/


dev-policy-policy-distribution-75b76b55f7-drknk

17:50:01

03:15

PASS

69/


Closed Loop


Requirement

Area

Priority

Min. Level

Stretch Goal

Level Descriptions (Abbreviated)

Resiliency

High

Level 2 – run-time projects
Level 1 – remaining projects

Level 3 – run-time projects
Level 2 – remaining projects

•1 – manual failure and recovery (< 30 minutes)
•2 – automated detection and recovery (single site) (<30 minutes)
•3 – automated detection and recovery (geo redundancy)