Casablanca Maintenance Release Resiliency Testing Status

We will use vFW use case as the baseline to test this:

Pre-requirement: Instantiate a vFW with closed loop running.

Note: Pods which failed in Casablanca Release Resiliency Testing Status were not re-tested for the Maintenance Release.

VNF Onboarding and Distribution: 

Pod Deleted

Start Time

Duration

Result

Logs

Notes

Pod Deleted

Start Time

Duration

Result

Logs

Notes

dev-aai-aai-5696cbd55-csck6

18:41:42

08:01

SUCCESS

50



dev-aai-aai-babel-85c85878d5-cdww7

18:49:44

09:11

SUCCESS

51



dev-aai-aai-cassandra-0

18:58:55

08:02

SUCCESS

52



dev-aai-aai-cassandra-1

19:06:58

07:12

SUCCESS

53



dev-aai-aai-cassandra-2

19:14:11

07:37

SUCCESS

54



dev-aai-aai-champ-8598c7bf4f-l5vts

19:21:49

10:05

SUCCESS

55



dev-aai-aai-data-router-7d6b484764-qgcv4

19:31:55

06:01

SUCCESS

56



dev-aai-aai-elasticsearch-669f4664c-75hxk

19:37:57

06:52

SUCCESS

57



dev-aai-aai-gizmo-784bf59bb8-tmdts

19:44:49

06:00

SUCCESS

58



dev-aai-aai-graphadmin-5f9b7645d6-7bg8r

19:50:49

07:09

SUCCESS

59



dev-aai-aai-modelloader-7bb76855b5-7qck6

19:57:58

06:47

SUCCESS

60



dev-aai-aai-resources-55bd894dfc-snsw7

20:04:46

06:06

SUCCESS

61



dev-aai-aai-search-data-8d77c9d-s5g4l

20:10:52

11:03

SUCCESS

62



dev-aai-aai-sparky-be-577b95c76d-9dspz

20:21:56

44:45

SUCCESS

63



dev-aai-aai-spike-59c6db6574-c5nck

21:06:41

45:06

SUCCESS

64



dev-aai-aai-traversal-86f575c7f7-5mgnk

21:51:48

05:59

SUCCESS

65



dev-sdc-sdc-be-694b9b557b-ckk9h

21:57:47

11:09

SUCCESS

66



dev-sdc-sdc-dcae-be-5bb587d95c-9gk2h

22:08:57

11:08

SUCCESS

67



dev-sdc-sdc-dcae-dt-68d8d77578-vsstx

22:20:05

07:43

SUCCESS

68



dev-sdc-sdc-dcae-fe-79675b4bc4-nckl5

22:27:49

08:09

SUCCESS

69



dev-sdc-sdc-dcae-tosca-lab-6d9f59cbf6-k7rpz

22:35:58

07:02

SUCCESS

70



dev-sdc-sdc-es-768f76f64f-kbtgn

22:43:01

07:53

SUCCESS

71



dev-sdc-sdc-fe-6d98d5b7-57rmp

22:50:54

10:53

SUCCESS

72



dev-sdc-sdc-kb-7466db5dd-r2xr4

23:01:47

04:53

SUCCESS

73



dev-sdc-sdc-onboarding-be-76c6779fbf-gv8c9

23:06:41

08:18

SUCCESS

74



dev-sdc-sdc-wfd-be-7b84b4bf97-d9ngz

23:15:00

05:55

SUCCESS

75



dev-sdc-sdc-wfd-fe-86597867d6-bgm7q

23:20:55

05:53

SUCCESS

76



dev-sdnc-controller-blueprints-d9974c9fb-hdbb6

23:26:49

05:02

SUCCESS

77



dev-sdnc-controller-blueprints-db-0

23:31:52

04:59

SUCCESS

78



dev-sdnc-nengdb-0

23:36:52

06:59

SUCCESS

79



dev-sdnc-network-name-gen-d89496744-lrj88

23:43:52

07:12

SUCCESS

80



dev-sdnc-sdnc-0

23:51:04

12:44

SUCCESS

81



dev-sdnc-sdnc-ansible-server-5f8b66b9b4-qfqgj

00:03:49

07:04

SUCCESS

82



dev-sdnc-sdnc-db-0

00:10:54

05:58

SUCCESS

83



dev-sdnc-sdnc-dgbuilder-865fbf655d-2zbw7

00:16:53

07:53

SUCCESS

84



dev-sdnc-sdnc-dmaap-listener-7c9f456bf6-kt49n

00:24:47

07:15

SUCCESS

85



dev-sdnc-sdnc-portal-597cb97887-b4t42

00:32:02

06:50

SUCCESS

86



dev-sdnc-sdnc-ueb-listener-859d9d8b64-g5kkv

00:38:53

06:55

SUCCESS

87



dev-so-so-7c78d4c6cd-g5k7k

00:45:49

04:58

SUCCESS

88



dev-so-so-bpmn-infra-856bb6dd54-nkv55

00:50:47

06:53

SUCCESS

89



dev-so-so-catalog-db-adapter-768cf4c795-s7t5t

00:57:41

06:02

SUCCESS

90



dev-so-so-mariadb-7d548967c8-sbn56

01:03:44

09:20

FAILURE

91

Appears to be transient lab infrastructure issue.

dev-so-so-monitoring-68b8b96b6b-pzzml

01:13:04

07:45

SUCCESS

92



dev-so-so-openstack-adapter-54f6cdd667-vj2ff

01:20:50

05:01

SUCCESS

93



dev-so-so-request-db-adapter-7758c846f9-6hg4j

01:25:52

05:04

SUCCESS

94



dev-so-so-sdc-controller-6cd84f4797-cbjbn

01:30:56

05:03

SUCCESS

95



dev-so-so-sdnc-adapter-6b7c558f56-hbh9n

01:36:00

04:57

SUCCESS

96



dev-so-so-vfc-adapter-8664b4db56-gmp5p

01:40:57

05:01

SUCCESS

97



VNF Instantiation: 

Pod Deleted

Start Time

Duration

Result

Logs

Notes

Pod Deleted

Start Time

Duration

Result

Logs

Notes

dev-aai-aai-5696cbd55-sdslw

13:50:03

08:23

FAILURE

11

Dictionary does not contain value 'Ete_vFWSNK_8a24da92-3cb2-47dc-be01-dc486e358156'.

dev-aai-aai-babel-85c85878d5-vv6ks

13:58:26

06:54

FAILURE

12

Transient infrastructure issue.

dev-aai-aai-cassandra-0

14:05:21

12:23

SUCCESS

13



dev-aai-aai-cassandra-1

14:17:44

11:06

SUCCESS

14



dev-aai-aai-cassandra-2

14:28:51

11:05

SUCCESS

15



dev-aai-aai-champ-8598c7bf4f-swbcl

14:39:57

13:45

SUCCESS

16



dev-aai-aai-data-router-7d6b484764-6c6wt

14:53:42

13:05

SUCCESS

17



dev-aai-aai-elasticsearch-669f4664c-q84kf

15:06:48

15:16

SUCCESS

18



dev-aai-aai-gizmo-784bf59bb8-vj8hp

15:22:04

11:28

SUCCESS

19



dev-aai-aai-graphadmin-5f9b7645d6-nl6xp

15:33:33

11:32

SUCCESS

20



dev-aai-aai-modelloader-7bb76855b5-khhws

15:45:06

10:48

SUCCESS

21



dev-aai-aai-resources-55bd894dfc-bq6qm

15:55:54

08:22

FAILURE

22

Transient infrastructure issue.

dev-aai-aai-search-data-8d77c9d-462bx

16:04:17

11:40

SUCCESS

23



dev-aai-aai-sparky-be-577b95c76d-dqfn6

16:15:57

14:03

SUCCESS

24



dev-aai-aai-spike-59c6db6574-wrlq2

16:30:01

14:13

SUCCESS

25



dev-aai-aai-traversal-86f575c7f7-q5gvh

16:44:14

12:32

SUCCESS

26



dev-multicloud-multicloud-7cd96f75fb-hmgwp

16:56:46

10:49

SUCCESS

27



dev-multicloud-multicloud-azure-85548b6469-m4tqt

17:07:36

11:58

SUCCESS

28



dev-multicloud-multicloud-ocata-cfbc7dd4b-nxqnv

17:19:35

11:05

SUCCESS

29



dev-multicloud-multicloud-pike-57778fb4fc-q8gt2

17:30:40

10:50

SUCCESS

30



dev-multicloud-multicloud-vio-794f8b7d85-7h8qt

17:41:31

11:18

SUCCESS

31



dev-multicloud-multicloud-windriver-75b7659f96-m7kh9

17:52:49

10:52

SUCCESS

32



dev-sdc-sdc-be-694b9b557b-49bk7

18:03:41

12:19

SUCCESS

33



dev-sdc-sdc-cs-76f6c49f5d-r9lng

18:16:01

44:52

FAILURE

34

After deletion of sdc-cs, sdc-be pod falls into 1/2 Running status and never recovered. sdc-be required manual pod deletion to recover.

Logged SDC-1960: sdc-cs fails resilience testClosed

dev-sdc-sdc-dcae-be-5bb587d95c-bnbp9

20:00:54

47:03

SUCCESS

35



dev-sdc-sdc-dcae-dt-68d8d77578-9hfrw

20:47:58

11:51

SUCCESS

36



dev-sdc-sdc-dcae-fe-79675b4bc4-lgqql

20:59:49

11:38

SUCCESS

37



dev-sdc-sdc-dcae-tosca-lab-6d9f59cbf6-ltbqg

21:11:28

12:13

SUCCESS

38



dev-sdc-sdc-es-768f76f64f-dqxjb

21:23:42

13:24

SUCCESS

39



dev-sdc-sdc-fe-6d98d5b7-nzc6g

21:37:06

04:23

FAILURE

40

Transient infrastructure issue.

dev-sdc-sdc-kb-7466db5dd-sdkqk

21:41:30

12:58

SUCCESS

41



dev-sdc-sdc-onboarding-be-76c6779fbf-n82mh

21:54:28

12:43

SUCCESS

42



dev-sdc-sdc-wfd-be-7b84b4bf97-bsr7n

22:07:12

11:48

SUCCESS

43



dev-sdc-sdc-wfd-fe-86597867d6-kkhnt

22:19:00

14:03

SUCCESS

44



dev-sdnc-controller-blueprints-d9974c9fb-xdnnb

22:33:03

15:50

SUCCESS

45



dev-sdnc-controller-blueprints-db-0

22:48:54

23:42

SUCCESS

46



dev-sdnc-nengdb-0

23:12:37

16:08

SUCCESS

47



dev-sdnc-network-name-gen-d89496744-fskhr

23:28:45

15:02

SUCCESS

48



dev-sdnc-sdnc-0

23:43:48

21:11

SUCCESS

49



dev-sdnc-sdnc-ansible-server-5f8b66b9b4-hch2v

00:05:00

21:57

SUCCESS

50



dev-sdnc-sdnc-dgbuilder-865fbf655d-dzg5h

00:26:58

21:55

SUCCESS

51



dev-sdnc-sdnc-dmaap-listener-7c9f456bf6-7hs9x

00:48:53

11:47

SUCCESS

52



dev-sdnc-sdnc-portal-597cb97887-mpp4v

01:00:41

13:35

SUCCESS

53



dev-sdnc-sdnc-ueb-listener-859d9d8b64-7mzr5

01:14:16

16:01

SUCCESS

54



dev-so-so-7c78d4c6cd-8m5mf

01:30:18

15:19

SUCCESS

55



dev-so-so-bpmn-infra-856bb6dd54-zxr8d

01:45:37

21:42

SUCCESS

56



dev-so-so-catalog-db-adapter-768cf4c795-2c97j

02:07:19

09:45

FAILURE

57

Transient infrastructure issue.

dev-so-so-mariadb-7d548967c8-g6bbr

02:17:05

23:46

SUCCESS

58



dev-so-so-monitoring-68b8b96b6b-7b22z

02:40:52

20:01

SUCCESS

59



dev-so-so-openstack-adapter-54f6cdd667-9lhpq

03:00:54

14:41

SUCCESS

60



dev-so-so-request-db-adapter-7758c846f9-52gwj

03:15:35

14:07

SUCCESS

61



dev-so-so-sdc-controller-6cd84f4797-r6smq

03:29:42

16:12

SUCCESS

62



dev-so-so-sdnc-adapter-6b7c558f56-85j5t

03:45:55

14:08

SUCCESS

63



dev-so-so-vfc-adapter-8664b4db56-2bcd5

04:00:04

07:49

FAILURE

64



dev-vid-vid-78bc8969d9-blqpj

04:07:53

18:48

SUCCESS

65



Closed Loop: 

Pod Deleted

Start Time

Duration

Result

Logs

Notes

Pod Deleted

Start Time

Duration

Result

Logs

Notes

dep-dcae-snmptrap-collector-85c97f9db9-lzq4k

09:01:59

25:44

SUCCESS

76



dep-dcae-tca-analytics-f5f879f6f-2mdpp

09:27:44

02:09

SUCCESS

77



dep-dcae-ves-collector-77fdf87d88-z22k5

09:29:53

04:32

SUCCESS

78



dep-deployment-handler-674c8b8449-lzsqz

09:34:26

03:43

SUCCESS

79



dep-holmes-engine-mgmt-5558cb44b8-k7rsr

09:38:09

02:55

SUCCESS

80



dep-holmes-rule-mgmt-876f65647-b9874

09:41:04

03:08

SUCCESS

81



dep-inventory-654547f4-vnsw7

09:44:13

04:16

SUCCESS

82



dep-policy-handler-7cc8db9958-6vqzr

09:48:29

02:27

SUCCESS

83



dep-pstg-write-f67b57947-nz4s8

09:50:56

05:00

SUCCESS

84



dep-service-change-handler-679f589846-dsvn8

09:55:57

03:00

SUCCESS

85



dev-aai-aai-5696cbd55-csck6

09:58:58

02:49

SUCCESS

86



dev-aai-aai-babel-85c85878d5-cdww7

10:01:47

03:11

SUCCESS

87



dev-aai-aai-cassandra-0

10:04:59

03:37

SUCCESS

88



dev-aai-aai-cassandra-1

10:08:36

03:27

SUCCESS

89



dev-aai-aai-cassandra-2

10:12:03

02:53

SUCCESS

90



dev-aai-aai-champ-8598c7bf4f-l5vts

10:14:56

02:50

SUCCESS

91



dev-aai-aai-data-router-7d6b484764-qgcv4

10:17:47

02:23

SUCCESS

92



dev-aai-aai-elasticsearch-669f4664c-75hxk

10:20:10

02:54

SUCCESS

93



dev-aai-aai-gizmo-784bf59bb8-tmdts

10:23:04

03:06

SUCCESS

94



dev-aai-aai-graphadmin-5f9b7645d6-7bg8r

10:26:11

02:52

SUCCESS

95



dev-aai-aai-modelloader-7bb76855b5-7qck6

10:29:04

02:35

SUCCESS

96



dev-aai-aai-resources-55bd894dfc-snsw7

10:31:39

03:06

SUCCESS

97



dev-aai-aai-search-data-8d77c9d-s5g4l

10:34:45

02:10

SUCCESS

98



dev-aai-aai-sparky-be-577b95c76d-9dspz

10:36:56

02:05

SUCCESS

99



dev-aai-aai-spike-59c6db6574-c5nck

10:39:02

02:34

SUCCESS

100



dev-aai-aai-traversal-86f575c7f7-5mgnk

10:41:37

02:17

SUCCESS

101



dev-appc-appc-ansible-server-7fb64488c6-nv4g2

10:43:54

03:50

SUCCESS

102



dev-appc-appc-cdt-6f6784d579-7m69z

10:47:45

02:21

SUCCESS

103



dev-appc-appc-db-0

10:50:07

05:37

SUCCESS

104



dev-appc-appc-db-1

10:55:45

03:53

SUCCESS

105



dev-appc-appc-db-2

10:59:39

03:21

SUCCESS

106



dev-appc-appc-dgbuilder-687d76975d-k4gpx

11:03:01

03:04

SUCCESS

107



dev-dcaegen2-dcae-bootstrap-7c5c8d7d76-rk9lq

11:06:05

03:36

SUCCESS

108



dev-dcaegen2-dcae-cloudify-manager-67b576dd66-p9vj4

11:09:42

03:57

SUCCESS

109



dev-dcaegen2-dcae-db-0

11:13:39

04:28

SUCCESS

110



dev-dcaegen2-dcae-db-1

11:18:08

03:52

SUCCESS

111



dev-dcaegen2-dcae-healthcheck-65c6548689-jtpcr

11:22:00

04:36

SUCCESS

112



dev-dcaegen2-dcae-pgpool-78fbd864cd-nwfr9

11:26:37

03:36

SUCCESS

113



dev-dcaegen2-dcae-pgpool-78fbd864cd-tbn49

11:30:14

03:25

SUCCESS

114



dev-dcaegen2-dcae-redis-0

11:33:39

04:22

SUCCESS

115



dev-dcaegen2-dcae-redis-1

11:38:01

04:05

SUCCESS

116



dev-dcaegen2-dcae-redis-2

11:42:06

04:36

SUCCESS

117



dev-dcaegen2-dcae-redis-3

11:46:43

04:05

SUCCESS

118



dev-dcaegen2-dcae-redis-4

11:50:49

04:46

SUCCESS

119



dev-dcaegen2-dcae-redis-5

11:55:36

03:12

SUCCESS

120



dev-dmaap-dbc-pg-0

11:58:48

02:47

SUCCESS

121



dev-dmaap-dbc-pg-1

12:01:35

04:09

SUCCESS

122



dev-dmaap-dbc-pgpool-7b748d5894-r2fqq

12:05:45

03:14

SUCCESS

123



dev-dmaap-dbc-pgpool-7b748d5894-rxdj9

12:08:59

03:00

SUCCESS

124



dev-dmaap-dmaap-bus-controller-765495b674-c4nlr

12:12:00

03:40

SUCCESS

125



dev-dmaap-dmaap-dr-db-56b956df8d-qlqnm

12:15:41

02:17

SUCCESS

126



dev-dmaap-dmaap-dr-node-64864d5cc-4nzq7

12:17:58

04:38

SUCCESS

127



dev-dmaap-dmaap-dr-prov-69bd7c6665-kb45b

12:22:37

04:32

SUCCESS

128



dev-dmaap-message-router-647bbfc54d-4x7xt

12:27:09

07:30

SUCCESS

129



dev-dmaap-message-router-kafka-64465d9ff4-hgg8w

12:34:40

03:55

SUCCESS

130



dev-policy-brmsgw-d98d8dc5-zxjs5

12:38:36

04:22

SUCCESS

131



dev-policy-drools-0

12:42:58

08:49

SUCCESS

132



dev-policy-nexus-7d5f84d8b6-tq6pd

12:51:47

05:19

SUCCESS

133



dev-policy-pap-cd84c4fff-4ktpw

12:57:07

03:52

SUCCESS

134



dev-policy-pdp-0

13:00:59

04:55

SUCCESS

135



dev-policy-policy-apex-pdp-0

13:05:55

03:47

SUCCESS

136



dev-policy-policy-distribution-669bc685d4-rc2bx

13:09:42

04:12

SUCCESS

137



dev-policy-policydb-74d46689cb-5nvtw

13:13:55

06:47

SUCCESS

138



dep-dcae-datafile-collector-b58756f44-xz7hp

13:20:43

03:54

SUCCESS

139



dep-dcae-hv-ves-collector-689bdfb74-7lpks

13:24:38

01:32

FAILURE

140

Appears to be transient lab infrastructure issue.

dep-dcae-prh-7bcfc57dc-mprdp

13:26:10

03:37

SUCCESS

141



dep-config-binding-service-65986446f4-8m2fr

13:29:47

04:23

SUCCESS

142







Requirement

Area

Priority

Min. Level

Stretch Goal

Level Descriptions (Abbreviated)

Resiliency

High

Level 2 – run-time projects
Level 1 – remaining projects

Level 3 – run-time projects
Level 2 – remaining projects

•1 – manual failure and recovery (< 30 minutes)
•2 – automated detection and recovery (single site) (<30 minutes)
•3 – automated detection and recovery (geo redundancy)