Honolulu Resiliency and Backup and Restore test

Owned by Morgan Richomme
Last updated: Apr 28, 2021
2 min read

key summary type created updated due assignee reporter priority status resolution
Refresh



Worker restart test

List of pods tested for eviction

  • onap-aaf-cass

  • onap-aaf-cm-9ccbdf758-rl9k5

  • onap-aaf-fs-7b6648db7f-shcn5

  • onap-aaf-oauth-5896545fb7-x6grg

  • onap-aaf-sms-quorumclient-2

  • onap-aai-babel-6687fd4646-j4nld

  • onap-aai-modelloader-86d95c994b-87tsh

  • onap-aai-schema-service-75575cb488-7fxs4

  • onap-appc-cdt-58cb4766b6-vl78q

  • onap-appc-cdt-74db7bf896-slj5b

  • onap-appc-db-0

  • onap-appc-db-2

  • onap-appc-cdt-74db7bf896-dnz7r: init error

  • onap-appc-db

  • onap-appc-dgbuilder-5bb94d46bd-h2gbs: init error

  • onap-awx-0

  • onap-cassandra-1

  • onap-cassandra-1

  • onap-cds-blueprints-processor-76f8b9b5c7-hb5bg: init error (samsung) / was ok on my side (orange)

  • onap-cds-command-executor-cb5c67d44-9gdls

  • onap-cds-db-0

  • onap-cds-ui-6b476496c6-rzxqz: init error

  • onap-consul-server-2

  • onap-cps-postgres

  • onap-dbc-pg

  • onap-dcae-db-primary-7f7844b57-n7ltd

  • onap-dcae-db-replica-5b75c6c545-fpjpv

  • onap-dcae-inventory-api-6689866db6-khg8n

  • onap-dcae-config

  • onap-dcae-hv

  • onap-dcaemod-bd

  • onap-dcae-bootstrap: startup error (cfy secret create -s UmH7vbj6 cmpass: An error occurred on the server: 409: <Secret id=`cmpass` tenant=`default_tenant`> already exists on <Tenant name=`default_tenant`> or with global visibili

  • onap-dcaemod-designtool-77599757db-bf67k

  • onap-dcaemod-healthcheck

  • onap-dcaemod-runtime-api-66ff9555f6-lqnwb

  • onap-dcae-servicechange-handler-69c6b8b499-5sppx

  • onap-dcae-ves-openapi-manager-7b88b66495-rblgz

  • onap-dmaap-dr-db-1

  • onap-ejbca-6cbdb7d6dd-hmw6z

  • onap-esr-gui

  • onap-esr-server

  • onap-holmes-pg-replica-d46965c86-w8hgv

  • onap-kube2msb-858f46f95c-jws4m

  • onap-mariadb-galera-2

  • onap-message-router-0

  • onap-message-router-kafka-0

  • onap-message-router-kafka-1

  • onap-message-router-kafka-2

  • onap-message-router-zookeeper-0

  • onap-msb-eag

  • onap-msb-discovery

  • onap-music-64d55f77f9-hjddv

  • onap-nbi-mongo-0

  • onap-netbox-nginx-76ff6f4c7-29m8j

  • onap-multicloud-794c6dffc8-bfwr8

  • onap-multicoud-pike

  • onap-multicloud-starlingx-58f6b86c55-mff89

  • onap-multicloud-vio-584d556876-87lxn

  • onap-multicloud-windriver

  • onap-music-64d55f77f9-6gz4h

  • onap-music-cassandra-0

  • onap-netbox-nginx-8667d6675d-vszhb

  • onap-netbox-postgres

  • onap-network-name-gen-6b85d998b-5f98m

  • onap-oof-cmso-optimizer-d4db49d65-bz9r9

  • onap-oof-has

  • onap-policy-api-6dbf8485d7-k7cpv

  • onap-policy-api-747d66b79-h8vsm: exception on restart?

  • onap-policy-clamp-be-6d77597477-4mffk

  • onap-policy-pap-5d9d484bff-k5tjk

  • onap-policy-pap-785bd79759-xxhvx

  • onap-policy-xacml-pdp-7d8fd58d59-d4m7g

  • onap-portal-app-66d9f54446-vs474

  • onap-portal-sdk

  • onap-portal-db

  • onap-sdc-be-5f99c6c644-dcdz8: ? init error (samsung) but OK but after 2 retries and 23m

  • onap-sdc-fe-7577d58fb5-kwxpj

  • onap-sdc-wfd-fe-6997567759-gl9g6

  • onap-sdnc: ? first attempt OK but second (in a raw) seems to keep it in Init (with sdc-be)

  • onap-sdnc-dgbuilder-564d6475fd-xwwrz

  • onaps-dnc-dmaap-listener-5bc486c6b9-8llkr

  • onap-sdnrdb-master-0

  • onap-so-admin-cockpit-6c5b44694-h4d2n

  • onap-so-d64bc6597-v8vcs

  • onap-so-etsi-sol003-adapter-c9bf4464-pwn97

  • onap-so-etsi-sol005-adapter-c4c784ddc-2pm7q

  • onap-so-sdc-controller-6899b98b8b-hfgvc

  • onap-vfc-huawei

  • onap-vfc-mariadb-1

  • onap-vfc-generic-vnfm-driver-65b688f5bc-twtfl

  • onap-vfc-nslcm-6c67677546-xcvl2: init error

  • onap-vfc-vnflcm-6d98c4667-t4l6r: init error (on one test but not the other - only vfc pod evicted)

  • onap-vfc-vnflcm-78ff4d8778-sgtv6

  • onap-vfc-vnfres-6c96f9ff5b-swq5z: init error

  • onap-vid-598788dbb5-cgqvt

  • onap-vnfsdk-postgres

in bold pods with issues...

To be completed..

Discussions

We install ONAP several times a day in our different CI/CD chains however we always resinstall a new ONAP.

The Backup&Restore feature is a MUST HAVE for production, it may be invoked for different levels of B&R and different associated tests:



Type of B&R

Description

Comments

Type

Type of B&R

Description

Comments

Type

Disaster Recovery

Your ONAP is corrupted, despite of brilliant tricks, you are not able to re-enable it

You reinstall a new ONAP (same version) and restore everything you can..

Downtime is acceptable

Restart of component is acceptable

Basically everything is acceptable as the first ONAP is no more usable...

Reinstallation is a fresh reinstallation (with new passwords)

Full

Iso version Migration

Your ONAP is running well but for any reasons (capability, fun, ..) you want to switch and need a new ONAP (same version)

You install ONAP' on a new infrastructure

You freeze ONAP

you backup ONAP

you restore on ONAP'

you switch from ONAP to ONAP'

No Downtime

Full

ONAP Active/Standby

Belts and Braces

You are rich you have an ONAP in production and another ONAP in case the first one crashes.

You need to regularly resync the stanby ONAP

No Downtime

Differential/Mirror

Smooth Full Upgrade

You want to move from version N to version N+1 and keep all your services/VSP/VF, SO history/Loop/...

You install a version N+1 in // of version N

You freeze your version N, save everything you can

You restore N on N+1

You switch from N to N+1

No Downtime

Possible compatibility issues...infra, k8, ONAP API

Not it is possible to upgrade 1 component by editing the deployment...but it is out of scope event if a test could be also interesting

Incremental

Wild Upgrade

Same than Upgrade but between 2 consecutive versions (LTS versions)

No Downtime

compatibility issues ++

Incremental











3 or 4 types of Backup are usually defined:

  • Full

  • Incremental

  • Differential

  • (Mirror)

https://intrinium.com/the-four-most-common-types-of-data-backup/



OOM already worked on scripts to B&R some of the databases...