AAI Rolling Upgrade

From Dublin AAI would use the common instance of Cassandra cluster. The common Cassandra cluster is a new sub component of ONAP and can be installed by setting cassandra.enabled=true in the overrides or command line.

1. Fresh Dublin:

helm deploy demo local/onap --version=4.0.0 --namespace onap --set global.pullPolicy=IfNotPresent --set aai.enabled=true --set cassandra.enabled=true --set aaf.enabled=true --verbose --timeout 1200

root@new-k8s-1:~/Mahendra/OOM_Dublin# kubectl get pod -n onap
NAME READY STATUS RESTARTS AGE
demo-aai-aai-965567b77-4fh5w 1/1 Running 0 8m
demo-aai-aai-babel-55b56bdfd7-v2lq2 2/2 Running 0 8m
demo-aai-aai-champ-85947b9f8d-5lxp5 2/2 Running 0 8m
demo-aai-aai-data-router-75485c5556-p94t8 2/2 Running 0 8m
demo-aai-aai-elasticsearch-5fdc4f656-shnf5 1/1 Running 0 8m
demo-aai-aai-gizmo-6846bf8c58-s88pp 2/2 Running 0 8m
demo-aai-aai-graphadmin-5d5fc64cb7-l5c42 2/2 Running 0 8m
demo-aai-aai-graphadmin-create-db-schema-q8gbk 0/1 Completed 0 8m
demo-aai-aai-modelloader-65998fd66d-vldll 2/2 Running 0 8m
demo-aai-aai-resources-7b4d496cd8-5nww8 2/2 Running 0 8m
demo-aai-aai-schema-service-b9875f6f5-klm8n 2/2 Running 0 8m
demo-aai-aai-search-data-657996f67-xtts6 2/2 Running 0 8m
demo-aai-aai-sparky-be-547749c6b9-4kfd9 2/2 Running 0 8m
demo-aai-aai-spike-6c767c8cf7-z4g48 0/2 Init:0/1 0 8m
demo-aai-aai-traversal-544d64cd74-lztph 2/2 Running 0 8m
demo-aai-aai-traversal-update-query-data-mlngn 0/1 Completed 0 3m
demo-cassandra-cassandra-0 1/1 Running 0 8m
demo-cassandra-cassandra-1 1/1 Running 0 6m
demo-cassandra-cassandra-2 1/1 Running 0 5m

helm deploy demo local/onap --version=4.0.0 --namespace onap --set global.pullPolicy=IfNotPresent --set aai.enabled=true --set cassandra.enabled=true --set aaf.enabled=true --verbose --timeout 1200

root@new-k8s-1:~/Mahendra/OOM_Dublin# kubectl get pod -n onap
NAME READY STATUS RESTARTS AGE
demo-aai-aai-965567b77-4fh5w 1/1 Running 0 8m
demo-aai-aai-babel-55b56bdfd7-v2lq2 2/2 Running 0 8m
demo-aai-aai-champ-85947b9f8d-5lxp5 2/2 Running 0 8m
demo-aai-aai-data-router-75485c5556-p94t8 2/2 Running 0 8m
demo-aai-aai-elasticsearch-5fdc4f656-shnf5 1/1 Running 0 8m
demo-aai-aai-gizmo-6846bf8c58-s88pp 2/2 Running 0 8m
demo-aai-aai-graphadmin-5d5fc64cb7-l5c42 2/2 Running 0 8m
demo-aai-aai-graphadmin-create-db-schema-q8gbk 0/1 Completed 0 8m
demo-aai-aai-modelloader-65998fd66d-vldll 2/2 Running 0 8m
demo-aai-aai-resources-7b4d496cd8-5nww8 2/2 Running 0 8m
demo-aai-aai-schema-service-b9875f6f5-klm8n 2/2 Running 0 8m
demo-aai-aai-search-data-657996f67-xtts6 2/2 Running 0 8m
demo-aai-aai-sparky-be-547749c6b9-4kfd9 2/2 Running 0 8m
demo-aai-aai-spike-6c767c8cf7-z4g48 0/2 Init:0/1 0 8m
demo-aai-aai-traversal-544d64cd74-lztph 2/2 Running 0 8m
demo-aai-aai-traversal-update-query-data-mlngn 0/1 Completed 0 3m
demo-cassandra-cassandra-0 1/1 Running 0 8m
demo-cassandra-cassandra-1 1/1 Running 0 6m
demo-cassandra-cassandra-2 1/1 Running 0 5m

Note: aai-spike is in INIT as no dmaap (lack of resources).

2. Upgrade Casablanca to Dublin (In-place upgrade):

To allow in-place upgrade, Cassandra upgrade strategy should be RollingUpdate in the Casablanca.

2.1 Install Casablanca:

helm deploy demo local/onap --version=3.0.0 --namespace onap -f onap/resources/environments/disable-allcharts.yaml --set aai.enabled=true --set aai.global.aafEnabled=false --set global.pullPolicy=IfNotPresent --timeout 1200 --verbose

root@new-k8s-1:~/Mahendra/OOM_Dublin/AAI_ROLLING_UP_MERGED/oom/kubernetes# kubectl get pod -n onap|grep -v aaf
NAME READY STATUS RESTARTS AGE
demo-aai-aai-56db756b59-ztjkq 1/1 Running 0 10m
demo-aai-aai-babel-646b7859c8-52h7c 2/2 Running 0 10m
demo-aai-aai-cassandra-0 1/1 Running 2 10m
demo-aai-aai-cassandra-1 1/1 Running 0 6m
demo-aai-aai-cassandra-2 1/1 Running 0 5m
demo-aai-aai-champ-57fdb6f574-dtkvl 1/2 Running 0 10m
demo-aai-aai-data-router-75485c5556-rzgk2 2/2 Running 0 10m
demo-aai-aai-elasticsearch-64dc77ddbf-msrvd 1/1 Running 0 10m
demo-aai-aai-gizmo-6794664d68-hqff8 2/2 Running 0 10m
demo-aai-aai-graphadmin-8b4587bb7-2snlb 2/2 Running 0 10m
demo-aai-aai-graphadmin-create-db-schema-ntwdp 0/1 Completed 0 10m
demo-aai-aai-modelloader-65998fd66d-jncrl 2/2 Running 0 10m
demo-aai-aai-resources-55fb6b85c7-4fgv8 2/2 Running 0 10m
demo-aai-aai-search-data-657996f67-vmvqk 2/2 Running 0 10m
demo-aai-aai-sparky-be-75556466c4-lhhcm 2/2 Running 0 10m
demo-aai-aai-spike-6777bdcd97-6lgbs 0/1 Init:0/1 0 10m
demo-aai-aai-traversal-76b4c56997-sx65w 2/2 Running 0 10m
demo-aai-aai-traversal-update-query-data-wtps6 0/1 Completed 0 10m

helm deploy demo local/onap --version=3.0.0 --namespace onap -f onap/resources/environments/disable-allcharts.yaml --set aai.enabled=true --set aai.global.aafEnabled=false --set global.pullPolicy=IfNotPresent --timeout 1200 --verbose

root@new-k8s-1:~/Mahendra/OOM_Dublin/AAI_ROLLING_UP_MERGED/oom/kubernetes# kubectl get pod -n onap|grep -v aaf
NAME READY STATUS RESTARTS AGE
demo-aai-aai-56db756b59-ztjkq 1/1 Running 0 10m
demo-aai-aai-babel-646b7859c8-52h7c 2/2 Running 0 10m
demo-aai-aai-cassandra-0 1/1 Running 2 10m
demo-aai-aai-cassandra-1 1/1 Running 0 6m
demo-aai-aai-cassandra-2 1/1 Running 0 5m
demo-aai-aai-champ-57fdb6f574-dtkvl 1/2 Running 0 10m
demo-aai-aai-data-router-75485c5556-rzgk2 2/2 Running 0 10m
demo-aai-aai-elasticsearch-64dc77ddbf-msrvd 1/1 Running 0 10m
demo-aai-aai-gizmo-6794664d68-hqff8 2/2 Running 0 10m
demo-aai-aai-graphadmin-8b4587bb7-2snlb 2/2 Running 0 10m
demo-aai-aai-graphadmin-create-db-schema-ntwdp 0/1 Completed 0 10m
demo-aai-aai-modelloader-65998fd66d-jncrl 2/2 Running 0 10m
demo-aai-aai-resources-55fb6b85c7-4fgv8 2/2 Running 0 10m
demo-aai-aai-search-data-657996f67-vmvqk 2/2 Running 0 10m
demo-aai-aai-sparky-be-75556466c4-lhhcm 2/2 Running 0 10m
demo-aai-aai-spike-6777bdcd97-6lgbs 0/1 Init:0/1 0 10m
demo-aai-aai-traversal-76b4c56997-sx65w 2/2 Running 0 10m
demo-aai-aai-traversal-update-query-data-wtps6 0/1 Completed 0 10m

2.2 Upgrade to Dublin:

helm deploy demo local/onap --version=4.0.0 --namespace onap --set global.pullPolicy=IfNotPresent --set aai.enabled=true --set aai.global.jobs.migration.enabled=true --set aai.global.cassandra.existingInstServiceName=aai-cassandra --set cassandra.enabled=true --set aaf.enabled=true --set robot.enabled=true --verbose --timeout 1200

NAME READY STATUS RESTARTS AGE
demo-aai-aai-6fd446fd6b-hrjnf 1/1 Running 0 52m
demo-aai-aai-babel-55b56bdfd7-spk2n 2/2 Running 0 52m
demo-aai-aai-champ-85947b9f8d-27kmp 1/2 Running 0 52m
demo-aai-aai-data-router-75485c5556-rzgk2 2/2 Running 0 1h
demo-aai-aai-elasticsearch-5fdc4f656-4zj2j 1/1 Running 0 52m
demo-aai-aai-gizmo-6846bf8c58-kvjfd 2/2 Running 0 52m
demo-aai-aai-graphadmin-849b96d756-sp8jt 2/2 Running 0 52m
demo-aai-aai-graphadmin-db-backup-job-9hjcr 0/1 Completed 0 53m
demo-aai-aai-graphadmin-db-config-8pf7x 0/1 Completed 0 50m
demo-aai-aai-modelloader-65998fd66d-jncrl 2/2 Running 0 1h
demo-aai-aai-resources-5d6756fd59-4p2nv 2/2 Running 0 52m
demo-aai-aai-schema-service-b9875f6f5-cmpxx 2/2 Running 0 52m
demo-aai-aai-search-data-657996f67-vmvqk 2/2 Running 0 1h
demo-aai-aai-sparky-be-547749c6b9-9fm97 2/2 Running 0 52m
demo-aai-aai-spike-6c767c8cf7-j4hvr 0/2 Init:0/1 5 52m
demo-aai-aai-traversal-57b474df6c-fzklm 2/2 Running 0 52m
demo-aai-aai-traversal-update-query-data-xqtfw 0/1 Completed 0 46m
demo-cassandra-cassandra-0 1/1 Running 0 53m
demo-cassandra-cassandra-1 1/1 Running 0 51m
demo-cassandra-cassandra-2 1/1 Running 0 49m
demo-robot-robot-785f67844-6x89r 1/1 Running 0 3m

root@new-k8s-1:~/Mahendra/OOM_Dublin/AAI_ROLLING_UP_MERGED/oom/kubernetes# helm ls
NAME REVISION UPDATED STATUS CHART NAMESPACE
demo 3 Mon Mar 18 10:27:30 2019 DEPLOYED onap-4.0.0 onap
demo-aaf 2 Mon Mar 18 10:27:31 2019 DEPLOYED aaf-4.0.0 onap
demo-aai 2 Mon Mar 18 09:38:01 2019 DEPLOYED aai-4.0.0 onap
demo-cassandra 2 Mon Mar 18 10:27:33 2019 DEPLOYED cassandra-4.0.0 onap
demo-robot 1 Mon Mar 18 10:27:41 2019 DEPLOYED robot-4.0.0 onap

helm deploy demo local/onap --version=4.0.0 --namespace onap --set global.pullPolicy=IfNotPresent --set aai.enabled=true --set aai.global.jobs.migration.enabled=true --set aai.global.cassandra.existingInstServiceName=aai-cassandra --set cassandra.enabled=true --set aaf.enabled=true --set robot.enabled=true --verbose --timeout 1200

NAME READY STATUS RESTARTS AGE
demo-aai-aai-6fd446fd6b-hrjnf 1/1 Running 0 52m
demo-aai-aai-babel-55b56bdfd7-spk2n 2/2 Running 0 52m
demo-aai-aai-champ-85947b9f8d-27kmp 1/2 Running 0 52m
demo-aai-aai-data-router-75485c5556-rzgk2 2/2 Running 0 1h
demo-aai-aai-elasticsearch-5fdc4f656-4zj2j 1/1 Running 0 52m
demo-aai-aai-gizmo-6846bf8c58-kvjfd 2/2 Running 0 52m
demo-aai-aai-graphadmin-849b96d756-sp8jt 2/2 Running 0 52m
demo-aai-aai-graphadmin-db-backup-job-9hjcr 0/1 Completed 0 53m
demo-aai-aai-graphadmin-db-config-8pf7x 0/1 Completed 0 50m
demo-aai-aai-modelloader-65998fd66d-jncrl 2/2 Running 0 1h
demo-aai-aai-resources-5d6756fd59-4p2nv 2/2 Running 0 52m
demo-aai-aai-schema-service-b9875f6f5-cmpxx 2/2 Running 0 52m
demo-aai-aai-search-data-657996f67-vmvqk 2/2 Running 0 1h
demo-aai-aai-sparky-be-547749c6b9-9fm97 2/2 Running 0 52m
demo-aai-aai-spike-6c767c8cf7-j4hvr 0/2 Init:0/1 5 52m
demo-aai-aai-traversal-57b474df6c-fzklm 2/2 Running 0 52m
demo-aai-aai-traversal-update-query-data-xqtfw 0/1 Completed 0 46m
demo-cassandra-cassandra-0 1/1 Running 0 53m
demo-cassandra-cassandra-1 1/1 Running 0 51m
demo-cassandra-cassandra-2 1/1 Running 0 49m
demo-robot-robot-785f67844-6x89r 1/1 Running 0 3m

root@new-k8s-1:~/Mahendra/OOM_Dublin/AAI_ROLLING_UP_MERGED/oom/kubernetes# helm ls
NAME REVISION UPDATED STATUS CHART NAMESPACE
demo 3 Mon Mar 18 10:27:30 2019 DEPLOYED onap-4.0.0 onap
demo-aaf 2 Mon Mar 18 10:27:31 2019 DEPLOYED aaf-4.0.0 onap
demo-aai 2 Mon Mar 18 09:38:01 2019 DEPLOYED aai-4.0.0 onap
demo-cassandra 2 Mon Mar 18 10:27:33 2019 DEPLOYED cassandra-4.0.0 onap
demo-robot 1 Mon Mar 18 10:27:41 2019 DEPLOYED robot-4.0.0 onap

3. Install Dublin as a parallel site to Casablanca (The Cassandra cluster of AAI from Casablanca site should be exposed to outside world):

helm deploy demo local/onap --version=4.0.0 --namespace onap --set aai.enabled=true --set global.pullPolicy=IfNotPresent --verbose --timeout 1200 --set aai.global.jobs.migration.enabled=true --set aai.global.jobs.migration.remoteCassandra.enabled=true --set aai.global.jobs.migration.remoteCassandra.storage.hostname=10.53.234.224 --set cassandra.enabled=true --set aaf.enabled=true --set robot.enabled=true

4. Subsequent upgrades from Dublin onward:

a. With Migration:

helm deploy demo local/onap --version=4.0.0 --namespace onap --set global.pullPolicy=IfNotPresent --set aai.enabled=true --set aai.global.jobs.migration.enabled=true --set cassandra.enabled=true --set aaf.enabled=true --set robot.enabled=true --verbose --timeout 1200

helm deploy demo local/onap --version=4.0.0 --namespace onap --set global.pullPolicy=IfNotPresent --set aai.enabled=true --set aai.global.jobs.migration.enabled=true --set cassandra.enabled=true --set aaf.enabled=true --set robot.enabled=true --verbose --timeout 1200

b. Without Migration:

helm deploy demo local/onap --version=4.0.0 --namespace onap --set global.pullPolicy=IfNotPresent --set aai.enabled=true --set cassandra.enabled=true --set aaf.enabled=true --set robot.enabled=true --verbose --timeout 1200

helm deploy demo local/onap --version=4.0.0 --namespace onap --set global.pullPolicy=IfNotPresent --set aai.enabled=true --set cassandra.enabled=true --set aaf.enabled=true --set robot.enabled=true --verbose --timeout 1200

5. Running AAI with its own cluster:

AAI will create its own cassandra cluster for its specific use instead of connecting to the common Cassandra cluster, Below command will instantiate the aai cassandra instances:

helm deploy demo local/onap --version=4.0.0 --namespace onap --set aai.enabled=true --set aai.global.cassandra.localCluster=true --set aai.global.cassandra.serviceName=aai-cassandra

helm deploy demo local/onap --version=4.0.0 --namespace onap --set aai.enabled=true --set aai.global.cassandra.localCluster=true --set aai.global.cassandra.serviceName=aai-cassandra

Note:The AAI specific cluster will be using the common Cassandra charts to instantiate the nodes of the cluster.