The new Beijing release scalability, resiliency, and manageablity are described here.   These capabilities apply to the OOM/Kubernetes installation.

Installation

Follow the OOM installation instructions at 

http://onap.readthedocs.io/en/latest/submodules/oom.git/docs/index.html

1.  

   1. Find Rancher IP in R_Control-Plane tenant - we use “onap_dev” key and “Ubuntu” user to SSH. For example: “ssh -i onap_dev ubuntu@192.168.xx.xxx”.

   2. Login as ubuntu, then run "sudo -i" to login as root. The “oom” git repo is in the rancher vm's root directory, under “/root/oom”.

   3. Edit portal files at /root/oom/kubernetes/portal....

   4. When complete run the following from /root/oom/kubernetes  dir

      > make portal .... > make onap .... # This will do the upgrade: helm upgrade dev local/onap --namespace onap -f onap/resources/environments/disable-allcharts.yaml --set portal.enabled=true # When complete run the following to ensure its in the process of coming up or complete. kubectl get pods -n onap -o wide | grep portal

      In case, if it is a fresh install, try below command:

      helm install ./onap/ -n onap --namespace onap -f onap/resources/environments/disable-allcharts.yaml --set portal.enabled=true

      Stop command to stop any old namespace:

      helm del onap --purge

   5. Rancher gui is at 192.168.xx.xxx:8080
      There is an interactive cli as well in the rancher gui where you can run kubectl commands.

   6. Below command will show the list of portal services.

Overview of the running system

Healthchecks

Verify that the portal healtcheck passes by the robot framework:

https://jenkins.onap.org/view/External%20Labs/job/lab-tlab-beijing-oom-deploy/325/robot/OpenECOMP%20ETE/Robot/Testsuites/Health-Check/

Open

portal-app Scaling

To scale a new portal-app, set the replica count appropriately.   

In our tests below, we are going to work with the OOM portal component in isolation.   In this exercise, we scale the portal-app with 2 new replicas.

The below code needs to be added to the integration-override.yaml file.

Run below command from "/root” folder to do helm upgrade

^{helm upgrade -i dev local/onap -f integration-override.yaml}

portal-app Resiliency

A portal-app container failure can be simulated by stopping the portal-app container.    The kubernetes liveness operation will detect that the ports are down, inferring there's a problem with the service, and in turn, will restart the container. 

Here is where one of the running instances was deleted. And another is starting.

Open

After Deleteing 1 portal App.

Here the new instance has started and the old one is still terminating

After the deleted instance has terminated the 3 instances are all running normally.

During this time there was no issues with the Portal Website.

TODO: In the following releases, the portal-db, portal-widget needs to be tested with similar resiliency and scaling tests. The portal-cassandra and portal-zookeeper requires coordination with MUSIC team to test the resiliency as there are some plans to provide MUSIC as a service. 

Sanity Tests 

Check Portal UI and perform sanity tests.

1. After 3 instances of Portal are up, edit IP in /etc/hosts file, and logon as demo user on http://portal.api.simpledemo.onap.org:30215/ONAPPORTAL/login.htm
   
   

2. Then killed 1 instance, I am able to continue on Portal page seamlessly
   
   

3. Another test on failover timing, when killed all 3 instances, the new Portal processes are coming up within 30 seconds

Troubleshoot

In case of failures, below commands may get handy for accessing the pods/logs and troubleshooting the issue.

To find all portal pods:

From above list, now to check DB logs:

To check portal-app logs:

To get inside the portal-app docker and access the application logs:

Portal’s Rocket Chat channel - http://onap-integration.eastus.cloudapp.azure.com:3000/channel/portal

References:

1. K8S / helm basic commands for ONAP integration