This wiki describes how to set up a Kubernetes cluster with kuberadm, and then deploying SDN-C within that Kubernetes cluster.

(To view the current page, Chrome is the preferred browser. IE may add extra "CR LF"  each line, which causes problems). 

In the OpenStack lab, the controller executes the function of partitioning resources. The compute nodes are the collection of resources (memory, CPUs, hard drive space) to be partitioned. When creating a VM with "X" memory, "Y" CPUs and "Z" hard drive space, OpenStack's controller reviews its pool of available resources, allocates the quota, and then creates the VM on one of the available compute nodes. Many VMs can be created on a single compute node. OpenStack's controller uses a lot of criteria to choose a compute node, but if an application spans multiple VMs, Affinity rules can be used to ensure the VMs don’t congregate on a single compute node. This would not be good for resilience.

Kubernetes is similar to OpenStack in that it manages resources. Instead of scheduling VMs, Kubernetes schedules Pods. In a Kubernetes cluster, there is a single master node and multiple worker nodes. The Kubernetes’s master node is like the OpenStack controller in that it allocates resources for Pods. Kubernetes worker nodes are the pool of resources to be allocated, similar to OpenStack’s compute nodes. Pods, like VMs, can have Affinity rules configured in order to increase Apps resilience.

If you would like more information on these subjects, please explore these links:

• https://www.openstack.org/

• https://kubernetes.io/

• https://www.docker.com/

The Kubernetes deployment in this tutorial will be set up on top of OpenStack VMs. Let's call this the undercloud. undercloud can be physical boxes, or VMs. The VMs can come from different cloud providers, but in this tutorial we will use OpenStack. The following table shows the layers of software that need to be considered when thinking about resilience:

Setting up an OpenStack lab is out of scope for this tutorial. Assuming that you have a lab, you will need to create 1+n VMs: one to be configured as the Kubernetes master node, and "n" to be considered as Kubernetes worker nodes. We will create 3 Kubernetes worker nodes for this tutorial because we want each of our SDN-Cs to appear on a different VM for resiliency.

Create the Undercloud

The examples here will use the OpenStackClient; however, the Openstack Horizon GUI could be used. Start by creating 4 VMs with the hostnames: k8s-master, k8s-node0, k8s-node1, and k8s-node1. Each VM should have internet access and approximately:

• 16384 MB

• 20 GB

• 4 vCPUs
  
  
  

Use the ubuntu 16.04 cloud image to create the VMs. This image can be found at https://cloud-images.ubuntu.com/.

Exactly how to create VMs in OpenStack is out of scope for this tutorial. However, here is some examples of what OpenStackClient commands can be used to perform this job:

Repeat the following steps on each VM:

Make sure the VMs are: 

• Up to date

• The clocks are synchonized 
  
  
  

As ubuntu user run the followings.

Question: Did you check date on all K8S nodes to make sure they are in synch?

The ONAP apps are pakages in Docker containers.

The following snippet was taken from https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#install-docker-ce:

Just install the pakages; there is no need to configure them yet.

The following snippet was taken from https://kubernetes.io/docs/setup/independent/install-kubeadm/:

Note: If you intend to remove kubernetes packages use  "apt autoremove kubelet; apt autoremove kubeadm;apt autoremove kubectl " .

kubeadm is a utility provided by Kubernetes which simplifies the process of configuring a Kubernetes cluster.  Details about how to use it can be found here: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/.

The kubeadm init command sets up the Kubernetes master node. SSH to the k8s-master VM and invoke the following command. It is important to capture the output into a log file as there is information which you will need to refer to afterwards.

Note: A new add-on named as "kube-dns" will be added to the master node.  However, there is a recommended option to replace it with "CoreDNS", by providing "--feature-gates=CoreDNS=true" parameter to "kubeadm init" command.

The output of "kubeadm init" (with kube-dns addon) will look like below:

Execute the following snippet (as ubuntu user) to get kubectl to work. 

Verify a set of pods are created. The coredns or kubedns will be in pending state.

A "Pod network" must be deployed to use the cluster. This will let pods to communicate with eachother.

There are many different pod networks to choose from. See https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network for choices. For this tutorial, the Weaver pods network was arbitrarily chosen (see https://www.weave.works/docs/net/latest/kubernetes/kube-addon/ for more information).

The following snippet will install the Weaver Pod network:

Pay attention to the new pod (and serviceaccount) for "wave-net" . This pod provdes pod-to-pod connectivity.

Verfiy status of the pods. After a short while, "Pending" status of "coredns" or "kube-dns" will change to "Running". 

• If any of the weave pods face a problem and gets stuck at "ImagePullBackOff " state, you can try running the " sudo kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')" " again. 

• Sometimes, you need to delete the problematic pod, to let it terminate and start fresh. Use "kubectl delete po/<pod-name> -n <name-space> " to delete a pod.

• To "Unjoin" a worker node "kubectl delete node <node-name> (go through the "Undeploy SDNC" process at the end if you have an SDNC cluster running)

• If for any reason you need to re-create kubernetes cluster, first remove /etc/kubernetes/ and /var/lib/etcd and /etc/systemd/system/kubelet.service.d/ . Then run kubeadm init command. 

ONAP uses Helm, a package manager for kubernetes.

Install helm (client side). The following instructions were taken from https://docs.helm.sh/using_helm/#installing-helm:

Note: You may need to install older version of helm, then follow "Downgrade helm" section (scroll down). 

Install Tiller(server side of hlem)

Tiller manages installation of helm packages (charts).  Tiller requires ServiceAccount setup in Kubernetes before being initialized. The following snippet will do that for you:

(Chrome as broswer is preferred. IE may add  extra "CR LF" to each line, which causes problems). 

Initialize helm....This command installs Tiller. It also discovers Kubernetes clusters by reading $KUBECONFIG (default '~/.kube/config') and using the default context.

Setting up cluster nodes is very easy. Just refer back to the "kubeadm init" output logs (/root/kubeadm_init.log). In the last line of the the logs, there is a “kubeadm join” command with token information and other parameters.

Capture those parameters and then execute it as root on each of the Kubernetes worker nodes:  k8s-node1, k8s-node2, and k8s-node3.

After running the "kubeadm join" command on a worker node,

• 2 new pods (proxy and weave) will be created on Master node and will be assigned to the worker node. 

• The tiller pod status will change to "running" . 

• The AVAILABLE flag for tiller-deploy deployment will be changed to "1".

• The worker node will join the cluster.

The command looks like the follwing snippet (find the command in bottom of /root/kubeadm_init.log):

Verify the results from master node:

Make sure you run the same "kubeadm join" command on all worker nodes once and verify the results. 

Return to the Kubernetes master node VM, execute the “kubectl get nodes“ command (from master node) to see all the Kubernetes nodes. It might take some time, but eventually each node will have a status of "Ready":

Make sure that the tiller pod is running. Execute the following command (from master node) and look for a po/tiller-deploy-xxxx with a “Running” status. For example:

(In the case of using coredns instead of kube-dns, you notice it will only one container)

Now you have a Kubernetes cluster with 3 worker nodes and 1 master node.

You can run " kubectl describe node" on the Master node and get a complete report on nodes (including workers) and thier system resources.

This is a shared directory which must be mounted on all of the Kuberenetes VMs(master node and worker nodes). Because many of the ONAP pods use this directory to share data.

See 3. Share the /dockerdata-nfs Folder between Kubernetes Nodes for instruction on how to set this up.

Clone OOM project only on Kuberentes Master Node

As ubuntu user, clone the oom repository. 

Get the following 2 gerrit changes from Configure SDN-C Cluster Deployment .

• Get New startODL.sh Script From Gerrit Topic SDNC-163 (download startODL_new.sh.zip script and copy into /dockerdata-nfs/cluster/script/startODL.sh)

• Get SDN-C Cluster Templates From Gerrit Topic SDNC-163 (skip step 1 and 2. The gerrit change 25467 has been already merged. Just update sdnc-statefulset.yaml and values.yaml)

Optional,  if you have a local nexus3 for you docker repo can use the following snippet to update the oom to pull from you local. This will speed up deployment time.  

As ubuntu user, 

Wait for the ONAP config pod to change state from ContainerCreating to Running  and finaly to Completed. It should have a "Completed" status:

3 SDNC pods will be created and will be each assigned to a separate Kuberentes worker nodes. 

2 DBhost pods will be created and will be each assigned to a separate Kuberentes worker nodes. 

Refer to Validate the SDN-C ODL cluster.

Access to RestConf UI is via https://<Kuberbetes-Master Node-IP>:30202/apidoc/explorer/index.html (admin user)

Run the following command to make sure installation is error free.

Get more detail about a single pod by using "describe" with the resource name. The resource name is shown with the get all command used above.

Get logs of containers inside each pod:

List of Presistent Volumes..

Each DB pod, has got a presistent volume cliam (pvc), lined to a pv.  PVC capacity must be less than or equal to PV. Their status must be "Bound". 

Scale up or down SDNC or DB pods

sudo ntpdate -s 10.247.5.11