SDN-C Clustering on Kubernetes

SDN-C Clustering on Kubernetes

 

 

About the SDN-C Clustering

The following diagram illustrates the desired SDN-C cluster deployment (as described in https://lf-onap.atlassian.net/browse/SDNC-163) from POD and SERVICE view, which will have:

  • 2 DB pods

  • 3 sdnc (ODL) pods

  • 1 admin (portal) pod

  • 1 dgbuilder pod

Gliffy Diagram

SDN-C DB Clustering details

See details from SDN-C DB (MySQL) Clustered Deployment.

Deployment Setup Steps

You can use Rancher or kubeadm to deploy your Kubernetes cluster:

 

Additional Information

SDN-C Startup Order

Gliffy Diagram

Troubleshoot SDN-C pods are not start up in order

Usually, this is caused by init-container malfuntion. You can troubleshoot it by the following steps:

  • Do a describe of the pod using command kubectl describe pod <pod-name>, to ensure the Init Container section is presented as your defined in your yaml template.

  • If it does not exist, it is possible that your Kubernetes version supports a different format for the init-containers:

    init-containers syntax supported by Kubernetes 1.5, 1.6 and 1.7 and not supported by Kubernetes 1.8 and greater

    initContainers syntax supported from Kubernetes 1.6 and greater

    init-containers syntax supported by Kubernetes 1.5, 1.6 and 1.7 and not supported by Kubernetes 1.8 and greater

    initContainers syntax supported from Kubernetes 1.6 and greater

    Init container is defined under

    spec.template.metadata.annotations."pod.beta.kubernetes.io/init-containers

    apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: sdnc ... spec: ... template: metadata: ... annotations: pod.beta.kubernetes.io/init-containers: '[ { "args": [ "--container-name", "sdnc-db-container" ], "command": [ "/root/ready.py" ], "env": [ { "name": "NAMESPACE", "valueFrom": { "fieldRef": { "apiVersion": "v1", "fieldPath": "metadata.namespace" } } } ], "image": "{{ .Values.image.readiness }}", "imagePullPolicy": "{{ .Values.pullPolicy }}", "name": "sdnc-readiness" } ]' spec: containers: ...

    Init container is defined under spec.template.spec.initContainers

    apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: sdnc ... spec: ... template: ... spec: initContainers: - command: - /root/ready.py - "--container-name" - "sdnc-db-container" env: - name: NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace image: "{{ .Values.image.readiness }}" imagePullPolicy: {{ .Values.pullPolicy }} name: sdnc-readiness containers: ...

Restart Dead Instance (start a new one in its place)

When a pod is dead, Kubernetes automatically starts a new one to replace the dead pod.

Examples:

  • Pod-level restart:

Description of Initial sdnc-0

Description of restarted sdnc-0

Description of Initial sdnc-0

Description of restarted sdnc-0

ubuntu@sdnc-k8s:~/oom/kubernetes/config$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-sdnc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:37:00 +0000
Labels: app=sdnc
controller-revision-hash=sdnc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.107.61
Created By: StatefulSet/sdnc
Controlled By: StatefulSet/sdnc

...

ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-sdnc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:54:10 +0000
Labels: app=sdnc
controller-revision-hash=sdnc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.185.227
Created By: StatefulSet/sdnc
Controlled By: StatefulSet/sdnc

...

  • Container-level restart:

ubuntu@sdnc-k8s-2:~$ kubectl describe pod sdnc-0 -n onap-sdnc Name: sdnc-0 Namespace: onap-sdnc Node: sdnc-k8s-2/10.147.90.6 Start Time: Fri, 24 Nov 2017 16:41:42 +0000 Labels: app=sdnc ... Containers: sdnc-controller-container: Container ID: docker://d3cd42925952d43fa5d6cca359c52d7033fa8bdf5e31881eea424664dde0f634 Image: nexus3.onap.org:10001/onap/sdnc-image:1.2-STAGING-latest Image ID: docker-pullable://nexus3.onap.org:10001/onap/sdnc-image@sha256:87171a78f8f3090a5f31b63bfead858ec80d220f3ddb4a08e5c98d02cc5edeca Ports: 8181/TCP, 8101/TCP Command: /opt/onap/sdnc/bin/startODL.sh State: Running Started: Fri, 24 Nov 2017 17:37:41 +0000 Last State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 24 Nov 2017 16:42:15 +0000 Finished: Fri, 24 Nov 2017 17:37:37 +0000 Ready: True Restart Count: 1 ...

Source Code

The source code is currently temporary shared through gerrit topic SDNC-163 until it is commited into gerrit.