About the SDN-C Clustering

The following diagram illustrates the desired SDN-C cluster deployment (as described in SDNC-163 - Getting issue details... STATUS ) from POD and SERVICE view, which will have:

2 DB pods
3 sdnc (ODL) pods
1 admin (portal) pod
1 dgbuilder pod

SDN-C DB Clustering details

See details from SDN-C DB (MySQL) Clustered Deployment.

Deployment Setup Steps

You can use Rancher or kubeadm to deploy your Kubernetes cluster:

Additional Information

SDN-C Startup Order

Troubleshoot SDN-C pods are not start up in order

Usually, this is caused by init-container malfuntion. You can troubleshoot it by the following steps:

Do a describe of the pod using command kubectl describe pod <pod-name>, to ensure the Init Container section is presented as your defined in your yaml template.

If it does not exist, it is possible that your Kubernetes version supports a different format for the init-containers:

init-containers syntax supported by Kubernetes 1.5, 1.6 and 1.7 and not supported by Kubernetes 1.8 and greater

initContainers syntax supported from Kubernetes 1.6 and greater

Init container is defined under

spec.template.metadata.annotations."pod.beta.kubernetes.io/init-containers

An example of init container by beta annotation

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: sdnc
  ...
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
          {
              "args": [
                  "--container-name",
                  "sdnc-db-container"
              ],
              "command": [
                  "/root/ready.py"
              ],
              "env": [
                  {
                      "name": "NAMESPACE",
                      "valueFrom": {
                          "fieldRef": {
                              "apiVersion": "v1",
                              "fieldPath": "metadata.namespace"
                          }
                      }
                  }
              ],
              "image": "{{ .Values.image.readiness }}",
              "imagePullPolicy": "{{ .Values.pullPolicy }}",
              "name": "sdnc-readiness"
          }
          ]'
    spec:
      containers:
      ...

Init container is defined under spec.template.spec.initContainers

An example of init container by spec.initContainers

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: sdnc
  ...
spec:
  ...
  template:
    ...
    spec:
      initContainers:
      - command:
        - /root/ready.py
        - "--container-name"
        - "sdnc-db-container"
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: "{{ .Values.image.readiness }}"
        imagePullPolicy: {{ .Values.pullPolicy }}
        name: sdnc-readiness
      containers:
      ...

Restart Dead Instance (start a new one in its place)

When a pod is dead, Kubernetes automatically starts a new one to replace the dead pod.

Examples:

Pod-level restart:

Example of restarting dead instance by deleting pod

sdnc-0 pod runs fine at the beginning

manually delete the sdnc-0 pod

once the sdnc-0 pod is fully terminated, a new sdnc-0 pod will be initialized

The new sdnc-0 pod will be brought to running status to replace the initial sdnc-0

The new and initial sdnc-0 pod has the same name (although with different internal IP)

Description of Initial sdnc-0

Description of restarted sdnc-0

ubuntu@sdnc-k8s:~/oom/kubernetes/config$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-sdnc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:37:00 +0000
Labels: app=sdnc
controller-revision-hash=sdnc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.107.61
Created By: StatefulSet/sdnc
Controlled By: StatefulSet/sdnc

...

ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-sdnc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:54:10 +0000
Labels: app=sdnc
controller-revision-hash=sdnc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.185.227
Created By: StatefulSet/sdnc
Controlled By: StatefulSet/sdnc

...

Container-level restart:

Example of restarting dead instance by stopping karaf inside SDNC pod

Pod sdnc-0 is in Running STATUS with 0 RESTARTS

Enter pod container and stop Karaf

Kubernetes discovers pod process stopped and restarts the container

sdnc-0 pod will be brought up with the restart container and running with 2 out of 2 container READY

Description of the pod

ubuntu@sdnc-k8s-2:~$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-sdnc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Fri, 24 Nov 2017 16:41:42 +0000
Labels: app=sdnc

...

Containers:
sdnc-controller-container:
Container ID: docker://d3cd42925952d43fa5d6cca359c52d7033fa8bdf5e31881eea424664dde0f634
Image: nexus3.onap.org:10001/onap/sdnc-image:1.2-STAGING-latest
Image ID: docker-pullable://nexus3.onap.org:10001/onap/sdnc-image@sha256:87171a78f8f3090a5f31b63bfead858ec80d220f3ddb4a08e5c98d02cc5edeca
Ports: 8181/TCP, 8101/TCP
Command:
/opt/onap/sdnc/bin/startODL.sh
State: Running
Started: Fri, 24 Nov 2017 17:37:41 +0000
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 24 Nov 2017 16:42:15 +0000
Finished: Fri, 24 Nov 2017 17:37:37 +0000
Ready: True
Restart Count: 1

...

Source Code

The source code is currently temporary shared through gerrit topic SDNC-163 until it is commited into gerrit.

ONAP Wiki

SDN-C Clustering on Kubernetes