1 About the SDN-C Clustering
- 1.1 SDN-C DB Clustering details
2 Deployment Setup Steps
3 Additional Information
- 3.1 SDN-C Startup Order
  - 3.1.1 Troubleshoot SDN-C pods are not start up in order
    - 3.1.1.1 An example of init container by beta annotation
    - 3.1.1.2 An example of init container by spec.initContainers
- 3.2 Restart Dead Instance (start a new one in its place)
4 Source Code

About the SDN-C Clustering

The following diagram illustrates the desired SDN-C cluster deployment (as described in https://lf-onap.atlassian.net/browse/SDNC-163) from POD and SERVICE view, which will have:

2 DB pods
3 sdnc (ODL) pods
1 admin (portal) pod
1 dgbuilder pod

SDN-C DB Clustering details

See details from SDN-C DB (MySQL) Clustered Deployment.

Deployment Setup Steps

You can use Rancher or kubeadm to deploy your Kubernetes cluster:

Additional Information

SDN-C Startup Order

Troubleshoot SDN-C pods are not start up in order

Usually, this is caused by init-container malfuntion. You can troubleshoot it by the following steps:

Do a describe of the pod using command kubectl describe pod <pod-name>, to ensure the Init Container section is presented as your defined in your yaml template.
If it does not exist, it is possible that your Kubernetes version supports a different format for the init-containers:

init-containers syntax supported by Kubernetes 1.5, 1.6 and 1.7 and not supported by Kubernetes 1.8 and greater	initContainers syntax supported from Kubernetes 1.6 and greater

init-containers syntax supported by Kubernetes 1.5, 1.6 and 1.7 and not supported by Kubernetes 1.8 and greater

initContainers syntax supported from Kubernetes 1.6 and greater

Init container is defined under

spec.template.metadata.annotations."pod.beta.kubernetes.io/init-containers

An example of init container by beta annotation

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: sdnc
  ...
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
          {
              "args": [
                  "--container-name",
                  "sdnc-db-container"
              ],
              "command": [
                  "/root/ready.py"
              ],
              "env": [
                  {
                      "name": "NAMESPACE",
                      "valueFrom": {
                          "fieldRef": {
                              "apiVersion": "v1",
                              "fieldPath": "metadata.namespace"
                          }
                      }
                  }
              ],
              "image": "{{ .Values.image.readiness }}",
              "imagePullPolicy": "{{ .Values.pullPolicy }}",
              "name": "sdnc-readiness"
          }
          ]'
    spec:
      containers:
      ...

Init container is defined under spec.template.spec.initContainers

An example of init container by spec.initContainers

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: sdnc
  ...
spec:
  ...
  template:
    ...
    spec:
      initContainers:
      - command:
        - /root/ready.py
        - "--container-name"
        - "sdnc-db-container"
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: "{{ .Values.image.readiness }}"
        imagePullPolicy: {{ .Values.pullPolicy }}
        name: sdnc-readiness
      containers:
      ...

Restart Dead Instance (start a new one in its place)

When a pod is dead, Kubernetes automatically starts a new one to replace the dead pod.

Examples:

Pod-level restart:

Container-level restart:

ubuntu@sdnc-k8s-2:~$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-sdnc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Fri, 24 Nov 2017 16:41:42 +0000
Labels: app=sdnc

...

Containers:
sdnc-controller-container:
Container ID: docker://d3cd42925952d43fa5d6cca359c52d7033fa8bdf5e31881eea424664dde0f634
Image: nexus3.onap.org:10001/onap/sdnc-image:1.2-STAGING-latest
Image ID: docker-pullable://nexus3.onap.org:10001/onap/sdnc-image@sha256:87171a78f8f3090a5f31b63bfead858ec80d220f3ddb4a08e5c98d02cc5edeca
Ports: 8181/TCP, 8101/TCP
Command:
/opt/onap/sdnc/bin/startODL.sh
State: Running
Started: Fri, 24 Nov 2017 17:37:41 +0000
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 24 Nov 2017 16:42:15 +0000
Finished: Fri, 24 Nov 2017 17:37:37 +0000
Ready: True
Restart Count: 1

...

Source Code

The source code is currently temporary shared through gerrit topic SDNC-163 until it is commited into gerrit.

SDN-C Clustering on Kubernetes