...

Table of Contents

Note, this page is based on the SDNC-C Clustering on Kubernetes page as created by Beili Zhou.

...

1 DB pod* (2 or more DB pods to be supported with the introduction of MariaDB and Galera)
3 appc (ODL) pods
1 dgbuilder pod
1 cdt pod
1 ansible pod

Gliffy

name	Pod, service, deploy kind Copy
pagePin	79

APPC DB Clustering details

...

Deployment with Kubernetes Cluster Configured by Rancher or
Deploying Setting Up Kubernetes Cluster with kubeadmand Deploying for APPC

Additional Information

APPC Startup Order

Gliffy

name

...

APPC POD Startup Order
pagePin	2

Troubleshoot situation where APPC pods do not start up in order

...

Do a describe of the pod using command kubectl describe pod <pod-name>, to ensure the Init Container section is presented as your defined in your yaml template.

If it does not exist, it is possible that your Kubernetes version supports a different format for the init-containers:

init-containers syntax supported by Kubernetes 1.5, 1.6 and 1.7 and not supported by Kubernetes 1.8 and greater

initContainers syntax supported from Kubernetes 1.6 and greater

Init container is defined under

spec.template.metadata.annotations."pod.beta.kubernetes.io/init-containers

Code Block

language	xml
title	An example of init container by beta annotation
linenumbers	true
collapse	true

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: appc
  ...
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
          {
              "args": [
                  "--container-name",
                  "appc-db-container"
              ],
              "command": [
                  "/root/ready.py"
              ],
              "env": [
                  {
                      "name": "NAMESPACE",
                      "valueFrom": {
                          "fieldRef": {
                              "apiVersion": "v1",
                              "fieldPath": "metadata.namespace"
                          }
                      }
                  }
              ],
              "image": "{{ .Values.image.readiness }}",
              "imagePullPolicy": "{{ .Values.pullPolicy }}",
              "name": "appc-readiness"
          }
          ]'
    spec:
      containers:
      ...

Init container is defined under spec.template.spec.initContainers

Code Block

language	xml
title	An example of init container by spec.initContainers
linenumbers	true
collapse	true

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: sdnc
  ...
spec:
  ...
  template:
    ...
    spec:
      initContainers:
      - command:
        - /root/ready.py
        - "--container-name"
        - "appc-db-container"
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: "{{ .Values.image.readiness }}"
        imagePullPolicy: {{ .Values.pullPolicy }}
        name: appc-readiness
      containers:
      ...

...

Expand

title	Example of restarting dead instance by deleting pod

Expand

title	sdncappc-0 pod runs fine at the beginning

Image Removed

Expand

title	manually delete the sdnc-0 pod

Image Removed

Expand

title	once the sdnc-0 pod is fully terminated, a new sdnc-0 pod will be initialized

Image Removed

Expand

title	The new sdnc-0 pod will be brought to running status to replace the initial sdnc-0

Image Removed

Expand

title	The new and initial sdnc-0 pod has the same name (although with different internal IP)

Description of Initial appc-0

Description of restarted appc-0

ubuntu@sdnc-k8s:~/oom/kubernetes/config$ kubectl describe pod appc-0 -n onap-appc
Name: appc-0
Namespace: onap-appc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:37:00 +0000
Labels: app=sdnc
controller-revision-hash=sdnc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.107.61
Created By: StatefulSet/appc
Controlled By: StatefulSet/appc

...

ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-appc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:54:10 +0000
Labels: app=sdnc
controller-revision-hash=appc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.185.227
Created By: StatefulSet/appc
Controlled By: StatefulSet/appc

...

Container-level restart:

Expand

title	Example of restarting dead instance by stopping karaf inside SDNC pod

Expand

title	Pod sdnc-0 is in Running STATUS with 0 RESTARTS

Image Removed

Expand

title	Enter pod container and stop Karaf

Image Removed

Expand

title	Kubernetes discovers pod process stopped and restarts the container

Image Removed

Expand

title	sdnc-0 pod will be brought up with the restart container and running with 2 out of 2 container READY

Image Removed

Expand

title	Description of the pod

ubuntu@appc-k8s-appc2:~$ kubectl describe pod appc-0 -n onap-appc
Name: appc-0
Namespace: onap-appc
Node: k8s-appc2/10.147.90.6
Start Time: Fri, 24 Nov 2017 16:41:42 +0000
Labels: app=appc

...

Containers:
appc-controller-container:
Container ID: docker://d3cd42925952d43fa5d6cca359c52d7033fa8bdf5e31881eea424664dde0f634
Image: nexus3.onap.org:10001/onap/sdnc-image:1.2-STAGING-latest
Image ID: docker-pullable://nexus3.onap.org:10001/onap/sdnc-image@sha256:87171a78f8f3090a5f31b63bfead858ec80d220f3ddb4a08e5c98d02cc5edeca
Ports: 8181/TCP, 8101/TCP
Command:
/opt/onap/appc/bin/startODL.sh
State: Running
Started: Fri, 24 Nov 2017 17:37:41 +0000
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 24 Nov 2017 16:42:15 +0000
Finished: Fri, 24 Nov 2017 17:37:37 +0000
Ready: True
Restart Count: 1

...$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running   5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running   5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running   5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running   21         27d
kube-system   kube-proxy-25tm5                                1/1       Running   8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running   4          27d
kube-system   kube-proxy-jmv67                                1/1       Running   4          27d
kube-system   kube-proxy-l8fks                                1/1       Running   6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running   5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running   0          4d
kube-system   weave-net-bz7wr                                 2/2       Running   20         27d
kube-system   weave-net-c2pxd                                 2/2       Running   13         27d
kube-system   weave-net-jw29c                                 2/2       Running   20         27d
kube-system   weave-net-kxxpl                                 2/2       Running   13         27d
onap          dev-appc-0                                      2/2       Running   0          2h
onap          dev-appc-1                                      2/2       Running   0          1h
onap          dev-appc-2                                      2/2       Running   0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running   0          2h
onap          dev-appc-db-0                                   2/2       Running   0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running   0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running   0          2h

Expand

title	manually delete the appc-0 pod

$ kubectl delete pod dev-appc-0 -n onap
pod "dev-appc-0" deleted

$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS        RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running       5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running       5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running       5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running       21         27d
kube-system   kube-proxy-25tm5                                1/1       Running       8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running       4          27d
kube-system   kube-proxy-jmv67                                1/1       Running       4          27d
kube-system   kube-proxy-l8fks                                1/1       Running       6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running       5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running       0          4d
kube-system   weave-net-bz7wr                                 2/2       Running       20         27d
kube-system   weave-net-c2pxd                                 2/2       Running       13         27d
kube-system   weave-net-jw29c                                 2/2       Running       20         27d
kube-system   weave-net-kxxpl                                 2/2       Running       13         27d
onap          dev-appc-0                                      2/2       Terminating   0          2h
onap          dev-appc-1                                      2/2       Running       0          1h
onap          dev-appc-2                                      2/2       Running       0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running       0          2h
onap          dev-appc-db-0                                   2/2       Running       0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running       0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running       0          2h

Expand

title	The new appc-0 pod will be brought to running status to replace the initial appc-0

$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running   5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running   5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running   5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running   21         27d
kube-system   kube-proxy-25tm5                                1/1       Running   8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running   4          27d
kube-system   kube-proxy-jmv67                                1/1       Running   4          27d
kube-system   kube-proxy-l8fks                                1/1       Running   6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running   5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running   0          4d
kube-system   weave-net-bz7wr                                 2/2       Running   20         27d
kube-system   weave-net-c2pxd                                 2/2       Running   13         27d
kube-system   weave-net-jw29c                                 2/2       Running   20         27d
kube-system   weave-net-kxxpl                                 2/2       Running   13         27d
onap          dev-appc-0                                      2/2       Running   0          30s
onap          dev-appc-1                                      2/2       Running   0          1h
onap          dev-appc-2                                      2/2       Running   0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running   0          2h
onap          dev-appc-db-0                                   2/2       Running   0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running   0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running   0          2h

Versions Compared

Old Version 9

New Version Current

Key

APPC DB Clustering details

Additional Information

APPC Startup Order

Troubleshoot situation where APPC pods do not start up in order

Page Comparison

Versions Compared

Old Version 9

New Version Current

Key

APPC DB Clustering details

Additional Information

APPC Startup Order

Troubleshoot situation where APPC pods do not start up in order