Versions Compared

Old Version 1

changes.mady.by.user Aaron Hay

Saved on

compared with

New Version Current

changes.mady.by.user Takamune Cho

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

...

...

...


Table of Contents


Note, this page is based on the SDNC-C Clustering on Kubernetes page as created by Beili Zhou.

About the APPC-C Clustering

The following diagram illustrates the desired following pod layout is desired for the APPC cluster deployment (as described in 

Jira Legacy
serverSystem Jira
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyAPPC-476
) from POD and SERVICE view, which will have:

  • 1 DB pod*  (2 or more DB pods to be supported with the introduction of MariaDB and Galera)
  • 3 appc (ODL) pods
  • 1 admin (portal) dgbuilder pod
  • 1 cdt pod
  • 1 dgbuilder ansible pod


Gliffy
namePod, service, deploy kind Copy
pagePin9

APPC DB Clustering details

...


Additional Information

APPC Startup Order

Gliffy
sdnc pods start up order
nameAPPC POD Startup Order
pagePin2


Troubleshoot situation where APPC pods do not start up in order

...

  • Do a describe of the pod using command kubectl describe pod <pod-name>, to ensure the Init Container section is presented as your defined in your yaml template.

  • If it does not exist, it is possible that your Kubernetes version supports a different format for the init-containers:

    init-containers syntax supported by Kubernetes 1.5, 1.6 and 1.7 and not supported by Kubernetes 1.8 and greaterinitContainers syntax supported from Kubernetes 1.6 and greater

    Init container is defined under

    spec.template.metadata.annotations."pod.beta.kubernetes.io/init-containers

    Code Block
    languagexml
    titleAn example of init container by beta annotation
    linenumberstrue
    collapsetrue
    apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: appc
  ...
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
          {
              "args": [
                  "--container-name",
                  "appc-db-container"
              ],
              "command": [
                  "/root/ready.py"
              ],
              "env": [
                  {
                      "name": "NAMESPACE",
                      "valueFrom": {
                          "fieldRef": {
                              "apiVersion": "v1",
                              "fieldPath": "metadata.namespace"
                          }
                      }
                  }
              ],
              "image": "{{ .Values.image.readiness }}",
              "imagePullPolicy": "{{ .Values.pullPolicy }}",
              "name": "appc-readiness"
          }
          ]'
    spec:
      containers:
      ...


    Init container is defined under spec.template.spec.initContainers

    Code Block
    languagexml
    titleAn example of init container by spec.initContainers
    linenumberstrue
    collapsetrue
    apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: sdnc
  ...
spec:
  ...
  template:
    ...
    spec:
      initContainers:
      - command:
        - /root/ready.py
        - "--container-name"
        - "appc-db-container"
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: "{{ .Values.image.readiness }}"
        imagePullPolicy: {{ .Values.pullPolicy }}
        name: appc-readiness
      containers:
      ...



Restart Dead Instance (start a new one in its place)

...

Expand
titleExample of restarting dead instance by deleting pod


Expand
titlesdncappc-0 pod runs fine at the beginning

Image Removed

Expand
titlemanually delete the sdnc-0 pod

Image Removed

Expand
titleonce the sdnc-0 pod is fully terminated, a new sdnc-0 pod will be initialized

Image Removed

Expand
titleThe new sdnc-0 pod will be brought to running status to replace the initial sdnc-0

Image Removed

Expand
titleThe new and initial sdnc-0 pod has the same name (although with different internal IP)
Description of Initial appc-0Description of restarted appc-0

ubuntu@sdnc-k8s:~/oom/kubernetes/config$ kubectl describe pod appc-0 -n onap-appc
Name: appc-0
Namespace: onap-appc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:37:00 +0000
Labels: app=sdnc
controller-revision-hash=sdnc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.107.61
Created By: StatefulSet/appc
Controlled By: StatefulSet/appc

...

ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-appc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:54:10 +0000
Labels: app=sdnc
controller-revision-hash=appc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.185.227
Created By: StatefulSet/appc
Controlled By: StatefulSet/appc

...

  • Container-level restart:
Expand
titleExample of restarting dead instance by stopping karaf inside SDNC pod
Expand
titlePod sdnc-0 is in Running STATUS with 0 RESTARTS

Image Removed

Expand
titleEnter pod container and stop Karaf

Image Removed

Expand
titleKubernetes discovers pod process stopped and restarts the container

Image Removed

Expand
titlesdnc-0 pod will be brought up with the restart container and running with 2 out of 2 container READY

Image Removed

Expand
titleDescription of the pod

ubuntu@appc-k8s-appc2:~$ kubectl describe pod appc-0 -n onap-appc
Name: appc-0
Namespace: onap-appc
Node: k8s-appc2/10.147.90.6
Start Time: Fri, 24 Nov 2017 16:41:42 +0000
Labels: app=appc

...

Containers:
 appc-controller-container:
Container ID: docker://d3cd42925952d43fa5d6cca359c52d7033fa8bdf5e31881eea424664dde0f634
Image: nexus3.onap.org:10001/onap/sdnc-image:1.2-STAGING-latest
Image ID: docker-pullable://nexus3.onap.org:10001/onap/sdnc-image@sha256:87171a78f8f3090a5f31b63bfead858ec80d220f3ddb4a08e5c98d02cc5edeca
Ports: 8181/TCP, 8101/TCP
Command:
/opt/onap/appc/bin/startODL.sh
State: Running
Started: Fri, 24 Nov 2017 17:37:41 +0000
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 24 Nov 2017 16:42:15 +0000
Finished: Fri, 24 Nov 2017 17:37:37 +0000
Ready: True
Restart Count: 1

...$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running   5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running   5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running   5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running   21         27d
kube-system   kube-proxy-25tm5                                1/1       Running   8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running   4          27d
kube-system   kube-proxy-jmv67                                1/1       Running   4          27d
kube-system   kube-proxy-l8fks                                1/1       Running   6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running   5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running   0          4d
kube-system   weave-net-bz7wr                                 2/2       Running   20         27d
kube-system   weave-net-c2pxd                                 2/2       Running   13         27d
kube-system   weave-net-jw29c                                 2/2       Running   20         27d
kube-system   weave-net-kxxpl                                 2/2       Running   13         27d
onap          dev-appc-0                                      2/2       Running   0          2h
onap          dev-appc-1                                      2/2       Running   0          1h
onap          dev-appc-2                                      2/2       Running   0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running   0          2h
onap          dev-appc-db-0                                   2/2       Running   0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running   0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running   0          2h


Expand
titlemanually delete the appc-0 pod

$ kubectl delete pod dev-appc-0 -n onap
pod "dev-appc-0" deleted

$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS        RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running       5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running       5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running       5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running       21         27d
kube-system   kube-proxy-25tm5                                1/1       Running       8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running       4          27d
kube-system   kube-proxy-jmv67                                1/1       Running       4          27d
kube-system   kube-proxy-l8fks                                1/1       Running       6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running       5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running       0          4d
kube-system   weave-net-bz7wr                                 2/2       Running       20         27d
kube-system   weave-net-c2pxd                                 2/2       Running       13         27d
kube-system   weave-net-jw29c                                 2/2       Running       20         27d
kube-system   weave-net-kxxpl                                 2/2       Running       13         27d
onap          dev-appc-0                                      2/2       Terminating   0          2h
onap          dev-appc-1                                      2/2       Running       0          1h
onap          dev-appc-2                                      2/2       Running       0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running       0          2h
onap          dev-appc-db-0                                   2/2       Running       0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running       0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running       0          2h


Expand
titleThe new appc-0 pod will be brought to running status to replace the initial appc-0

$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running   5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running   5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running   5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running   21         27d
kube-system   kube-proxy-25tm5                                1/1       Running   8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running   4          27d
kube-system   kube-proxy-jmv67                                1/1       Running   4          27d
kube-system   kube-proxy-l8fks                                1/1       Running   6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running   5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running   0          4d
kube-system   weave-net-bz7wr                                 2/2       Running   20         27d
kube-system   weave-net-c2pxd                                 2/2       Running   13         27d
kube-system   weave-net-jw29c                                 2/2       Running   20         27d
kube-system   weave-net-kxxpl                                 2/2       Running   13         27d
onap          dev-appc-0                                      2/2       Running   0          30s
onap          dev-appc-1                                      2/2       Running   0          1h
onap          dev-appc-2                                      2/2       Running   0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running   0          2h
onap          dev-appc-db-0                                   2/2       Running   0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running   0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running   0          2h