Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Table of Contents

...

  • Do a describe of the pod using command kubectl describe pod <pod-name>, to ensure the Init Container section is presented as your defined in your yaml template.
  • If it does not exist, it is possible that your Kubernetes version supports a different format for the init-containers:

    init-containers syntax supported by Kubernetes 1.5, 1.6 and 1.7 and not supported by Kubernetes 1.8 and greaterinitContainers syntax supported from Kubernetes 1.6 and greater

    Init container is defined under

    spec.template.metadata.annotations."pod.beta.kubernetes.io/init-containers

    Code Block
    languagexml
    titleAn example of init container by beta annotation
    linenumberstrue
    collapsetrue
    apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: appc
      ...
    spec:
      ...
      template:
        metadata:
          ...
          annotations:
            pod.beta.kubernetes.io/init-containers: '[
              {
                  "args": [
                      "--container-name",
                      "appc-db-container"
                  ],
                  "command": [
                      "/root/ready.py"
                  ],
                  "env": [
                      {
                          "name": "NAMESPACE",
                          "valueFrom": {
                              "fieldRef": {
                                  "apiVersion": "v1",
                                  "fieldPath": "metadata.namespace"
                              }
                          }
                      }
                  ],
                  "image": "{{ .Values.image.readiness }}",
                  "imagePullPolicy": "{{ .Values.pullPolicy }}",
                  "name": "appc-readiness"
              }
              ]'
        spec:
          containers:
          ...

    Init container is defined under spec.template.spec.initContainers

    Code Block
    languagexml
    titleAn example of init container by spec.initContainers
    linenumberstrue
    collapsetrue
    apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: sdnc
      ...
    spec:
      ...
      template:
        ...
        spec:
          initContainers:
          - command:
            - /root/ready.py
            - "--container-name"
            - "appc-db-container"
            env:
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            image: "{{ .Values.image.readiness }}"
            imagePullPolicy: {{ .Values.pullPolicy }}
            name: appc-readiness
          containers:
          ...

...

Expand
titleExample of restarting dead instance by deleting pod
Expand
titlesdncappc-0 pod runs fine at the beginning

Image Removed

Expand
titlemanually delete the sdnc-0 pod

Image Removed

Expand
titleonce the sdnc-0 pod is fully terminated, a new sdnc-0 pod will be initialized

Image Removed

Expand
titleThe new sdnc-0 pod will be brought to running status to replace the initial sdnc-0

Image Removed

Expand
titleThe new and initial sdnc-0 pod has the same name (although with different internal IP)
Description of Initial appc-0Description of restarted appc-0

ubuntu@sdnc-k8s:~/oom/kubernetes/config$ kubectl describe pod appc-0 -n onap-appc
Name: appc-0
Namespace: onap-appc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:37:00 +0000
Labels: app=sdnc
controller-revision-hash=sdnc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.107.61
Created By: StatefulSet/appc
Controlled By: StatefulSet/appc

...

ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-appc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:54:10 +0000
Labels: app=sdnc
controller-revision-hash=appc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.185.227
Created By: StatefulSet/appc
Controlled By: StatefulSet/appc

...

  • Container-level restart:
Expand
titleExample of restarting dead instance by stopping karaf inside SDNC pod
Expand
titlePod sdnc-0 is in Running STATUS with 0 RESTARTS

Image Removed

Expand
titleEnter pod container and stop Karaf

Image Removed

Expand
titleKubernetes discovers pod process stopped and restarts the container

Image Removed

Expand
titlesdnc-0 pod will be brought up with the restart container and running with 2 out of 2 container READY

Image Removed

Expand
titleDescription of the pod

ubuntu@appc-k8s-appc2:~$ kubectl describe pod appc-0 -n onap-appc
Name: appc-0
Namespace: onap-appc
Node: k8s-appc2/10.147.90.6
Start Time: Fri, 24 Nov 2017 16:41:42 +0000
Labels: app=appc

...

Containers:
 appc-controller-container:
Container ID: docker://d3cd42925952d43fa5d6cca359c52d7033fa8bdf5e31881eea424664dde0f634
Image: nexus3.onap.org:10001/onap/sdnc-image:1.2-STAGING-latest
Image ID: docker-pullable://nexus3.onap.org:10001/onap/sdnc-image@sha256:87171a78f8f3090a5f31b63bfead858ec80d220f3ddb4a08e5c98d02cc5edeca
Ports: 8181/TCP, 8101/TCP
Command:
/opt/onap/appc/bin/startODL.sh
State: Running
Started: Fri, 24 Nov 2017 17:37:41 +0000
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 24 Nov 2017 16:42:15 +0000
Finished: Fri, 24 Nov 2017 17:37:37 +0000
Ready: True
Restart Count: 1

...$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running   5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running   5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running   5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running   21         27d
kube-system   kube-proxy-25tm5                                1/1       Running   8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running   4          27d
kube-system   kube-proxy-jmv67                                1/1       Running   4          27d
kube-system   kube-proxy-l8fks                                1/1       Running   6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running   5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running   0          4d
kube-system   weave-net-bz7wr                                 2/2       Running   20         27d
kube-system   weave-net-c2pxd                                 2/2       Running   13         27d
kube-system   weave-net-jw29c                                 2/2       Running   20         27d
kube-system   weave-net-kxxpl                                 2/2       Running   13         27d
onap          dev-appc-0                                      2/2       Running   0          2h
onap          dev-appc-1                                      2/2       Running   0          1h
onap          dev-appc-2                                      2/2       Running   0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running   0          2h
onap          dev-appc-db-0                                   2/2       Running   0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running   0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running   0          2h

Expand
titlemanually delete the appc-0 pod

$ kubectl delete pod dev-appc-0 -n onap
pod "dev-appc-0" deleted

$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS        RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running       5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running       5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running       5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running       21         27d
kube-system   kube-proxy-25tm5                                1/1       Running       8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running       4          27d
kube-system   kube-proxy-jmv67                                1/1       Running       4          27d
kube-system   kube-proxy-l8fks                                1/1       Running       6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running       5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running       0          4d
kube-system   weave-net-bz7wr                                 2/2       Running       20         27d
kube-system   weave-net-c2pxd                                 2/2       Running       13         27d
kube-system   weave-net-jw29c                                 2/2       Running       20         27d
kube-system   weave-net-kxxpl                                 2/2       Running       13         27d
onap          dev-appc-0                                      2/2       Terminating   0          2h
onap          dev-appc-1                                      2/2       Running       0          1h
onap          dev-appc-2                                      2/2       Running       0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running       0          2h
onap          dev-appc-db-0                                   2/2       Running       0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running       0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running       0          2h

Expand
titleThe new appc-0 pod will be brought to running status to replace the initial appc-0

$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running   5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running   5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running   5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running   21         27d
kube-system   kube-proxy-25tm5                                1/1       Running   8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running   4          27d
kube-system   kube-proxy-jmv67                                1/1       Running   4          27d
kube-system   kube-proxy-l8fks                                1/1       Running   6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running   5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running   0          4d
kube-system   weave-net-bz7wr                                 2/2       Running   20         27d
kube-system   weave-net-c2pxd                                 2/2       Running   13         27d
kube-system   weave-net-jw29c                                 2/2       Running   20         27d
kube-system   weave-net-kxxpl                                 2/2       Running   13         27d
onap          dev-appc-0                                      2/2       Running   0          30s
onap          dev-appc-1                                      2/2       Running   0          1h
onap          dev-appc-2                                      2/2       Running   0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running   0          2h
onap          dev-appc-db-0                                   2/2       Running   0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running   0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running   0          2h