Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »




Note, this page is based on the SDNC-C Clustering on Kubernetes page as created by Beili Zhou.

About the APPC-C Clustering

The following pod layout is desired for the APPC cluster deployment (as described in  APPC-476 - Getting issue details... STATUS ) from POD and SERVICE view, which will have:

  • 1 DB pod (2 or more DB pods to supported with the introduction of MariaDB and Galera)
  • 3 appc (ODL) pods
  • 1 dgbuilder pod
  • 1 cdt pod


APPC DB Clustering details

See details from APPC DB (MySQL) Clustered Deployment.

Deployment Setup Steps

You can use Rancher or kubeadm to deploy your Kubernetes cluster:


Additional Information

APPC Startup Order


Troubleshoot situation where APPC pods do not start up in order

Usually, this is caused by init-container malfuntion. You can troubleshoot it by the following steps:

  • Do a describe of the pod using command kubectl describe pod <pod-name>, to ensure the Init Container section is presented as your defined in your yaml template.
  • If it does not exist, it is possible that your Kubernetes version supports a different format for the init-containers:

    init-containers syntax supported by Kubernetes 1.5, 1.6 and 1.7 and not supported by Kubernetes 1.8 and greaterinitContainers syntax supported from Kubernetes 1.6 and greater

    Init container is defined under

    spec.template.metadata.annotations."pod.beta.kubernetes.io/init-containers

    An example of init container by beta annotation
    apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: appc
      ...
    spec:
      ...
      template:
        metadata:
          ...
          annotations:
            pod.beta.kubernetes.io/init-containers: '[
              {
                  "args": [
                      "--container-name",
                      "appc-db-container"
                  ],
                  "command": [
                      "/root/ready.py"
                  ],
                  "env": [
                      {
                          "name": "NAMESPACE",
                          "valueFrom": {
                              "fieldRef": {
                                  "apiVersion": "v1",
                                  "fieldPath": "metadata.namespace"
                              }
                          }
                      }
                  ],
                  "image": "{{ .Values.image.readiness }}",
                  "imagePullPolicy": "{{ .Values.pullPolicy }}",
                  "name": "appc-readiness"
              }
              ]'
        spec:
          containers:
          ...

    Init container is defined under spec.template.spec.initContainers

    An example of init container by spec.initContainers
    apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: sdnc
      ...
    spec:
      ...
      template:
        ...
        spec:
          initContainers:
          - command:
            - /root/ready.py
            - "--container-name"
            - "appc-db-container"
            env:
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            image: "{{ .Values.image.readiness }}"
            imagePullPolicy: {{ .Values.pullPolicy }}
            name: appc-readiness
          containers:
          ...

Restart Dead Instance (start a new one in its place)

When a pod is dead, Kubernetes automatically starts a new one to replace the dead pod.

Examples:

  • Pod-level restart:
 Example of restarting dead instance by deleting pod
 sdnc-0 pod runs fine at the beginning

 manually delete the sdnc-0 pod

 once the sdnc-0 pod is fully terminated, a new sdnc-0 pod will be initialized

 The new sdnc-0 pod will be brought to running status to replace the initial sdnc-0

 The new and initial sdnc-0 pod has the same name (although with different internal IP)
Description of Initial appc-0Description of restarted appc-0

ubuntu@sdnc-k8s:~/oom/kubernetes/config$ kubectl describe pod appc-0 -n onap-appc
Name: appc-0
Namespace: onap-appc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:37:00 +0000
Labels: app=sdnc
controller-revision-hash=sdnc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:37:08Z","finishedAt":"2017-11-23T22:37:53Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.107.61
Created By: StatefulSet/appc
Controlled By: StatefulSet/appc

...

ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl describe pod sdnc-0 -n onap-sdnc
Name: sdnc-0
Namespace: onap-appc
Node: sdnc-k8s-2/10.147.90.6
Start Time: Thu, 23 Nov 2017 22:54:10 +0000
Labels: app=sdnc
controller-revision-hash=appc-2443484326
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"onap-sdnc","name":"sdnc","uid":"a576a1a3-d09f-11e7-957f-0269cb13eff1","apiVersio...
pod.alpha.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z...
pod.alpha.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NA...
pod.beta.kubernetes.io/init-container-statuses=[{"name":"sdnc-readiness","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2017-11-23T22:54:15Z","finishedAt":"2017-11-23T22:54:15Z"...
pod.beta.kubernetes.io/init-containers=[{"name":"sdnc-readiness","image":"oomk8s/readiness-check:1.0.0","command":["/root/ready.py"],"args":["--container-name","sdnc-db-container"],"env":[{"name":"NAM...
Status: Running
IP: 10.42.185.227
Created By: StatefulSet/appc
Controlled By: StatefulSet/appc

...

  • Container-level restart:
 Example of restarting dead instance by stopping karaf inside SDNC pod
 Pod sdnc-0 is in Running STATUS with 0 RESTARTS

 Enter pod container and stop Karaf

 Kubernetes discovers pod process stopped and restarts the container

 sdnc-0 pod will be brought up with the restart container and running with 2 out of 2 container READY

 Description of the pod

ubuntu@appc-k8s-appc2:~$ kubectl describe pod appc-0 -n onap-appc
Name: appc-0
Namespace: onap-appc
Node: k8s-appc2/10.147.90.6
Start Time: Fri, 24 Nov 2017 16:41:42 +0000
Labels: app=appc

...

Containers:
 appc-controller-container:
Container ID: docker://d3cd42925952d43fa5d6cca359c52d7033fa8bdf5e31881eea424664dde0f634
Image: nexus3.onap.org:10001/onap/sdnc-image:1.2-STAGING-latest
Image ID: docker-pullable://nexus3.onap.org:10001/onap/sdnc-image@sha256:87171a78f8f3090a5f31b63bfead858ec80d220f3ddb4a08e5c98d02cc5edeca
Ports: 8181/TCP, 8101/TCP
Command:
/opt/onap/appc/bin/startODL.sh
State: Running
Started: Fri, 24 Nov 2017 17:37:41 +0000
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 24 Nov 2017 16:42:15 +0000
Finished: Fri, 24 Nov 2017 17:37:37 +0000
Ready: True
Restart Count: 1

...



  • No labels