APPC Clustering on Kubernetes

Owned by Aaron Hay
Last updated: Feb 18, 2019 by Takamune Cho
5 min read



Note, this page is based on the SDNC-C Clustering on Kubernetes page as created by Beili Zhou.

About the APPC-C Clustering

The following pod layout is desired for the APPC cluster deployment (as described in  APPC-476 - Getting issue details... STATUS ) from POD and SERVICE view, which will have:

  • 1 DB pod*  (2 or more DB pods to be supported with the introduction of MariaDB and Galera)
  • 3 appc (ODL) pods
  • 1 dgbuilder pod
  • 1 cdt pod
  • 1 ansible pod


APPC DB Clustering details

See details from APPC DB (MySQL) Clustered Deployment.

Deployment Setup Steps

You can use Rancher or kubeadm to deploy your Kubernetes cluster:


Additional Information

APPC Startup Order


Troubleshoot situation where APPC pods do not start up in order

Usually, this is caused by init-container malfuntion. You can troubleshoot it by the following steps:

  • Do a describe of the pod using command kubectl describe pod <pod-name>, to ensure the Init Container section is presented as your defined in your yaml template.

  • If it does not exist, it is possible that your Kubernetes version supports a different format for the init-containers:

    init-containers syntax supported by Kubernetes 1.5, 1.6 and 1.7 and not supported by Kubernetes 1.8 and greaterinitContainers syntax supported from Kubernetes 1.6 and greater

    Init container is defined under

    spec.template.metadata.annotations."pod.beta.kubernetes.io/init-containers

    An example of init container by beta annotation 
    apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: appc
  ...
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
          {
              "args": [
                  "--container-name",
                  "appc-db-container"
              ],
              "command": [
                  "/root/ready.py"
              ],
              "env": [
                  {
                      "name": "NAMESPACE",
                      "valueFrom": {
                          "fieldRef": {
                              "apiVersion": "v1",
                              "fieldPath": "metadata.namespace"
                          }
                      }
                  }
              ],
              "image": "{{ .Values.image.readiness }}",
              "imagePullPolicy": "{{ .Values.pullPolicy }}",
              "name": "appc-readiness"
          }
          ]'
    spec:
      containers:
      ...

    Init container is defined under spec.template.spec.initContainers

    An example of init container by spec.initContainers 
    apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: sdnc
  ...
spec:
  ...
  template:
    ...
    spec:
      initContainers:
      - command:
        - /root/ready.py
        - "--container-name"
        - "appc-db-container"
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: "{{ .Values.image.readiness }}"
        imagePullPolicy: {{ .Values.pullPolicy }}
        name: appc-readiness
      containers:
      ...

Restart Dead Instance (start a new one in its place)

When a pod is dead, Kubernetes automatically starts a new one to replace the dead pod.

Examples:

  • Pod-level restart:
 Example of restarting dead instance by deleting pod
 appc-0 pod runs fine at the beginning

$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running   5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running   5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running   5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running   21         27d
kube-system   kube-proxy-25tm5                                1/1       Running   8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running   4          27d
kube-system   kube-proxy-jmv67                                1/1       Running   4          27d
kube-system   kube-proxy-l8fks                                1/1       Running   6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running   5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running   0          4d
kube-system   weave-net-bz7wr                                 2/2       Running   20         27d
kube-system   weave-net-c2pxd                                 2/2       Running   13         27d
kube-system   weave-net-jw29c                                 2/2       Running   20         27d
kube-system   weave-net-kxxpl                                 2/2       Running   13         27d
onap          dev-appc-0                                      2/2       Running   0          2h
onap          dev-appc-1                                      2/2       Running   0          1h
onap          dev-appc-2                                      2/2       Running   0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running   0          2h
onap          dev-appc-db-0                                   2/2       Running   0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running   0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running   0          2h

 manually delete the appc-0 pod

$ kubectl delete pod dev-appc-0 -n onap
pod "dev-appc-0" deleted

$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS        RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running       5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running       5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running       5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running       21         27d
kube-system   kube-proxy-25tm5                                1/1       Running       8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running       4          27d
kube-system   kube-proxy-jmv67                                1/1       Running       4          27d
kube-system   kube-proxy-l8fks                                1/1       Running       6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running       5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running       0          4d
kube-system   weave-net-bz7wr                                 2/2       Running       20         27d
kube-system   weave-net-c2pxd                                 2/2       Running       13         27d
kube-system   weave-net-jw29c                                 2/2       Running       20         27d
kube-system   weave-net-kxxpl                                 2/2       Running       13         27d
onap          dev-appc-0                                      2/2       Terminating   0          2h
onap          dev-appc-1                                      2/2       Running       0          1h
onap          dev-appc-2                                      2/2       Running       0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running       0          2h
onap          dev-appc-db-0                                   2/2       Running       0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running       0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running       0          2h

 The new appc-0 pod will be brought to running status to replace the initial appc-0

$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                            READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s-master                                 1/1       Running   5          15d
kube-system   kube-apiserver-k8s-master                       1/1       Running   5          15d
kube-system   kube-controller-manager-k8s-master              1/1       Running   5          15d
kube-system   kube-dns-86f4d74b45-px44s                       3/3       Running   21         27d
kube-system   kube-proxy-25tm5                                1/1       Running   8          27d
kube-system   kube-proxy-6dt4z                                1/1       Running   4          27d
kube-system   kube-proxy-jmv67                                1/1       Running   4          27d
kube-system   kube-proxy-l8fks                                1/1       Running   6          27d
kube-system   kube-scheduler-k8s-master                       1/1       Running   5          15d
kube-system   tiller-deploy-84f4c8bb78-s6bq5                  1/1       Running   0          4d
kube-system   weave-net-bz7wr                                 2/2       Running   20         27d
kube-system   weave-net-c2pxd                                 2/2       Running   13         27d
kube-system   weave-net-jw29c                                 2/2       Running   20         27d
kube-system   weave-net-kxxpl                                 2/2       Running   13         27d
onap          dev-appc-0                                      2/2       Running   0          30s
onap          dev-appc-1                                      2/2       Running   0          1h
onap          dev-appc-2                                      2/2       Running   0          1h
onap          dev-appc-cdt-8cbf9d4d9-mhp4b                    1/1       Running   0          2h
onap          dev-appc-db-0                                   2/2       Running   0          2h
onap          dev-appc-dgbuilder-54766c5b87-xw6c6             1/1       Running   0          2h
onap          dev-robot-785b9bfb45-9s2rs                      1/1       Running   0          2h