Implementing Geo-Redundancy via Affinity/AntiAffinity

Geo-Redundancy:-

Problem/Requirement:-

•  As an ONAP operator- We require the capability of ONAP -K8 containers deployment in a Geo-Red fashion i.e. into specific zone/regions thus providing protection from a single cluster failure.
•  Besides HA, We should have the ability to deploy any container(POD) with our choice with use cases as such if a container used Intel DPDK technology the pod may state that it has to be deployed to an Intel processor based host node, Also another use case could be to ensure placement of a DCAE complex close to the VNFs generating high volumes of traffic thus minimizing networking cost.

Solution Delivered (POC):-

•  Affinity/Anti-Affinity is one of the K8 feature which enable us to define affinity-anti affinity rules to make sure the POD placement works extensively, So for POC, We used VNFSDK component with deploying it with certain conditions like:-

1. 1. 1. No Application container replicas to be on the same node(host).
      2. No two DB container replicas on the same node (host).
      3. Both the APP and DB container for 1 replica set to be co located on a single node(host).

For achieving this we used a 4 node setup as shown below 

And we used the default labels provided by kubernetes to each nodes 

AntiAffinity between DB Pods

Now lets see how the codes looks like for configuring the Anti-Affinity for VNFSDK-POSTGRESS PODs (just for the POC purpose we increased the replica count to 4)

affinity:

  podAntiAffinity:

    requiredDuringSchedulingIgnoredDuringExecution:

    - labelSelector:

        matchExpressions:

        - key: app

          operator: In

          values:

          - vnfsdk-postgres   

      topologyKey: "kubernetes.io/hostname"

This snippet of values.yaml  under vnfsdk-postgress dir  ensures that the db pods of vnfsdk-postgress will never reside on the same nodes

Affinity for DB and App pods and Antiaffinity for APP pods

Now for the Affinity between the DBand the APPpods and also the antiaffinity between APP pods we used the below code snippet in values.yaml of vnfsdk

affinity:

  podAntiAffinity:

    requiredDuringSchedulingIgnoredDuringExecution:

    - labelSelector:

        matchExpressions:

        - key: app

          operator: In

          values:

          - vnfsdk

      topologyKey: "kubernetes.io/hostname"

  podAffinity:

    requiredDuringSchedulingIgnoredDuringExecution:

    - labelSelector:

        matchExpressions:

        - key: app

          operator: In

          values:

          - vnfsdk-postgres

      topologyKey: "kubernetes.io/hostname"

When the code was deployed the result was as below 

default goodly-squid-vnfsdk-556f59ccd9-jx9q8 1/1 Running 0 32s 10.42.81.107 k8s-4 default goodly-squid-vnfsdk-556f59ccd9-n95jh 1/1 Running 0 32s 10.42.73.55 k8s-2 default goodly-squid-vnfsdk-556f59ccd9-snlzc 1/1 Running 0 32s 10.42.233.242 k8s-3 default goodly-squid-vnfsdk-556f59ccd9-xtzff 1/1 Running 0 32s 10.42.129.25 k8s-1 default goodly-squid-vnfsdk-postgres-78d58775c4-98dr9 1/1 Running 0 32s 10.42.130.80 k8s-4 default goodly-squid-vnfsdk-postgres-78d58775c4-9cf5g 1/1 Running 0 32s 10.42.222.207 k8s-3 default goodly-squid-vnfsdk-postgres-78d58775c4-9rnhh 1/1 Running 0 32s 10.42.50.224 k8s-1 default goodly-squid-vnfsdk-postgres-78d58775c4-s4l8r 1/1 Running 0 32s 10.42.49.238 k8s-2

Replica sets on different nodes with the APP and DB on the same node while the APP and DB are also never colocated on the same node

So with this, We achieved a deployment Active-Active Geo-redundancy.

• Now for each and every component using specific DB/application need to be tested for Geo-redundancy like SDNC/Clamp already started.
• Federation feature can also be used for achieving HA across multi-cluster deployments.

Node affinity (beta feature)

Node affinity was introduced as alpha in Kubernetes 1.2. Node affinity is conceptually similar to nodeSelector – it allows you to constrain which nodes your pod is eligible to be scheduled on, based on labels on the node.

There are currently two types of node affinity, called requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution. You can think of them as “hard” and “soft” respectively, in the sense that the former specifies rules that must be met for a pod to be scheduled onto a node (just like nodeSelector but using a more expressive syntax), while the latter specifies preferences that the scheduler will try to enforce but will not guarantee. The “IgnoredDuringExecution” part of the names means that, similar to how nodeSelector works, if labels on a node change at runtime such that the affinity rules on a pod are no longer met, the pod will still continue to run on the node. In the future we plan to offer requiredDuringSchedulingRequiredDuringExecution which will be just like requiredDuringSchedulingIgnoredDuringExecution except that it will evict pods from nodes that cease to satisfy the pods’ node affinity requirements.

Thus an example of requiredDuringSchedulingIgnoredDuringExecution would be “only run the pod on nodes with Intel CPUs” and an example preferredDuringSchedulingIgnoredDuringExecution would be “try to run this set of pods in availability zone XYZ, but if it’s not possible, then allow some to run elsewhere”.

Node affinity is specified as field nodeAffinity of field affinity in the PodSpec.

Here’s an example of a pod that uses node affinity:

pods/pod-with-node-affinity.yaml

apiVersion: v1 kind: Pod metadata: name: with-node-affinity spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/e2e-az-name operator: In values: - e2e-az1 - e2e-az2 preferredDuringSchedulingIgnoredDuringExecution: - weight: 1 preference: matchExpressions: - key: another-node-label-key operator: In values: - another-node-label-value containers: - name: with-node-affinity image: k8s.gcr.io/pause:2.0

This node affinity rule says the pod can only be placed on a node with a label whose key is kubernetes.io/e2e-az-name and whose value is either e2e-az1 or e2e-az2. In addition, among nodes that meet that criteria, nodes with a label whose key is another-node-label-key and whose value is another-node-label-value should be preferred.

You can see the operator In being used in the example. The new node affinity syntax supports the following operators: In, NotIn, Exists, DoesNotExist, Gt, Lt. You can use NotIn and DoesNotExist to achieve node anti-affinity behavior, or usenode taints to repel pods from specific nodes.

If you specify both nodeSelector and nodeAffinity, both must be satisfied for the pod to be scheduled onto a candidate node.

If you specify multiple nodeSelectorTerms associated with nodeAffinity types, then the pod can be scheduled onto a node if one of the nodeSelectorTerms is satisfied.

If you specify multiple matchExpressions associated with nodeSelectorTerms, then the pod can be scheduled onto a node only if all matchExpressions can be satisfied.

If you remove or change the label of the node where the pod is scheduled, the pod won’t be removed. In other words, the affinity selection works only at the time of scheduling the pod.

The weight field in preferredDuringSchedulingIgnoredDuringExecution is in the range 1-100. For each node that meets all of the scheduling requirements (resource request, RequiredDuringScheduling affinity expressions, etc.), the scheduler will compute a sum by iterating through the elements of this field and adding “weight” to the sum if the node matches the corresponding MatchExpressions. This score is then combined with the scores of other priority functions for the node. The node(s) with the highest total score are the most preferred.

For more information on node affinity, see the design doc.

Inter-pod affinity and anti-affinity (beta feature)

Inter-pod affinity and anti-affinity were introduced in Kubernetes 1.4. Inter-pod affinity and anti-affinity allow you to constrain which nodes your pod is eligible to be scheduled based on labels on pods that are already running on the node rather than based on labels on nodes. The rules are of the form “this pod should (or, in the case of anti-affinity, should not) run in an X if that X is already running one or more pods that meet rule Y”. Y is expressed as a LabelSelector with an associated list of namespaces; unlike nodes, because pods are namespaced (and therefore the labels on pods are implicitly namespaced), a label selector over pod labels must specify which namespaces the selector should apply to. Conceptually X is a topology domain like node, rack, cloud provider zone, cloud provider region, etc. You express it using a topologyKey which is the key for the node label that the system uses to denote such a topology domain, e.g. see the label keys listed above in the section Interlude: built-in node labels.

Note: Inter-pod affinity and anti-affinity require substantial amount of processing which can slow down scheduling in large clusters significantly. We do not recommend using them in clusters larger than several hundred nodes.

Note: Pod anti-affinity requires nodes to be consistently labelled, i.e. every node in the cluster must have an appropriate label matching topologyKey. If some or all nodes are missing the specified topologyKey label, it can lead to unintended behavior.

As with node affinity, there are currently two types of pod affinity and anti-affinity, called requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution which denote “hard” vs. “soft” requirements. See the description in the node affinity section earlier. An example of requiredDuringSchedulingIgnoredDuringExecution affinity would be “co-locate the pods of service A and service B in the same zone, since they communicate a lot with each other” and an example preferredDuringSchedulingIgnoredDuringExecution anti-affinity would be “spread the pods from this service across zones” (a hard requirement wouldn’t make sense, since you probably have more pods than zones).

Inter-pod affinity is specified as field podAffinity of field affinity in the PodSpec. And inter-pod anti-affinity is specified as field podAntiAffinity of field affinity in the PodSpec.

An example of a pod that uses pod affinity:

pods/pod-with-pod-affinity.yaml

apiVersion: v1 kind: Pod metadata: name: with-pod-affinity spec: affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: security operator: In values: - S1 topologyKey: failure-domain.beta.kubernetes.io/zone podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: security operator: In values: - S2 topologyKey: kubernetes.io/hostname containers: - name: with-pod-affinity image: k8s.gcr.io/pause:2.0

The affinity on this pod defines one pod affinity rule and one pod anti-affinity rule. In this example, the podAffinity is requiredDuringSchedulingIgnoredDuringExecution while the podAntiAffinity is preferredDuringSchedulingIgnoredDuringExecution. The pod affinity rule says that the pod can be scheduled onto a node only if that node is in the same zone as at least one already-running pod that has a label with key “security” and value “S1”. (More precisely, the pod is eligible to run on node N if node N has a label with key failure-domain.beta.kubernetes.io/zone and some value V such that there is at least one node in the cluster with key failure-domain.beta.kubernetes.io/zone and value V that is running a pod that has a label with key “security” and value “S1”.) The pod anti-affinity rule says that the pod prefers not to be scheduled onto a node if that node is already running a pod with label having key “security” and value “S2”. (If the topologyKey were failure-domain.beta.kubernetes.io/zone then it would mean that the pod cannot be scheduled onto a node if that node is in the same zone as a pod with label having key “security” and value “S2”.) See the design doc for many more examples of pod affinity and anti-affinity, both the requiredDuringSchedulingIgnoredDuringExecution flavor and the preferredDuringSchedulingIgnoredDuringExecution flavor.

The legal operators for pod affinity and anti-affinity are In, NotIn, Exists, DoesNotExist.

In principle, the topologyKey can be any legal label-key. However, for performance and security reasons, there are some constraints on topologyKey:

1. For affinity and for requiredDuringSchedulingIgnoredDuringExecution pod anti-affinity, empty topologyKey is not allowed.
2. For requiredDuringSchedulingIgnoredDuringExecution pod anti-affinity, the admission controller LimitPodHardAntiAffinityTopology was introduced to limit topologyKey to kubernetes.io/hostname. If you want to make it available for custom topologies, you may modify the admission controller, or simply disable it.
3. For preferredDuringSchedulingIgnoredDuringExecution pod anti-affinity, empty topologyKey is interpreted as “all topologies” (“all topologies” here is now limited to the combination of kubernetes.io/hostname, failure-domain.beta.kubernetes.io/zone and failure-domain.beta.kubernetes.io/region).
4. Except for the above cases, the topologyKey can be any legal label-key.

In addition to labelSelector and topologyKey, you can optionally specify a list namespaces of namespaces which the labelSelector should match against (this goes at the same level of the definition as labelSelector and topologyKey). If omitted or empty, it defaults to the namespace of the pod where the affinity/anti-affinity definition appears.

All matchExpressions associated with requiredDuringSchedulingIgnoredDuringExecution affinity and anti-affinity must be satisfied for the pod to be scheduled onto a node.

More Practical Use-cases

Interpod Affinity and AntiAffinity can be even more useful when they are used with higher level collections such as ReplicaSets, StatefulSets, Deployments, etc. One can easily configure that a set of workloads should be co-located in the same defined topology, eg., the same node.

Always co-located in the same node

In a three node cluster, a web application has in-memory cache such as redis. We want the web-servers to be co-located with the cache as much as possible. Here is the yaml snippet of a simple redis deployment with three replicas and selector label app=store. The deployment has PodAntiAffinity configured to ensure the scheduler does not co-locate replicas on a single node.

apiVersion: apps/v1 kind: Deployment metadata:  name: redis-cache spec:  selector:  matchLabels:  app: store  replicas: 3  template:  metadata:  labels:  app: store  spec:  affinity:  podAntiAffinity:  requiredDuringSchedulingIgnoredDuringExecution:  - labelSelector:  matchExpressions:  - key: app  operator: In  values:  - store  topologyKey: "kubernetes.io/hostname"  containers:  - name: redis-server  image: redis:3.2-alpine

The below yaml snippet of the webserver deployment has podAntiAffinity and podAffinity configured. This informs the scheduler that all its replicas are to be co-located with pods that have selector label app=store. This will also ensure that each web-server replica does not co-locate on a single node.

apiVersion: apps/v1 kind: Deployment metadata:  name: web-server spec:  selector:  matchLabels:  app: web-store  replicas: 3  template:  metadata:  labels:  app: web-store  spec:  affinity:  podAntiAffinity:  requiredDuringSchedulingIgnoredDuringExecution:  - labelSelector:  matchExpressions:  - key: app  operator: In  values:  - web-store  topologyKey: "kubernetes.io/hostname"  podAffinity:  requiredDuringSchedulingIgnoredDuringExecution:  - labelSelector:  matchExpressions:  - key: app  operator: In  values:  - store  topologyKey: "kubernetes.io/hostname"  containers:  - name: web-app  image: nginx:1.12-alpine

If we create the above two deployments, our three node cluster should look like below.

As you can see, all the 3 replicas of the web-server are automatically co-located with the cache as expected.

$ kubectl get pods -o wide
NAME                           READY     STATUS    RESTARTS   AGE       IP           NODE
redis-cache-1450370735-6dzlj   1/1       Running   0          8m        10.192.4.2   kube-node-3
redis-cache-1450370735-j2j96   1/1       Running   0          8m        10.192.2.2   kube-node-1
redis-cache-1450370735-z73mh   1/1       Running   0          8m        10.192.3.1   kube-node-2
web-server-1287567482-5d4dz    1/1       Running   0          7m        10.192.2.3   kube-node-1
web-server-1287567482-6f7v5    1/1       Running   0          7m        10.192.4.3   kube-node-3
web-server-1287567482-s330j    1/1       Running   0          7m        10.192.3.2   kube-node-2

Never co-located in the same node

The above example uses PodAntiAffinity rule with topologyKey: "kubernetes.io/hostname" to deploy the redis cluster so that no two instances are located on the same host. See ZooKeeper tutorial for an example of a StatefulSet configured with anti-affinity for high availability, using the same technique.

For more information on inter-pod affinity/anti-affinity, see the design doc.

You may want to check Taints as well, which allow a node to repel a set of pods.