All of the following commands are executed on federation master node.

1 Facing errors? Need to redeploy federation?

5.1 MODIFY KUBE CONFIG OF FEDERATION CLUSTER

Follow below steps for federation master node. change the default kubernetes config values to make them unique.

By default, we have below kube config after k8s cluster installation:

ubuntu@kubefed-1:~$ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://10.147.113.131:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED
ubuntu@kubefed-1:~$

As we intend to use federation cluster as the "host cluster", we added "host" to cluster,user and context names to existing kube config so as to make these names unique.

The config view on federation master node after making changes will look like below:

ubuntu@kubefed-1:~$ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://10.147.113.131:6443
  name: kubernetes-host
contexts:
- context:
    cluster: kubernetes-host
    user: kubernetes-adminhost
  name: kubernetes-admin-host
current-context: kubernetes-admin-host
kind: Config
preferences: {}
users:
- name: kubernetes-adminhost
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED
ubuntu@kubefed-1:~$

5.2 MERGE KUBE CONFIG OF SITES INTO KUBE CONFIG OF FEDERATION

Manual Merge

Edit ~/.kube/config on federation master node to incorporate underlying kubernetes site's configuration. From ~/.kube/config file on each site's master node, extract "block of data" for cluster, context and user and insert them in the ~/.kube/config file of federation master node. Make sure you follow yaml file rules, when putting the same set of data together. In other words,cluster, make sure "block of data" for: cluster context and use, for site-1, site-2, .. and federation site will be merged in a single file, under clusters, contexts and users.

This way, the ~/.kube/config file on the federation master node, will carry all information it needs to communicate with underlying sites (their master node).

Merge using commands

Copy kube config files to be merged (from site-1 master and site-2 master) to some path on federation master. Copy the federation master node config file at the same path. The kube config files can then be merged using below command -

Syntax:

KUBECONFIG=<path-to-first-site.kubeconfig>:<path-to-second-site.kubeconfig>:<path-to-federation-cluster.kubeconfig> \
kubectl config view --raw --flatten > federation.kubeconfig

As a result, federation.kubeconfig file will contain definitions of the local federation cluster, site-1-cluster, and site-2-cluster. This file can be then moved to kube config path as "config".

Example below:

#copy config files to be merged at some path
ubuntu@kubefed-1:~$# ls -lrt | grep config
-rw-r--r-- 1 root root  5448 Feb  1 14:51 first.config
-rw-r--r-- 1 root root  5453 Feb  1 14:53 second.config
-rw-r--r-- 1 root root  5453 Feb  1 14:53 fed.config

#below example command merges two sites config files with fed config file
ubuntu@kubefed-1:~$# KUBECONFIG=first.config:second.config:fed.config \
> kubectl config view --raw --flatten > federation.kubeconfig
 
#notice the new config file federation.kubeconfig
ubuntu@kubefed-1:~$# ls -lrt | grep config
-rw-r--r-- 1 root root  5448 Feb  1 14:51 first.config
-rw-r--r-- 1 root root  5453 Feb  1 14:53 second.config
-rw-r--r-- 1 root root  5453 Feb  1 14:53 fed.config
-rw-r--r-- 1 root root 10785 Feb  1 14:55 federation.kubeconfig
 
#move file to kube config and give appropriate permission
ubuntu@kubefed-1:~$# mv federation.kubeconfig $HOME/.kube/config
ubuntu@kubefed-1:~$# sudo chown $(id -u):$(id -g) $HOME/.kube/config
ubuntu@kubefed-1:~$#

Verify config after merging:

5.3 CHANGE CURRENT CONTEXT
We picked federation cluster as federation host cluster. So we need to execute "kubectl config use-context <contexts.context.name of host cluster>" on the federation master node to change current-context to that of the federation host cluster.

5.4 DEPLOY FEDERATION CONTROL PLANE

The command "kubefed init" is used to setup federation control plane and it should be fed with several parameters. See below syntax:

kubefed init <federation-name>\
--host-cluster-context=<your-host-cluster-context> \
--dns-provider="coredns" \
--dns-zone-name="example.com." \
--dns-provider-config="path to your coredns-provider.conf" \
--api-server-service-type="NodePort" \
--api-server-advertise-address="<federation master node ip address>"

where:

federation-name: An arbitrary string , to represent the federation. We picked "enterprise".
dns-provider: set to coredns, as explained in 5. Deploying Federation Control Plane

Federation Control Plane stores it's state in etcd which is stored in a persistent volume.

If you followed 5. Deploying Federation Control Plane , and created PV, then the folloing example should work for you. Make sure to set the api-server-advertise-address to the right value.

If you skip 5. Deploying Federation Control Plane , and have not created PV, then the "etcd-persistent-storage" parameter is set to false. Then the folloing example should work for you. Make sure to set the api-server-advertise-address to the right value.

Example output of a successful executions:

5.5 VERIFICATION LIST

PV and PVC status on host cluster

If persistent storage is disabled using parameter "etcd-persistent-storage=false", we do not need to verify PV and PVC status.

Upon successful execution of "kubefed init" command,

A new PVC (pv claim) is created.
The status of the exisitng pv and new pvc will change from "Available" to "Bound".
2 new deployments for apiserver and contoller-manager are created.
2 new federations pods are created under "federation-system" namespace.
2 new service accounts are created under "federation-system" namespace.
A new cluster is created. It's name is the "federation name" we provided as the first parameter for "kubefed init". (ere enterprise). The ~/.kube/config is also updated with infromation of the new cluster

New federation pods on host cluster

Two new pods are created unsed "federation-system" namespace.

Kube config update on federation master node

The ~/.kube/config file on federation master node is updated and a new cluster (here: enterprise) is added. Check below:

Facing errors? Need to redeploy federation?

Undeploy federation control plane with below steps:

Delete the federation-system namespace. This will terminate 2 pods which were created in the federation-system. Also the PV whcihch was bound to PVC, will be "Released".
Delete the PV which is in "Released" status.
Remove the new cluster "enterprise" from "~/.kube/config " . Then make sure "kubectl config view" returns back clean .

You can now try to redeploy federation control plane using kubefed init.