Manual Site Failover

1 Deprecated

Deprecated

The procedures documented here are only applicable to the Beijing release.

Follow below steps for manual site failover. All steps need to be run on coredns master node.

Please note the configuration in all examples for reference:

coredns master node IP address: 10.147.101.135

primary site (site1) master node IP address: 10.147.99.140

secondary site (site2) master node IP address: 10.147.101.23

Verify coredns server, to get the existing mapping. (here it is pointing to primary site(site 1))

#verify the address for sdnc.example.com resolves to primary site presently
root@coredns-1:/dockerdata-nfs# nslookup sdnc.example.com
Server:         10.96.0.10
Address:        10.96.0.10#53
Name:   sdnc.example.com
Address: 10.147.99.140

2. Edit zone file to comment out SDNC mapping to primary site (site1) and uncomment mapping to secondary site (site2)

root@coredns-1:~# vi /dockerdata-nfs/zone.db

3. Edit coredns configmap on coredns server to comment out SDNC mapping to primary site (site1) and uncomment mapping to secondary site (site2)

#Below command opens the codedns configmap for editing. Edit and save the file.
# Notice the A record for sdnc: "sdnc         IN  A  10.147.99.140" is commented out by appending ;; to the line (\n;;sdnc         IN  A  10.147.99.140\n)
# Notice the A record for sdnc: "sdnc         IN  A  10.147.101.23" is uncommented out by removing;; from the line (\nsdnc\t\t    IN A   10.147.101.23)
root@coredns-1:~# kubectl edit configmap coredns -n kube-system -oyaml
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        log
        health
        kubernetes cluster.local 10.96.0.0/12 {
           pods insecure
        }
        file /dockerdata-nfs/zone.db example.com
        prometheus
        proxy . /etc/resolv.conf
        cache 30
    }
  zone.db: "$ORIGIN example.com.     ; designates the start of this zone file in the
    namespace\n$TTL 1h         ; default expiration time of all resource records without
    their own TTL value\nexample.com.  IN  SOA   ns.example.com. username.example.com.
    ( 2007120710 1d 2h 4w 1h )\nexample.com.  IN  NS    ns                    ; ns.example.com
    is a nameserver for example.com\nexample.com.  IN  NS    ns.somewhere.example.
    ; ns.somewhere.example is a backup nameserver for example.com\nexample.com.  IN
    \ A     10.147.101.135             ; IPv4 address for example.com\nns            IN
    \ A     10.247.5.11             ; IPv4 address for ns.example.com\nwww           IN
    \ CNAME example.com.          ; www.example.com is an alias for example.com\nwwwtest
    \      IN  CNAME www              ; wwwtest.example.com is another alias for www.example.com\nsdnc.example.com.
    \   IN      SRV    30202 10 10 example.com.\n;;site1\n;;sdnc         IN  A  10.147.99.140\n;;site2\nsdnc\t\t
    IN A   10.147.101.23"
kind: ConfigMap
metadata:
  creationTimestamp: 2018-02-28T20:13:03Z
  name: coredns
  namespace: kube-system
  resourceVersion: "102077"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: c8489771-1cc3-11e8-a0cb-fa163eabcb60

configmap "coredns" edited

4. Note that there is a cache time configured in configmap. Wait for some time (30 seconds here) and then send signal to refresh the settings for coredns.

5. Verify the "sdnc.example.com" domain points to secondary site now.

It may take some time to refresh the address for DNS resolver depending on configured cache time. Send the refresh signal again (in step 4) after sometime if you're not able to see the update.