Setup Steps
We are using MySQL Master-Master replication strategy to set up replication across two kubernetes clusters (aka sites). The setup is done through below manual steps.
1. Verify MySQL server-id is different in both sites.
#Login to master db pod on site 1 and site 2 ubuntu@k8s-s1-master:~$ kubectl exec -it sdnc-dbhost-0 -n onap bash Defaulting container name to sdnc-db-container. Use 'kubectl describe pod/sdnc-dbhost-0' to see all of the containers in this pod. root@sdnc-dbhost-0:/# #Verify server-id on site 1 root@sdnc-dbhost-0:/# cd /etc/mysql/conf.d root@sdnc-dbhost-0:/etc/mysql/conf.d# cat server-id.cnf [mysqld] server-id=100 #Verify server-id on site 2 root@sdnc-dbhost-0:/# cd /etc/mysql/conf.d root@sdnc-dbhost-0:/etc/mysql/conf.d# cat server-id.cnf [mysqld] server-id=200
2. If MySQL site-id for site 1 and site 2, resulting from step 1, are not different, proceed with the below step to make them unique.
You can skip this step if the server-ids are verified to be unique in step #1. This step is only to be followed on site 2 (i.e. secondary site).
#Set server-id to a different number (e.g. 200) in the secondary site. #Assuming the current value for server-id on site 2 is 100, we aim to change it to 200. Login to master DB pod on the Secondary site and change it. ubuntu@k8s-s2-master:~$ kubectl exec -it sdnc-dbhost-0 -n onap bash root@sdnc-dbhost-0:/# sed -i 's/100/200/g' /etc/mysql/conf.d/server-id.cnf root@sdnc-dbhost-0:/etc/mysql/conf.d# service mysql restart [info] Stopping MySQL Community Server 5.7.21. ...........command terminated with exit code 137 ubuntu@k8s-s1-master:~$ #verify root@sdnc-dbhost-0:/# cd /etc/mysql/conf.d root@sdnc-dbhost-0:/etc/mysql/conf.d# cat server-id.cnf [mysqld] server-id=200
3. Verify NodePort Service Port Number on each site.
#verify nodeport service exists ubuntu@k8s-s1-master:~$ kubectl get svc -n onap | grep mysql mysql-nodeport NodePort 10.107.197.242 <none> 3306:30031/TCP,3307:30032/TCP 29s ubuntu@k8s-s1-master:~$ #verify the endpoints are assigned for the service #make a note of Node Port Number to use in step#5 ubuntu@k8s-s1-master:~$ kubectl describe svc mysql-nodeport -n onap Name: mysql-nodeport Namespace: onap Labels: statefulset.kubernetes.io/pod-name=sdnc-dbhost-0 Annotations: <none> Selector: statefulset.kubernetes.io/pod-name=sdnc-dbhost-0 Type: NodePort IP: 10.107.197.242 Port: sdnc-dbhost-0-port-3306 3306/TCP TargetPort: 3306/TCP NodePort: sdnc-dbhost-0-port-3306 30231/TCP Endpoints: 10.44.0.3:3306 Port: sdnc-dbhost-0-port-3307 3307/TCP TargetPort: 3307/TCP NodePort: sdnc-dbhost-0-port-3307 30232/TCP Endpoints: 10.44.0.3:3307 Session Affinity: None External Traffic Policy: Cluster Events: <none> ubuntu@k8s-s1-master:~$
4. Create NodePort service on both sites in "onap" namespace. Make sure the kubectl version is 1.9x. Use attached - mysql-nodeport.yaml.
You can skip this step if the node port service verified in step #3 already exists.
ubuntu@k8s-s1-master:~$ cat mysql-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql-nodeport
namespace: "onap"
labels:
statefulset.kubernetes.io/pod-name: sdnc-dbhost-0
spec:
ports:
- name: "sdnc-dbhost-0-port-3306"
port: 3306
targetPort: 3306
nodePort: 30231
- name: "sdnc-dbhost-0-port-3307"
port: 3307
targetPort: 3307
nodePort: 30232
type: NodePort
selector:
statefulset.kubernetes.io/pod-name: sdnc-dbhost-0
ubuntu@k8s-s1-master:~$
#create nodeport service
ubuntu@k8s-s1-master:~$ kubectl create -f mysql-nodeport.yaml
service "mysql-nodeport" created
#verify created service
ubuntu@k8s-s1-master:~$ kubectl get svc -n onap | grep mysql-nodeport
mysql-nodeport NodePort 10.107.197.242 <none> 3306:30031/TCP,3307:30032/TCP 29s
ubuntu@k8s-s1-master:~$
#verify the endpoints are assigned for the service
ubuntu@k8s-s1-master:~$ kubectl describe svc mysql-nodeport -n onap
Name: mysql-nodeport
Namespace: onap
Labels: statefulset.kubernetes.io/pod-name=sdnc-dbhost-0
Annotations: <none>
Selector: statefulset.kubernetes.io/pod-name=sdnc-dbhost-0
Type: NodePort
IP: 10.107.197.242
Port: sdnc-dbhost-0-port-3306 3306/TCP
TargetPort: 3306/TCP
NodePort: sdnc-dbhost-0-port-3306 30231/TCP
Endpoints: 10.44.0.3:3306
Port: sdnc-dbhost-0-port-3307 3307/TCP
TargetPort: 3307/TCP
NodePort: sdnc-dbhost-0-port-3307 30232/TCP
Endpoints: 10.44.0.3:3307
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
ubuntu@k8s-s1-master:~$
5. Setup Master-Master Replication.
On site 2 (Secondary site):
#Login to the master db pod ubuntu@k8s-s1-master:~$ kubectl exec -it sdnc-dbhost-0 -n onap bash Defaulting container name to sdnc-db-container. Use 'kubectl describe pod/sdnc-dbhost-0' to see all of the containers in this pod. #Login to mysql root@sdnc-dbhost-0:/etc/mysql/conf.d# mysql -u root -popenECOMP1.0 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 46083 Server version: 5.7.21-log MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> #find out log bin position mysql> show master status; +--------------------------+----------+--------------+------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +--------------------------+----------+--------------+------------------+-------------------+ | sdnc-dbhost-0-bin.000004 | 154 | | | | +--------------------------+----------+--------------+------------------+-------------------+ 1 row in set (0.00 sec) mysql> stop slave; Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> #Set: # - "master_host" to peer site's 'K8S master node' IP address # - "master_port" to port number of the NodePort service (from step#3) # - "master_user" to user name (here 'root') # - "master_password" to user's password (here 'root' user's default password) # - "master_log_file" to log_file name from "show master status" commands on the peer site (Primary) # - "master_log_pos" to log_position from "show master status" commands on the peer site (Primary) mysql> CHANGE MASTER TO master_host='10.147.112.136', master_port=30231, master_user='root', master_password='openECOMP1.0', master_log_file='sdnc-dbhost-0-bin.000002', master_log_pos=37433012; mysql> start slave; Query OK, 0 rows affected (0.00 sec) mysql>
On site 1 (Primary site):
#Login to the master db pod ubuntu@k8s-s1-master:~$ kubectl exec -it sdnc-dbhost-0 -n onap bash Defaulting container name to sdnc-db-container. Use 'kubectl describe pod/sdnc-dbhost-0' to see all of the containers in this pod. #Login to mysql root@sdnc-dbhost-0:/etc/mysql/conf.d# mysql -u root -popenECOMP1.0 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 46083 Server version: 5.7.21-log MySQL Community Server (GPL) Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> #find out log bin position mysql> show master status; +--------------------------+----------+--------------+------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +--------------------------+----------+--------------+------------------+-------------------+ | sdnc-dbhost-0-bin.000003 | 37433012 | | | | +--------------------------+----------+--------------+------------------+-------------------+ 1 row in set (0.00 sec) mysql> stop slave; Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> #Set: # - "master_host" to peer site's 'K8S master node' IP address # - "master_port" to port number of the NodePort service (from step#3) # - "master_user" to user name (here 'root') # - "master_password" to user's password (here 'root' user's default password) # - "master_log_file" to log_file name from "show master status" commands on peer site (Seconday) # - "master_log_pos" to log_position from "show master status" commands on the peer site (Seconday) mysql> CHANGE MASTER TO master_host='10.147.113.36', master_port=30231, master_user='root', master_password='openECOMP1.0', master_log_file='sdnc-dbhost-0-bin.000004',master_log_pos=154; Query OK, 0 rows affected, 2 warnings (0.01 sec) mysql> start slave; Query OK, 0 rows affected (0.00 sec) mysql>
Verify replication status on both sites.
mysql> show slave status\G;
It should look like below:
