Page Comparison

Setup Steps

We are using MySQL Master-Master replication strategy to set up replication across two kubernetes clusters (aka sites). The setup is done through below manual steps.

1. Verify MySQL server-id is different in both sites.

Code Block

#Login to master db pod on site 1 and site 2
ubuntu@k8s-s1-master:~$ kubectl exec -it sdnc-dbhost-0 -n onap bash
Defaulting container name to sdnc-db-container.
Use 'kubectl describe pod/sdnc-dbhost-0' to see all of the containers in this pod.
root@sdnc-dbhost-0:/#
 
#Verify server-id on site 1
root@sdnc-dbhost-0:/# cd /etc/mysql/conf.d
root@sdnc-dbhost-0:/etc/mysql/conf.d# cat server-id.cnf
[mysqld]
server-id=100

#Verify server-id on site 2
root@sdnc-dbhost-0:/# cd /etc/mysql/conf.d
root@sdnc-dbhost-0:/etc/mysql/conf.d# cat server-id.cnf
[mysqld]
server-id=200

2. If MySQL site-id for site 1 and site 2, resulting

...

from step 1, are not different, proceed with the below step to make them unique.

Info
You can skip this step if the server-ids are verified to be unique in step #1. This step is only to be followed on site 2 (i.e. secondary site).

Code Block

#Set server-id to a different number (e.g. 200) in the secondary site.
#Assuming the current value for server-id on site 2 is 100, we aim to change it to 200. Login to master DB pod on the Secondary site and change it.
ubuntu@k8s-s2-master:~$ kubectl exec -it sdnc-dbhost-0 -n onap bash
root@sdnc-dbhost-0:/# sed -i 's/100/200/g' /etc/mysql/conf.d/server-id.cnf
root@sdnc-dbhost-0:/etc/mysql/conf.d# service mysql restart
[info] Stopping MySQL Community Server 5.7.21.
...........command terminated with exit code 137
ubuntu@k8s-s1-master:~$
 
#verify
root@sdnc-dbhost-0:/# cd /etc/mysql/conf.d
root@sdnc-dbhost-0:/etc/mysql/conf.d# cat server-id.cnf
[mysqld]
server-id=200

3. Verify NodePort Service Port Number on each site.

Code Block
#verify nodeport service exists ubuntu@k8s-s1-master:~$ kubectl get svc -n onap \| grep mysql mysql-nodeport NodePort 10.107.197.242 <none> 3306:

...

30231/TCP,3307:

...

30232/TCP 29s
ubuntu@k8s-s1-master:~$
  
#verify the endpoints are assigned for the service
#make a note of Node Port Number to

...

use in step#5
ubuntu@k8s-s1-master:~$ kubectl describe svc mysql-nodeport -n onap
Name:                     mysql-nodeport
Namespace:                onap
Labels:                   statefulset.kubernetes.io/pod-name=sdnc-dbhost-0
Annotations:              <none>
Selector:                 statefulset.kubernetes.io/pod-name=sdnc-dbhost-0
Type:                     NodePort
IP:                       10.107.197.242
Port:                     sdnc-dbhost-0-port-3306  3306/TCP
TargetPort:               3306/TCP
NodePort:                 sdnc-dbhost-0-port-3306  30231/TCP
Endpoints:                10.44.0.3:3306
Port:                     sdnc-dbhost-0-port-3307  3307/TCP
TargetPort:               3307/TCP
NodePort:                 sdnc-dbhost-0-port-3307  30232/TCP
Endpoints:                10.44.0.3:3307
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
ubuntu@k8s-s1-master:~$

4.

...

Create NodePort service on both sites in "onap" namespace. Make sure the kubectl version is 1.9x. Use attached - mysql-nodeport.yaml.

Info
You can skip this step if the node port service are verified in step #3. This step is to be followed on both sites.

Code Block

ubuntu@k8s-s1-master:~$ cat mysql-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql-nodeport
  namespace: "onap"
  labels:
    statefulset.kubernetes.io/pod-name: sdnc-dbhost-0
spec:
  ports:
  - name: "sdnc-dbhost-0-port-3306"
    port: 3306
    targetPort: 3306
    nodePort: 30231
  - name: "sdnc-dbhost-0-port-3307"
    port: 3307
    targetPort: 3307
    nodePort: 30232
  type: NodePort
  selector:
    statefulset.kubernetes.io/pod-name: sdnc-dbhost-0
ubuntu@k8s-s1-master:~$
 
#create nodeport service
ubuntu@k8s-s1-master:~$ kubectl create -f mysql-nodeport.yaml
service "mysql-nodeport" created
 
#verify created service
ubuntu@k8s-s1-master:~$ kubectl get svc -n onap | grep mysql-nodeport
mysql-nodeport NodePort 10.107.197.242 <none> 3306:30031/TCP,3307:30032/TCP 29s
ubuntu@k8s-s1-master:~$
 
#verify the endpoints are assigned for the service
ubuntu@k8s-s1-master:~$ kubectl describe svc mysql-nodeport -n onap
Name:                     mysql-nodeport
Namespace:                onap
Labels:                   statefulset.kubernetes.io/pod-name=sdnc-dbhost-0
Annotations:              <none>
Selector:                 statefulset.kubernetes.io/pod-name=sdnc-dbhost-0
Type:                     NodePort
IP:                       10.107.197.242
Port:                     sdnc-dbhost-0-port-3306  3306/TCP
TargetPort:               3306/TCP
NodePort:                 sdnc-dbhost-0-port-3306  30231/TCP
Endpoints:                10.44.0.3:3306
Port:                     sdnc-dbhost-0-port-3307  3307/TCP
TargetPort:               3307/TCP
NodePort:                 sdnc-dbhost-0-port-3307  30232/TCP
Endpoints:                10.44.0.3:3307
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
ubuntu@k8s-s1-master:~$

5. Setup Master-Master Replication.

On site 2 (Secondary site):

...

Setup Master-Master Replication.

On site 2 (Secondary site):

Code Block

#Login to the master db pod
ubuntu@k8s-s1-master:~$ kubectl exec -it sdnc-dbhost-0 -n onap bash
Defaulting container name to sdnc-db-container.
Use 'kubectl describe pod/sdnc-dbhost-0' to see all of the containers in this pod.

#Login to mysql
root@sdnc-dbhost-0:/etc/mysql/conf.d# mysql -u root -popenECOMP1.0 
mysql: [Warning] Using a password on the command line interface can be insecure. 
Welcome to the MySQL monitor. Commands end with ; or \g. 
Your MySQL connection id is 46083 Server version: 5.7.21-log MySQL Community Server (GPL) 
 
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. 
 
Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. 
 
mysql>
 
#find out log bin position
mysql> show master status;
+--------------------------+----------+--------------+------------------+-------------------

...

+
| File                     | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+--------------------------+----------+--------------+------------------+-------------------+
| sdnc-dbhost-

...

0-bin.000004 | 154      |              |                  |                   |
+--------------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
mysql> stop slave;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql>


#Set: 
# - "master_host" to peer site's 'K8S master node' IP address
# - "master_port" to port number of the NodePort service (from step#3)
# - "master_user" to user name (here 'root')
# - "master_password" to user's password (here 'root' user's default password)
# - "master_log_file" to log_file name from "show master status" commands on the peer site (Primary)
# - "master_log_pos" to log_position from "show master status" commands on the peer site (Primary)
mysql> CHANGE MASTER TO master_host='10.147.112.136', master_port=30231, master_user='root', master_password='openECOMP1.0', master_log_file='sdnc-dbhost-0-bin.000002', master_log_pos=37433012;
mysql> start slave;
Query OK, 0 rows affected (0.00 sec)
mysql>

On site 1 (Primary site):

Code Block

#Login to the master db pod
ubuntu@k8s-s1-master:~$ kubectl exec -it sdnc-dbhost-0 -n onap bash
Defaulting container name to sdnc-db-container.
Use 'kubectl describe pod/sdnc-dbhost-0' to see all of the containers in this pod.

#Login to mysql
root@sdnc-dbhost-0:/etc/mysql/conf.d# mysql -u root -popenECOMP1.0 
mysql: [Warning] Using a password on the command line interface can be insecure. 
Welcome to the MySQL monitor. Commands end with ; or \g. 
Your MySQL connection id is 46083 Server version: 5.7.21-log MySQL Community Server (GPL) 
 
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. 
Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. 
mysql>
 
#find out log bin position
mysql> show master status;
+--------------------------+----------+--------------+------------------+-------------------

...

+
| File                     | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+--------------------------+----------+--------------+------------------+-------------------+
| sdnc-dbhost-0-bin.000003 | 37433012 |              |                  |                   |
+--------------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
mysql> stop slave;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql>


#Set: 
# - "master_host" to peer site's 'K8S master node' IP address
# - "master_port" to port number of the NodePort service (from step#3)
# - "master_user" to user name (here 'root')
# - "master_password" to user's password (here 'root' user's default password)
# - "master_log_file" to log_file name from "show master status" commands on peer site (Seconday)
# - "master_log_pos" to log_position from "show master status" commands on the peer site (Seconday)
mysql> CHANGE MASTER TO master_host='10.147.113.36', master_port=30231, master_user='root', master_password='openECOMP1.0', master_log_file='sdnc-dbhost-0-bin.000004',master_log_pos=154;
Query OK, 0 rows affected, 2 warnings (0.01 sec)
mysql> start slave;
Query OK, 0 rows affected (0.00 sec)
mysql>

Verify replication status on both sites.

Code Block
mysql> show slave status\G;

It should look like below:

Image Modified

Versions Compared

Old Version 10

New Version Current

Key