Page Comparison

What's

...

Desired

In order to replicate a MySQL database within SDN-C, follow these steps:

1. Have a Master and at least 1 Slave.
2. Reuse built-in MySQL inbuilt data replication for replicating data between the Master and the Slaves.
3. Using Use the Kubernetes scaling mechasism mechanism to scale the pods.

How Kubernetes

...

made Clustered Deployments Possible

• StatefulSet: 
  • Used to manage Stateful applications. 
  • Guarantees fixed numbering for a POD. 
  • Also using Using headless service, PODs were registered with there their own unique FQDN in DNS; this makes it possible for other PODs to find a POD even after a restart (albeit with a different IPAddress).
• Since we were using a single Kubernetes VM, hosting a volume dynamically on the local-store VM for newly spun slaves was not straight-forward (support wasn't inbuilt). However, Kubernetes does support writing external provisioners which did the job for us. This provisioner actually created a virtual NFS Server on top of a local store.The instance of nfs-provisioner will watch for PersistentVolumeClaims that ask for the StorageClass and automatically create NFS-backed  backed PersistentVolumes for them.
  
  

We used Kubernetes  this Kubernetes example to replicate MySQL server; this was modified to suit the needs for SDNCSDN-C DB.

Internals

For each MYSQL POD, 2 init-containers and 2 containers are spawned.:

• 2 init containers:
  • init-mysql
    1. Generates special Mysql MySQL config files based on an Ordinal index. (Save the ordinal index is saved in server-id.cnf)
    2. Uses config-map to copy the master.cnf/slave.cnf files to the conf.d directory.
  • clone-mysql:
    1. Performs clone operation first time the Slave comes up - assuming that Master already has some data on it when the Slave starts.
    2. Uses Opensource tool Percona for this job
    3. Hot backup the data from previous MySQL pod using binlog which is powered by Percona xtrabackup (https://www.percona.com/software/mysql-database/percona-xtrabackup)
• 2 containers:
  • mysqld:
    • Actual mysql MySQL server
  • xtrabackup Xtrabackup sidecar:
    1. Handles all of the replication between this server and the Master.
    2. Handles request requests from other Pods for data cloning.

2 Services are created:

• DBHost should be used for any write operation (writes to master)
• DBHost-Read should be used for read operations (can query Slaves besides master)

As mentioned above, used nfs-provisioner was used to dynamically create Persistent Volume Claims to enable dynamic scaling of slaves.

Master Failure, A Caveat

Unfortunately, if a master fails, then we need to write a script (or an application) to promote one of the slaves to be the master and instruct instruct the other slaves and applications to change to the new master. You can see more details here.

Other way is to use GTID based replication.To support Master faliover, recommended solution is using GTID-based replication combined with Automatic replication health monitoring and failover. 

Advantages 

• Can have multiple slaves Slaves with a Master server.
• Allows scaling slaves dynamically.
• Any data-write is done to Master, but a data-read can happen on Slaves as well. Hence a 'DBHost-Read' Service was introduced which should be used by Clients for data-fetch operations.
• For any write operation. , the write service - DBHost - can be used.
• Once a Slave is replicated from masterMaster, that Slave is then used to replicate data on any new Slave; Low this has a low impact on the Master server.

Examples:

1. Running

   mysql

   a MySQL client to create the DB and Table and fetch it using the DBHost-Read service:

   Code Block
   collapse true
   kubectl run mysql-client --image=mysql:5.7 -i --rm --restart=Never -- mysql -h sdnc-dbhost-0.dbhost.onap-sdnc -uroot -popenECOMP1.0 <<EOF CREATE DATABASE test; CREATE TABLE test.messages (message VARCHAR(250)); INSERT INTO test.messages VALUES ('hello'); EOF kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never -- mysql -uroot -popenECOMP1.0 -h dbhost-read.onap-sdnc -e "SELECT * FROM test.messages"

2. To demonstrate that DBHost-read distributes the service across slaves, see the ServerID changing in

   it's

   its response

   Code Block
   collapse true
   kubectl run mysql-client-loop --image=mysql:5.7 -i -t --rm --restart=Never -- bash -ic "while sleep 0.5; do mysql -uroot -popenECOMP1.0 -h dbhost-read.onap-sdnc -e 'SELECT @@server_id,NOW()'; done"

3. Can scale (up or down)

   mysql

   MySQL dynamically:

   Code Block
   collapse true
   Scale up: kubectl scale statefulset sdnc-dbhost -n onap-sdnc --replicas=5   Scale Down kubectl scale statefulset sdnc-dbhost -n onap-sdnc --replicas=2