Page Comparison

What's desired

For replicating database(MySQL) in SDNC following things were needed:

1. Have a Master and at least 1 Slave.
2. Reuse MySQL inbuilt data replication for replicating data between Master and Slaves.
3. Using Kubernetes scaling mechasism to scale the pods.

How Kubernetes enabled it

• StatefulSet: 
  • Used to manage Stateful applications. 
  • Guarantees fixed numbering for a POD. 
  • Also using headless service, PODs were registered with there own unique FQDN in DNS; this makes it possible for other PODs to find a POD even after restart (with different IPAddress).
• Since we were using a single Kubernetes VM, hosting a volume dynamically on the local-store VM for newly spun slaves was not straight-forward (support wasn't inbuilt). However, Kubernetes does support writing external provisioners which did the job for us. This provisioner actually created a virtual NFS Server on top of a local store.The instance of nfs-provisioner will watch for PersistentVolumeClaims that ask for the StorageClass and automatically create NFS-backed  PersistentVolumes for them.
  
  

We used Kubernetes example to replicate MySQL server; this was modified to suit the needs for SDNC-DB.

Internals

For each MYSQL POD, 2 init-containers and 2 containers are spawned.

...

As mentioned above, used nfs-provisioner to dynamically create Persistent Volume Claims to enable dynamic scaling of slaves.

Master Failure

Unfortunately if a master fails, we need to write a script (or an application) to promote one of the slaves to be the master and instruct other slaves and applications to change to the new master. You can see more details here.

Other way is to use GTID based replication.

Advantages 

• Can have multiple slaves with a Master server.
• Allows scaling slaves dynamically.
• Any data write is done to Master but data-read can happen on Slaves as well. Hence a 'DBHost-Read' Service was introduced which should be used by Clients for data fetch operations.
• For any write operation. the write service - DBHost - can be used.
• Once a Slave is replicated from master, that Slave is then used to replicate data on any new Slave; Low impact on the Master server.

Examples:

1. Running mysql client to create DB and Table and fetch it using DBHost-Read service:

   Code Block
   collapse true
   kubectl run mysql-client --image=mysql:5.7 -i --rm --restart=Never -- mysql -h sdnc-dbhost-0.dbhost.onap-sdnc -uroot -popenECOMP1.0 <<EOF CREATE DATABASE test; CREATE TABLE test.messages (message VARCHAR(250)); INSERT INTO test.messages VALUES ('hello'); EOF kubectl run mysql-client --image=mysql:5.7 -i -t --rm --restart=Never -- mysql -uroot -popenECOMP1.0 -h dbhost-read.onap-sdnc -e "SELECT * FROM test.messages"

2. To demonstrate that DBHost-read distributes service across slaves, see the ServerID changing in it's response

   Code Block
   collapse true
   kubectl run mysql-client-loop --image=mysql:5.7 -i -t --rm --restart=Never -- bash -ic "while sleep 0.5; do mysql -uroot -popenECOMP1.0 -h dbhost-read.onap-sdnc -e 'SELECT @@server_id,NOW()'; done"

3. Can scale (up or down) mysql dynamically:

   Code Block
   collapse true
   Scale up: kubectl scale statefulset sdnc-dbhost -n onap-sdnc --replicas=5   Scale Down kubectl scale statefulset sdnc-dbhost -n onap-sdnc --replicas=2