Backup and Restore Solution: ONAP-OOM

• Problem Statement and Requirement (User Story): -

As an ONAP Operator- We require the ability to backup and restore ONAP state data, We want to have Disaster recovery solution for ONAP deployment done over K8.

Basic Use case would be: -

1) Add/Update/Modify the POD Data or DB Data.
2) Simulate a Disaster
3) Restore using Backup.
4) POD Data/DB entries should be recovered.

• Solution Description: -

Narrowed down upon a tool which can be used for K8 Backup and Restoration for ONAP deployments named as Heptio-ARK

 Ark is an Opensource tool to back up and restore your Kubernetes cluster resources and persistent volumes. Ark lets you:

• Take backups of your cluster and restore in case of loss.

• Copy cluster resources across cloud providers. NOTE: Cloud volume migrations are not yet supported.

Replicate your production environment for development and testing environments.

Ark consists of:

• A server that runs on your cluster

• A command-line client that runs locally

Working Flow diagram: -

Open

• Installation: -

Prerequisites

• Access to a Kubernetes cluster, version 1.7 or later.

• A DNS server on the cluster

• kubectl installed

• Labels should be defined there.

Script Delivered: -

https://jira.onap.org/secure/attachment/12222/oom_ark_setup.sh

• Below script will setup the ARK server and Client as well, It is using the MINIO, an S3-compatible storage service that runs locally on your cluster, but yes it gives liberty to modify according to your cloud provider

#!/bin/bash

PWD=`pwd`

ARK_VERSION=0.9.3

#Download Ark repo

git clone https://github.com/heptio/ark.git

PWD=`pwd`

ARK_VERSION=0.9.3

#Run the Pre-requistites

kubectl apply -f $PWD/ark/examples/common/00-prereqs.yaml

#Run the Ark POD deployment

kubectl apply -f $PWD/ark/examples/minio/

#Download the Client and Make it executable

cd ark

wget https://github.com/heptio/ark/releases/download/v0.9.3/ark-v${ARK_VERSION}-linux-amd64.tar.gz

sudo tar -zxvf ark-v${ARK_VERSION}-linux-amd64.tar.gz

sudo chmod +x ./ark

sudo mv ./ark /usr/local/bin/ark

exit 0

Code Delivered:-

As Labels need to be defined, because that is a unique identity which we need to have for any backup of our k8 containers,

So in OOM code, Where -ever we don't have labels, We need to define that whether its configmap or secret, for eg below:-

labels:

app: {{ include "common.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}

• Running ARK Example (Backup and Restoration with Logs): -
  
  

1) INSTALL SO COMPONENT:-

root@k8s-1:/vaibhav/backup/oom/kubernetes# helm install so -n bkup --namespace test3

NAME:   bkup

LAST DEPLOYED: Fri Jul 20 06:59:09 2018

NAMESPACE: test3

STATUS: DEPLOYED

RESOURCES:

==> v1/Pod(related)

NAME                         READY  STATUS    RESTARTS  AGE

bkup-so-db-744fccd888-w67zk  0/1    Init:0/1  0         0s

bkup-so-7668c746c-vngk8      0/2    Init:0/1  0         0s

==> v1/Secret

NAME        TYPE    DATA  AGE

bkup-so-db  Opaque  1     0s

==> v1/ConfigMap

NAME                      DATA  AGE

confd-configmap           1     0s

so-configmap              5     0s

so-docker-file-configmap  1     0s

so-filebeat-configmap     1     0s

so-log-configmap          11    0s

==> v1/PersistentVolume

NAME        CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM             STORAGECLASS  REASON  AGE

bkup-so-db  2Gi       RWX           Retain          Bound   test3/bkup-so-db  0s

==> v1/PersistentVolumeClaim

NAME        STATUS  VOLUME      CAPACITY  ACCESS MODES  STORAGECLASS  AGE

bkup-so-db  Bound   bkup-so-db  2Gi       RWX           0s

==> v1/Service

NAME   TYPE      CLUSTER-IP   EXTERNAL-IP  PORT(S)                                                                     AGE

so-db  NodePort  10.43.63.96  <none>       3306:30252/TCP                                                              0s

so     NodePort  10.43.59.93  <none>       8080:30223/TCP,3904:30225/TCP,3905:30224/TCP,9990:30222/TCP,8787:30250/TCP  0s

==> v1beta1/Deployment

NAME        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE

bkup-so-db  1        1        1           0          0s

bkup-so     1        1        1           0          0s

NOTES:

1. Get the application URL by running these commands:

  export NODE_PORT=$(kubectl get --namespace test3 -o jsonpath="{.spec.ports[0].nodePort}" services so)

  export NODE_IP=$(kubectl get nodes --namespace test3 -o jsonpath="{.items[0].status.addresses[0].address}")

  echo http://$NODE_IP:$NODE_PORT

2) CHECKING STATUS OF POD:-

root@k8s-1:/vaibhav/backup/oom/kubernetes# kubectl get pods --all-namespaces | grep -i so

NAMESPACE       NAME                                    READY     STATUS             RESTARTS   AGE

test3           bkup-so-7668c746c-vngk8                 2/2       Running            0          8m

test3           bkup-so-db-744fccd888-w67zk             1/1       Running            0          8m

root@k8s-1:/vaibhav/backup/oom/kubernetes#

3) CREATING BACKUP OF DEPLOYMENT:-

Here I am using selector label as release name

root@k8s-1:/vaibhav/backup/oom/kubernetes# ark backup create so-backup --selector release=bkup

Backup request "so-backup" submitted successfully.

Run `ark backup describe so-backup` for more details.

root@k8s-1:/vaibhav/backup/oom/kubernetes#

4) CHECKING BACKUP LOGS:-

root@k8s-1:/vaibhav/backup/oom/kubernetes# ark backup describe so-backup

Name:         so-backup

Namespace:    heptio-ark

Labels:       <none>

Annotations:  <none>

Phase:  Completed

Namespaces:

  Included:  *

  Excluded:  <none>

Resources:

  Included:        *

  Excluded:        <none>

  Cluster-scoped:  auto

Label selector:  release=bkup

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2018-07-20 07:09:51 +0000 UTC

Completed:  2018-07-20 07:09:53 +0000 UTC

Expiration:  2018-08-19 07:09:51 +0000 UTC

Validation errors:  <none>

Persistent Volumes: <none included>

5) SIMULATING A DISASTER:-

root@k8s-1:/vaibhav/backup/oom/kubernetes# helm delete --purge bkup

release "bkup" deleted

6)CREATE BACKUP OF THE PODS USING ARK :-

root@k8s-1:/vaibhav/backup/oom/kubernetes# ark restore create --from-backup so-backup

Restore request "so-backup-20180720071236" submitted successfully.

Run `ark restore describe so-backup-20180720071236` for more details.

root@k8s-1:/vaibhav/backup/oom/kubernetes#

7) CHECKING RESTORATION LOGS:-

root@k8s-1:/vaibhav/backup/oom/kubernetes# ark restore describe so-backup-20180720071236

Name:         so-backup-20180720071236

Namespace:    heptio-ark

Labels:       <none>

Annotations:  <none>

Backup:  so-backup

Namespaces:

  Included:  *

  Excluded:  <none>

Resources:

  Included:        *

  Excluded:        nodes, events, events.events.k8s.io, backups.ark.heptio.com, restores.ark.heptio.com

  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Phase:  Completed

Validation errors:  <none>

Warnings:  <none>

Errors:    <none>

8)CHECK TARBALL:-

root@k8s-1:/vaibhav/backup/resources# tree

.

??? configmaps

?   ??? namespaces

?       ??? test3

?           ??? confd-configmap.json

?           ??? so-configmap.json

?           ??? so-docker-file-configmap.json

?           ??? so-filebeat-configmap.json

?           ??? so-log-configmap.json

??? deployments.apps

?   ??? namespaces

?       ??? test3

?           ??? bkup-so-db.json

?           ??? bkup-so.json

??? endpoints

?   ??? namespaces

?       ??? test3

?           ??? so-db.json

?           ??? so.json

??? persistentvolumeclaims

?   ??? namespaces

?       ??? test3

?           ??? bkup-so-db.json

??? persistentvolumes

?   ??? cluster

?       ??? bkup-so-db.json

??? pods

?   ??? namespaces

?       ??? test3

?           ??? bkup-so-7668c746c-vngk8.json

?           ??? bkup-so-db-744fccd888-w67zk.json

??? replicasets.apps

?   ??? namespaces

?       ??? test3

?           ??? bkup-so-7668c746c.json

?           ??? bkup-so-db-744fccd888.json

??? secrets

?   ??? namespaces

?       ??? test3

?           ??? bkup-so-db.json

??? services

    ??? namespaces

        ??? test3

            ??? so-db.json

            ??? so.json

26 directories, 18 files

9) RESTORE RUN :-

root@k8s-1:/vaibhav/backup/oom/kubernetes# ark restore get

NAME                          BACKUP         STATUS      WARNINGS   ERRORS    CREATED                         SELECTOR

so-backup-20180720071236      so-backup      Completed   0          0         2018-07-20 07:12:36 +0000 UTC   <none>

10) CHECK THE POD STATUS:-

root@k8s-1:/vaibhav/backup/oom/kubernetes# kubectl get pods --all-namespaces | grep -i so

NAMESPACE       NAME                                    READY     STATUS             RESTARTS   AGE

test3           bkup-so-7668c746c-vngk8                 2/2       Running            0          8m

test3           bkup-so-db-744fccd888-w67zk             1/1       Running            0          8m

Another Example with DB and PV Backup:-

***APPC COMPONENT BACKUP and RESTORATION**

root@rancher:~/oom/kubernetes# kubectl get pods --all-namespaces | grep -i appc
onap bk-appc-0 1/2 Running 0 1m
onap bk-appc-cdt-7cd6f6d674-5thwj 1/1 Running 0 1m
onap bk-appc-db-0 2/2 Running 0 1m
onap bk-appc-dgbuilder-59895d4d69-7rp9q 1/1 Running 0 1m
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#

** CREATING DUMMY ENTRY IN DB **

root@rancher:~/oom/kubernetes# kubectl exec -it -n default bk-appc-db-0 bash
Defaulting container name to appc-db.
Use 'kubectl describe pod/bk-appc-db-0 -n onap' to see all of the containers in this pod.
root@bk-appc-db-0:/#
root@bk-appc-db-0:/#
root@bk-appc-db-0:/#
root@bk-appc-db-0:/# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 42
Server version: 5.7.23-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
mysql>
mysql>
mysql> connect mysql
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Connection id: 44
Current database: mysql

mysql>
mysql>
mysql> select * from servers;
Empty set (0.00 sec)

mysql> desc servers;
+-------------+----------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+----------+------+-----+---------+-------+
| Server_name | char(64) | NO | PRI | | |
| Host | char(64) | NO | | | |
| Db | char(64) | NO | | | |
| Username | char(64) | NO | | | |
| Password | char(64) | NO | | | |
| Port | int(4) | NO | | 0 | |
| Socket | char(64) | NO | | | |
| Wrapper | char(64) | NO | | | |
| Owner | char(64) | NO | | | |
+-------------+----------+------+-----+---------+-------+
9 rows in set (0.00 sec)

mysql> insert into servers values ("test","ab","sql","user","pwd",1234,"test","wrp","vaib");
Query OK, 1 row affected (0.03 sec)

mysql>
mysql>
mysql>
mysql> select * from servers;
+-------------+------+-----+----------+----------+------+--------+---------+-------+
| Server_name | Host | Db | Username | Password | Port | Socket | Wrapper | Owner |
+-------------+------+-----+----------+----------+------+--------+---------+-------+
| abc | ab | sql | user | pwd | 1234 | test | wrp | vaib |
+-------------+------+-----+----------+----------+------+--------+---------+-------+
1 row in set (0.00 sec)

mysql>
mysql>
mysql> exit
Bye
root@bk-appc-db-0:/#
root@bk-appc-db-0:/#
root@bk-appc-db-0:/#
root@bk-appc-db-0:/#
root@bk-appc-db-0:/#
root@bk-appc-db-0:/#
root@bk-appc-db-0:/# exi
bash: exi: command not found
root@bk-appc-db-0:/# exit
exit
command terminated with exit code 127
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes# kubectl get pods --all-namespaces | grep -i appc
onap bk-appc-0 1/2 Running 0 5m
onap bk-appc-cdt-7cd6f6d674-5thwj 1/1 Running 0 5m
onap bk-appc-db-0 2/2 Running 0 5m
onap bk-appc-dgbuilder-59895d4d69-7rp9q 1/1 Running 0 5m
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#

*** CREATING DUMMY FILE IN APPC PV ***
root@rancher:~/oom/kubernetes# kubectl exec -it -n onap bk-appc-0 bash
Defaulting container name to appc.
Use 'kubectl describe pod/bk-appc-0 -n onap' to see all of the containers in this pod.
root@bk-appc-0:/#
root@bk-appc-0:/#
root@bk-appc-0:/#
root@bk-appc-0:/#
root@bk-appc-0:/#
root@bk-appc-0:/#
root@bk-appc-0:/#
root@bk-appc-0:/#
root@bk-appc-0:/#
root@bk-appc-0:/# cd /opt/opendaylight/current/d
daexim/ data/ deploy/
root@bk-appc-0:/# cd /opt/opendaylight/current/daexim/
root@bk-appc-0:/opt/opendaylight/current/daexim# ls
root@bk-appc-0:/opt/opendaylight/current/daexim# ls
root@bk-appc-0:/opt/opendaylight/current/daexim#
root@bk-appc-0:/opt/opendaylight/current/daexim#
root@bk-appc-0:/opt/opendaylight/current/daexim# touch abc.txt
root@bk-appc-0:/opt/opendaylight/current/daexim# ls
abc.txt
root@bk-appc-0:/opt/opendaylight/current/daexim# exit
exit
root@rancher:~/oom/kubernetes# kubectl get pods --all-namespaces | grep -i appc
onap bk-appc-0 1/2 Running 0 6m
onap bk-appc-cdt-7cd6f6d674-5thwj 1/1 Running 0 6m
onap bk-appc-db-0 2/2 Running 0 6m
onap bk-appc-dgbuilder-59895d4d69-7rp9q 1/1 Running 0 6m
root@rancher:~/oom/kubernetes#

** CREATING BACKUP USING ARK **

root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes# ark backup create appc-bkup1 --selector release=bk
Backup request "appc-bkup1" submitted successfully.
Run `ark backup describe appc-bkup1` for more details.
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes# ark backup describe appc-bkup1
Name: appc-bkup1
Namespace: heptio-ark
Labels: <none>
Annotations: <none>

Phase: Completed

Namespaces:
Included: *
Excluded: <none>

Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto

Label selector: release=bk

Snapshot PVs: auto

TTL: 720h0m0s

Hooks: <none>

Backup Format Version: 1

Started: 2018-08-27 05:07:45 +0000 UTC
Completed: 2018-08-27 05:07:47 +0000 UTC

Expiration: 2018-09-26 05:07:44 +0000 UTC

Validation errors: <none>

Persistent Volumes: <none included>
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes# pwd
/root/oom/kubernetes
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes# ls
aaf appc cli config contrib dist esr LICENSE Makefile multicloud onap policy portal README.md sdc sniro-emulator uui vid
aai clamp common consul dcaegen2 dmaap helm log msb nbi oof pomba readiness robot sdnc so vfc vnfsdk
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#

** SIMULATING DISASTER BY DELETING APPC **

root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes# helm delete --purge bk
release "bk" deleted
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#
root@rancher:~/oom/kubernetes#

** RESTORATION USING ARK **

root@rancher:~/oom/kubernetes# ark restore create --from-backup appc-bkup1
Restore request "appc-bkup1-20180827052651" submitted successfully.
Run `ark restore describe appc-bkup1-20180827052651` for more details.
root@rancher:~/oom/kubernetes# ark restore describe appc-bkup1-20180827052651
Name: appc-bkup1-20180827052651
Namespace: heptio-ark
Labels: <none>
Annotations: <none>

Backup: appc-bkup1

Namespaces:
Included: *
Excluded: <none>

Resources:
Included: *
Excluded: nodes, events, events.events.k8s.io, backups.ark.heptio.com, restores.ark.heptio.com
Cluster-scoped: auto

Namespace mappings: <none>

Label selector: <none>

Restore PVs: auto

Phase: InProgress