Backup Testing

Test backup functionality provided by OOM: oom/cronjob.yaml at master · onap/oom · GitHub

Container downloads mariadb image and runs some shell commands

initContainers: {{- include "common.readinessCheck.waitFor" . | nindent 12 }}
- name: mariadb-galera-backup-init
image: {{ include "repositoryGenerator.image.mariadb" . }}
imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
{{ include "common.containerSecurityContext" . | indent 14 | trim }}
command:
- /bin/bash
- -c
- |
remove_dir(){
dirToRemove=$1
rm -rf $dirToRemove
echo "Failed" > /backup/backup.log
echo "Backup failed!!!"
}

target_dir=/backup/backup-`date +%s`
mkdir -p $target_dir

mysqlhost={{ include "common.servicename" . }}.{{ include "common.namespace" . }}

mariabackup --backup --target-dir=$target_dir --user=root --password=$DB_PASS --host=$mysqlhost

ret_code=$?
if [ $ret_code -ne 0 ]; then
remove_dir $target_dir
exit 0
fi

echo "Starting Backup Preparation!!!"
mariabackup --prepare --target-dir=$target_dir
ret_code=$?
if [ $ret_code -ne 0 ]; then
remove_dir $target_dir
exit 0
fi
echo "Success" > /backup/backup.log
echo "Backup Successful!!!"

Testing

1. Check mariabackup version on localhost: mariabackup --version → mariabackup based on MariaDB server 10.3.25-MariaDB debian-linux-gnu (x86_64)
2. Start docker container using this version: docker run -it mariadb:10.3.25 bash
3. Check backup version in container:  mariabackup --version ->mariabackup based on MariaDB server 10.3.25-MariaDB debian-linux-gnu (x86_64)
4. Copy mariabackup command from the code above and replace variable with actual values: mariabackup --backup --target-dir=/tmp --user=root --password=root --host=host.docker.internal --port=3355 → 2021-05-19 11:55:10 0 [ERROR] InnoDB: File /var/lib/mysql/ib_logfile0: 'open' returned OS error 71. Cannot continue operation
   210519 11:55:10 [ERROR] mysqld got signal 6 ;
5. Try the same command locally: sudo mariabackup --backup --target-dir=/tmp --user=root --password=***** --host=localhost --port=3355 →[00] 2021-05-19 11:03:32 completed OK!
6. Create a new user on local mariadb server: create user backup identified by 'backup';
7. Grant all privileges to backup: GRANT ALL PRIVILEGES ON *.* TO 'backup'@'%'; FLUSH PRIVILEGES;
8. Re-try mariabackup from the container: mariabackup --backup --target-dir=/tmp --user=backup --password=backup --host=host.docker.internal --port=3355 → [ERROR] InnoDB: File /var/lib/mysql/ib_logfile0: 'open' returned OS error 71. Cannot continue operation
   210519 10:07:59 [ERROR] mysqld got signal 6 ;
9. Run the command locally: sudo mariabackup --backup --target-dir=/tmp --user=backup --password=backup --host=localhost --port=3355 → [00] 2021-05-19 11:10:27 completed OK!
10. Try using mysqldump in the container instead: mysqldump --all-databases --host=host.docker.internal --port=3355 --user=backup --password=backup → -- Dump completed on 2021-05-19 10:12:37
11. Try mysqldump locally: sudo mysqldump --all-databases --host=localhost --port=3355 --user=backup --password=backup → -- Dump completed on 2021-05-19 11:15:24
12. Try mysqldump in the container with the root user: mysqldump --all-databases --host=host.docker.internal --port=3355 --user=root --password=root → -- Dump completed on 2021-05-19 11:59:54
13. If we may the /var/lib/mysql directory from the localhost to the container, then mariabackup works in the container:  docker run -it -v /var/lib/mysql:/var/lib/mysql mariadb:10.3.25 bash → mariabackup --backup --target-dir=/tmp --user=root --password=root --host=host.docker.internal --port=3355 → [00] 2021-05-19 12:16:10 completed OK!
14. Also works with the backup user when the directory is mapped :  mariabackup --backup --target-dir=/tmp --user=backup --password=backup --host=host.docker.internal --port=3355 → [00] 2021-05-19 12:16:10 completed OK!

Conclusion

1. mariabackup does not work remotley  [MDEV-15075] unless the /var/lib/mysql directory from the database host can be mapped to a directory in the container.
2. mysqldump is another option and doesn't need to copy files from the file system.
3. mariabackup must be the same version as the one your database is using (the bitnami/mariadb:10.5.8 image used by OOM (mariabackup based on MariaDB server 10.5.8-MariaDB Linux (x86_64)) is not compatible with the database running on my localhost ) oom/values.yaml at master · onap/oom · GitHub
   
4. Looking in oom/values.yaml at master · onap/oom · GitHub datadir=/bitnami/mariadb/data, I don't see this directory mounted in the cronjob. Only 2 mounthPaths present mountPath: /backup and mountPath: /etc/localtime. I assume any additional directories required are being mounted by some common mechanism.
5. Work is currently being done on this project, a new stateful set template was added recently and includes references to MARIADB_GALERA_MARIABACKUP_USER and MARIADB_GALERA_MARIABACKUP_PASSWORD which are no in the current backup cronjob.