Backup Testing

Owned by Kevin Timoney
Last updated: Jul 05, 2021
2 min read

Test backup functionality provided by OOM: oom/cronjob.yaml at master · onap/oom · GitHub



Container downloads mariadb image and runs some shell commands



OOM Cronjob
initContainers: {{- include "common.readinessCheck.waitFor" . | nindent 12 }} - name: mariadb-galera-backup-init image: {{ include "repositoryGenerator.image.mariadb" . }} imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }} {{ include "common.containerSecurityContext" . | indent 14 | trim }} command: - /bin/bash - -c - | remove_dir(){ dirToRemove=$1 rm -rf $dirToRemove echo "Failed" > /backup/backup.log echo "Backup failed!!!" } target_dir=/backup/backup-`date +%s` mkdir -p $target_dir mysqlhost={{ include "common.servicename" . }}.{{ include "common.namespace" . }} mariabackup --backup --target-dir=$target_dir --user=root --password=$DB_PASS --host=$mysqlhost ret_code=$? if [ $ret_code -ne 0 ]; then remove_dir $target_dir exit 0 fi echo "Starting Backup Preparation!!!" mariabackup --prepare --target-dir=$target_dir ret_code=$? if [ $ret_code -ne 0 ]; then remove_dir $target_dir exit 0 fi echo "Success" > /backup/backup.log echo "Backup Successful!!!"



Testing

  1. Check mariabackup version on localhost: mariabackup --version → mariabackup based on MariaDB server 10.3.25-MariaDB debian-linux-gnu (x86_64)

  2. Start docker container using this version: docker run -it mariadb:10.3.25 bash

  3. Check backup version in container:  mariabackup --version ->mariabackup based on MariaDB server 10.3.25-MariaDB debian-linux-gnu (x86_64)

  4. Copy mariabackup command from the code above and replace variable with actual values: mariabackup --backup --target-dir=/tmp --user=root --password=root --host=host.docker.internal --port=3355 → 2021-05-19 11:55:10 0 [ERROR] InnoDB: File /var/lib/mysql/ib_logfile0: 'open' returned OS error 71. Cannot continue operation
    210519 11:55:10 [ERROR] mysqld got signal 6 ;

  5. Try the same command locally: sudo mariabackup --backup --target-dir=/tmp --user=root --password=***** --host=localhost --port=3355 →[00] 2021-05-19 11:03:32 completed OK!

  6. Create a new user on local mariadb server: create user backup identified by 'backup';

  7. Grant all privileges to backup: GRANT ALL PRIVILEGES ON *.* TO 'backup'@'%'; FLUSH PRIVILEGES;

  8. Re-try mariabackup from the container: mariabackup --backup --target-dir=/tmp --user=backup --password=backup --host=host.docker.internal --port=3355 → [ERROR] InnoDB: File /var/lib/mysql/ib_logfile0: 'open' returned OS error 71. Cannot continue operation
    210519 10:07:59 [ERROR] mysqld got signal 6 ;

  9. Run the command locally: sudo mariabackup --backup --target-dir=/tmp --user=backup --password=backup --host=localhost --port=3355 → [00] 2021-05-19 11:10:27 completed OK!

  10. Try using mysqldump in the container instead: mysqldump --all-databases --host=host.docker.internal --port=3355 --user=backup --password=backup → -- Dump completed on 2021-05-19 10:12:37

  11. Try mysqldump locally: sudo mysqldump --all-databases --host=localhost --port=3355 --user=backup --password=backup → -- Dump completed on 2021-05-19 11:15:24

  12. Try mysqldump in the container with the root user: mysqldump --all-databases --host=host.docker.internal --port=3355 --user=root --password=root → -- Dump completed on 2021-05-19 11:59:54

  13. If we map the /var/lib/mysql directory from the localhost to the container, then mariabackup works in the container:  docker run -it -v /var/lib/mysql:/var/lib/mysql mariadb:10.3.25 bash → mariabackup --backup --target-dir=/tmp --user=root --password=root --host=host.docker.internal --port=3355 → [00] 2021-05-19 12:16:10 completed OK!

  14. Also works with the backup user when the directory is mapped :  mariabackup --backup --target-dir=/tmp --user=backup --password=backup --host=host.docker.internal --port=3355 → [00] 2021-05-19 12:16:10 completed OK!



Conclusion

  1. mariabackup does not work remotley  [MDEV-15075] unless the /var/lib/mysql directory from the database host can be mapped to a directory in the container.

  2. mysqldump is another option and doesn't need to copy files from the file system.

  3. mariabackup must be the same version as the one your database is using (the bitnami/mariadb:10.5.8 image used by OOM (mariabackup based on MariaDB server 10.5.8-MariaDB Linux (x86_64)) is not compatible with the database running on my localhost ) oom/values.yaml at master · onap/oom · GitHub

  4. Looking in oom/values.yaml at master · onap/oom · GitHub datadir=/bitnami/mariadb/data, I don't see this directory mounted in the cronjob. Only 2 mounthPaths present mountPath: /backup and mountPath: /etc/localtime. I assume any additional directories required are being mounted by some common mechanism.

  5. Work is currently being done on this project, a new stateful set template was added recently and includes references to MARIADB_GALERA_MARIABACKUP_USER and MARIADB_GALERA_MARIABACKUP_PASSWORD which are no in the current backup cronjob.