3. Sharing /dockerdata-nfs folder between kubernetes nodes

You can skip step if your kubernetes deployment in on a single VM or physical system.

When setting up a kubernetes cluster, the folder /dockerdata-nfs must be shared between all the kubernetes nodes. This folder is used as a volume by the onap pods to share data and so there can only be one copy.

On this page we will attempt to do this by setting up nfs server on the kubernetes master node and then mount the exported directory on each of kubernetes’ nodes.

These instruction where written using VMs create from a ubuntu-16.04-server-cloudimg-amd64-disk1 image.

The actual /dockerdata-nfs folder will live on the kubernetes master VM.

In our environment we provision an Openstack volume to allocate more space instead of of using the VM default backing store.

This is optional and can be skipped. Create an Openstack Volume

The following script can be used to setup the NFS server on the kubernetes master VM

nfs server

sudo apt update
sudo apt  install nfs-kernel-server
mkdir ~/dockerdata-nfs
sudo mkdir -p /export/dockerdata-nfs
sudo chmod 777 /export
sudo chmod 777 /export/dockerdata-nfs

sudo vi /etc/exports
# append "/export/dockerdata-nfs *(rw,no_root_squash,no_subtree_check)"


sudo mount --bind  /home/ubuntu/dockerdata-nfs /export/dockerdata-nfs
sudo vi /etc/fstab 
# append "/home/ubuntu/dockerdata-nfs /export/dockerdata-nfs    none    bind  0  0"

sudo service nfs-kernel-server restart

Mount the Volume to Other VM Instances

Work in progress

More investigatio needed (as part of multi-nodes kubernetes cluser) as I'm having "Operation not permitted" error in sdnc-dbhost pod when deploying SDN-C cluster with the mounted /dockerdata-nfs from this instruction:

ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl logs sdnc-dbhost-3029711096-w1szw -n onap-sdnc
[Entrypoint] MySQL Docker Image 5.6.38-1.1.2
chown: changing ownership of '/var/lib/mysql/': Operation not permitted
ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ kubectl logs consul-agent-3312409084-3560z -n onap-consul
chown: /consul/config: Operation not permitted
ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$

#	Purpose	Command and Example
1	On the server of the attached VM instance
1.1	Install exportfs	if exportfs is not installed, install it with the following command: sudo apt install nfs-kernel-server Example of installing exportfs ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$ sudo apt install nfs-kernel-server Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 nfs-common python python-minimal python2.7 python2.7-minimal rpcbind Suggested packages: watchdog python-doc python-tk python2.7-doc binutils binfmt-support The following NEW packages will be installed: keyutils libnfsidmap2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libtirpc1 nfs-common nfs-kernel-server python python-minimal python2.7 python2.7-minimal rpcbind 0 upgraded, 13 newly installed, 0 to remove and 27 not upgraded. Need to get 4,383 kB of archives. After this operation, 18.5 MB of additional disk space will be used. Do you want to continue? [Y/n] Y … Replacing config file /etc/exports with new version Creating config file /etc/default/nfs-kernel-server with new version Processing triggers for libc-bin (2.23-0ubuntu9) ... Processing triggers for systemd (229-4ubuntu19) ... Processing triggers for ureadahead (0.100.0-19) ... ubuntu@sdnc-k8s:~/oom/kubernetes/oneclick$
1.2	Modify /etc/exports file to export the /dockerdata-nfs mount point	Example of /etc/exports for /dockerdata-nfs export mount point ubuntu@sdnc-k8s:~$ more /etc/exports # /etc/exports: the access control list for filesystems which may be exported # to NFS clients. See exports(5). # # Example for NFSv2 and NFSv3: # /srv/homes hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check) # # Example for NFSv4: # /srv/nfs4 gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check) # /srv/nfs4/homes gss/krb5i(rw,sync,no_subtree_check) # */dockerdata-nfs sdnc-k8s-2(rw,sync,no_subtree_check)*
1.3	Export the /dockerdata-nfs mount point	sudo exportfs -rav Example of exportfs command output exporting sdnc-k8s:/dockerdata-nfs
2	On the server of the other VM instance
2.1	Install nfs-common	sudo apt install nfs-common
2.2	Mount the volume as the /dockerdata-nfs directory	sudo mkdir -p /dockerdata-nfs sudo mount <mount point server IP>:/dockerdata-nfs /dockerdata-nfs
2.3	Validate the mount	Example of using df command to validate ubuntu@sdnc-k8s-2:~$ df Filesystem 1K-blocks Used Available Use% Mounted on udev 8209144 0 8209144 0% /dev tmpfs 1643244 9804 1633440 1% /run /dev/vda1 20263528 9778736 10468408 49% / tmpfs 8216216 2460 8213756 1% /dev/shm tmpfs 5120 0 5120 0% /run/lock tmpfs 8216216 0 8216216 0% /sys/fs/cgroup tmpfs 1643244 0 1643244 0% /run/user/1000 *10.147.132.10:/dockerdata-nfs 103081984 61440 97761280 1% /dockerdata-nfs*

Use the lazy (-l) option to force unmount the mount point.

For example,

sudo umount -l /dockerdata-nfs

3. Sharing /dockerdata-nfs folder between kubernetes nodes

Create NFS Server and export /dockerdata-nfs on Kubernetes master VM

On the server of the attached VM instance

Install exportfs

Modify /etc/exports file to export the /dockerdata-nfs mount point

Export the /dockerdata-nfs mount point

On the server of the other VM instance

Install nfs-common

Mount the volume as the /dockerdata-nfs directory

Validate the mount

Tips

unmount