...
Select no of instances 3, network with your created VPC, subnet IDs & Enable Auto-assign Public IP then click Next
Step-4: Add Storage:
Add disk storage as required then click Next
Step-5: Add Tags
Add Tags if needed, click Next
Step-6: Configure Security Group
Create new security group or select existing security group if already exists then click on Review and Launch
Step-7: Review and Launch
Note: While Launching select the Key Pair, check acknowledgement box and click Launch Instances
8. Apply Customization script for control plane VMs
Below is the Customization script, apply on all control plane VMs by running with “sudo <script.sh>”
| Code Block |
|---|
#!/bin/bash
DOCKER_VERSION=18.09.5
sudo apt-get update
curl https://releases.rancher.com/install-docker/$DOCKER_VERSION.sh | sh
sudo mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/docker.conf << EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry=nexus3.onap.org:10001
EOF
sudo usermod -aG docker ubuntu
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo apt-mark hold docker-ce
IP_ADDR=`ip address |grep eth|grep inet|awk '{print $2}'| awk -F / '{print $1}'`
HOSTNAME=`hostname`
echo "$IP_ADDR $HOSTNAME" >> /etc/hosts
sudo docker login -u docker -p docker nexus3.onap.org:10001
sudo apt-get install make -y
#nfs server
sudo apt-get install nfs-kernel-server -y
sudo mkdir -p /nfs_share
sudo chown nobody:nogroup /nfs_share/
exit 0
|
9. Create Kubernetes Worker VMs
The number and size of Worker VMs is dependent on the size of the ONAP deployment. By default, all ONAP applications are deployed. It’s possible to customize the deployment and enable a subset of the ONAP applications. For the purpose of this guide, however, we will deploy 12 Kubernetes Workers that have been sized to handle the entire ONAP application workload.
Step-1: Launch new instance and select the required image
Step-2: Choose an Instance Type
Select required configuration and click Next
Step-3: Configure Instances
Select no of instances, network and subnet details then click Next
Step-4: Add Storage
Add required disk storage then click Next
Step-5: Add Tags
Select Add Tags if needed then click
Step-6: Configure Security Group
Create new or Select existing security group then click Review and Launch
Step-7: Review Instance Launch
Review the configuration details the click Launch
Note: While Launching select an existing key pair or create new, click acknowledgement checkbox then Launch
10. Apply Customization script for Kubernetes worker VMs
Below is the Customization script, apply on all worker VMs by running with “sudo <script.sh>”
| Code Block |
|---|
#!/bin/bash
DOCKER_VERSION=18.09.5
sudo apt-get update
curl https://releases.rancher.com/install-docker/$DOCKER_VERSION.sh | sh
mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/docker.conf << EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry=nexus3.onap.org:10001
EOF
sudo usermod -aG docker ubuntu
systemctl daemon-reload
systemctl restart docker
apt-mark hold docker-ce
IP_ADDR=`ip address |grep ens|grep inet|awk '{print $2}'| awk -F / '{print $1}'`
HOSTNAME=`hostname`
echo "$IP_ADDR $HOSTNAME" >> /etc/hosts
docker login -u docker -p docker nexus3.onap.org:10001
sudo apt-get install make -y
# install nfs
sudo apt-get install nfs-common -y
exit 0
|
11. Configure Rancher Kubernetes Engine (RKE)
Download and install RKE on a VM, desktop or laptop. Binaries can be found here for Linux and Mac:
https://github.com/rancher/rke/releases/tag/v1.0.6
Execute below once RKE installed:
| Code Block |
|---|
mv rke_linux-amd64 rke
sudo chmod +x rke
sudo mv ./rke /usr/local/bin/rke
rke -version
|
RKE requires a cluster.yml as input. An example file is shown below that describes a Kubernetes cluster that will be mapped onto the AWS instances created earlier in this guide.
Below is an example of an HA Kubernetes cluster for ONAP
| Code Block |
|---|
nodes:
- address: 172.31.47.230
port: "22"
internal_address: 172.31.47.230
role:
- controlplane
- etcd
hostname_override: "onap-control-1"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.37.48
port: "22"
internal_address: 172.31.37.48
role:
- controlplane
- etcd
hostname_override: "onap-control-2"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.35.44
port: "22"
internal_address: 172.31.35.44
role:
- controlplane
- etcd
hostname_override: "onap-control-3"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.44.213
port: "22"
internal_address: 172.31.44.213
role:
- worker
hostname_override: "onap-wk-1"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.38.97
port: "22"
internal_address: 172.31.38.97
role:
- worker
hostname_override: "onap-wk-2"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.42.252
port: "22"
internal_address: 172.31.42.252
role:
- worker
hostname_override: "onap-wk-3"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.39.113
port: "22"
internal_address: 172.31.39.113
role:
- worker
hostname_override: "onap-wk-4"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.43.67
port: "22"
internal_address: 172.31.43.67
role:
- worker
hostname_override: "onap-wk-5"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.33.217
port: "22"
internal_address: 172.31.33.217
role:
- worker
hostname_override: "onap-wk-6"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.35.89
port: "22"
internal_address: 172.31.35.89
role:
- worker
hostname_override: "onap-wk-7"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.36.52
port: "22"
internal_address: 172.31.36.52
role:
- worker
hostname_override: "onap-wk-8"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.34.169
port: "22"
internal_address: 172.31.34.169
role:
- worker
hostname_override: "onap-wk-9"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.32.49
port: "22"
internal_address: 172.31.32.49
role:
- worker
hostname_override: "onap-wk-10"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.46.44
port: "22"
internal_address: 172.31.46.44
role:
- worker
hostname_override: "onap-wk-11"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
- address: 172.31.47.93
port: "22"
internal_address: 172.31.47.93
role:
- worker
hostname_override: "onap-wk-12"
user: ubuntu
ssh_key_path: "~/.ssh/onap_key"
services:
kube-api:
service_cluster_ip_range: 10.43.0.0/16
pod_security_policy: false
always_pull_images: false
kube-controller:
cluster_cidr: 10.42.0.0/16
service_cluster_ip_range: 10.43.0.0/16
kubelet:
cluster_domain: cluster.local
cluster_dns_server: 10.43.0.10
fail_swap_on: false
network:
plugin: canal
authentication:
strategy: x509
ssh_key_path: "~/.ssh/onap_key"
ssh_agent_auth: false
authorization:
mode: rbac
ignore_docker_version: false
kubernetes_version: "v1.15.11-rancher1-2"
private_registries:
- url: nexus3.onap.org:10001
user: docker
password: docker
is_default: true
cluster_name: "onap"
restore:
restore: false
snapshot_name: ""
|
Prepare Cluster.yml
Before this configuration file can be used the external address and the internal_address must be mapped for each control and worker node in this file.
Run RKE:
From within the same directory as the cluster.yml file, simply execute:
| Code Block |
|---|
> rke up
The output will look something like this:
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.33.217]
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.47.230]
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.33.217], try #1
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.47.230], try #1
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.39.113]
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.39.113], try #1
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.43.67]
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.43.67], try #1
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.35.89]
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.32.49]
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.35.89], try #1
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.32.49], try #1
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.37.48]
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.37.48], try #1
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.44.213]
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.44.213], try #1
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.38.97]
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.38.97], try #1
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.42.252]
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.42.252], try #1
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.35.44]
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.35.44], try #1
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.46.44]
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.46.44], try #1
INFO[0210] [cleanup] Successfully started [rke-log-cleaner] container on host [172.31.36.52]
INFO[0210] Removing container [rke-log-cleaner] on host [172.31.36.52], try #1
INFO[0210] [remove/rke-log-cleaner] Successfully removed container on host [172.31.47.230]
INFO[0210] [remove/rke-log-cleaner] Successfully removed container on host [172.31.35.89]
INFO[0210] [remove/rke-log-cleaner] Successfully removed container on host [172.31.34.169]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.32.49]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.35.44]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.39.113]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.33.217]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.43.67]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.37.48]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.42.252]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.38.97]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.46.44]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.36.52]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.44.213]
INFO[0211] [remove/rke-log-cleaner] Successfully removed container on host [172.31.47.93]
INFO[0211] [sync] Syncing nodes Labels and Taints
INFO[0215] [sync] Successfully synced nodes Labels and Taints
INFO[0215] [network] Setting up network plugin: canal
INFO[0215] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes
INFO[0215] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes
INFO[0215] [addons] Executing deploy job rke-network-plugin
INFO[0220] [addons] Setting up coredns
INFO[0220] [addons] Saving ConfigMap for addon rke-coredns-addon to Kubernetes
INFO[0220] [addons] Successfully saved ConfigMap for addon rke-coredns-addon to Kubernetes
INFO[0220] [addons] Executing deploy job rke-coredns-addon
INFO[0225] [addons] CoreDNS deployed successfully..
INFO[0225] [dns] DNS provider coredns deployed successfully
INFO[0225] [addons] Setting up Metrics Server
INFO[0225] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes
INFO[0225] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes
INFO[0225] [addons] Executing deploy job rke-metrics-addon
INFO[0230] [addons] Metrics Server deployed successfully
INFO[0230] [ingress] Setting up nginx ingress controller
INFO[0230] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes
INFO[0230] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes
INFO[0230] [addons] Executing deploy job rke-ingress-controller
INFO[0235] [ingress] ingress controller nginx deployed successfully
INFO[0235] [addons] Setting up user addons
INFO[0235] [addons] no user addons defined
INFO[0235] Finished building Kubernetes cluster successfully
|
12. Install Kubectl and Validate K8S cluster Deployment
Download and Install Kubectl. Binaries can be found here for Linux and Mac:
https://storage.googleapis.com/kubernetes-release/release/v1.15.11/bin/linux/amd64/kubectl
https://storage.googleapis.com/kubernetes-release/release/v1.15.11/bin/darwin/amd64/kubectl
Execute below after kubectl installation:
| Code Block |
|---|
chmod +x kubectl
sudo mv kubectl /usr/local/bin/
kubectl version --short --client
|
To Validate K8S cluster deployment, execute below:
| Code Block |
|---|
mkdir -p ~/.kube
sudo chown -R ubuntu kube_config_cluster.yml
cp kube_config_cluster.yml ~/.kube/config.onap
export KUBECONFIG=~/.kube/config.onap
kubectl config use-context onap
kubectl get nodes -o=wide
|
Output will look something like this after successful Deployment:
| Code Block |
|---|
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
onap-control-1 Ready controlplane,etcd 26d v1.15.11 172.31.47.230 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-control-2 Ready controlplane,etcd 26d v1.15.11 172.31.37.48 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-control-3 Ready controlplane,etcd 26d v1.15.11 172.31.35.44 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-1 Ready worker 26d v1.15.11 172.31.44.213 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-10 Ready worker 26d v1.15.11 172.31.32.49 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-11 Ready worker 26d v1.15.11 172.31.46.44 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-12 Ready worker 26d v1.15.11 172.31.47.93 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-2 Ready worker 26d v1.15.11 172.31.38.97 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-3 Ready worker 26d v1.15.11 172.31.42.252 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-4 Ready worker 26d v1.15.11 172.31.39.113 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-5 Ready worker 26d v1.15.11 172.31.43.67 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-6 Ready worker 26d v1.15.11 172.31.33.217 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-7 Ready worker 26d v1.15.11 172.31.35.89 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-8 Ready worker 26d v1.15.11 172.31.36.52 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
onap-wk-9 Ready worker 26d v1.15.11 172.31.34.169 <none> Ubuntu 18.04.4 LTS 5.3.0-1032-aws docker://18.9.5
|