Instructions for K8S cluster setup on AWS

This document provides instructions on how to setup HA Kubernetes cluster on AWS instances using Rancher Kubernetes Engine (RKE).

This document shows how to create and configure 3 control plane VMs (each with 4 vCPUs, 16GB RAM, 80GB disk storage & Ubuntu 18.04.4) and 12 worker VMs (each with 8 vCPUs, 32GB RAM, 160GB disk storage & Ubuntu 18.04.4) through AWS and deploy with HA kubernetes cluster through RKE.

Prerequisite: Users must have AWS account credentials to login and follow the below steps in AWS EC2/VPC dashboard.

1.Create Key Pair

A Key Pair is required to access the created AWS instances and will be used by RKE to configure the VMs for Kubernetes.

If key is already exists use an existing key pair and import through Import Key Pair

Go to AWS EC2 dashboard click on Key Pairs in the left panel then click on Actions to Import

To Create New Key Pair:

Go to AWS EC2 dashboard click on Key Pairs in the left panel then click on Create Key Pair

Note: Keep this downloaded key safe and copied into ~/.ssh/ from which it can be referenced.

Example:
       mv onap-key ~/.ssh
       chmod 600 ~/.ssh/onap-key

2.Create VPC

Go to AWS VPC dashboard click on Your VPCs in the left panel then click on Create VPC

3.Create Subnet

Go to AWS VPC dashboard click on Subnets in the left panel then click on Create Subnet

4.Create Internet Gateway

Go to AWS VPC dashboard click on Internet Gateways in the left panel then click on Create Internet Gateway

Note: Once IGW is created, on the top right corner you will see Attach to VPC. Click on that button to attach this IGW to your VPC as below

5.Add Routes with IGW

Go to AWS VPC dashboard click on Route Tables in the left panel then select your routing table click on Routes and Edit Routes to Add route with IGW

6.Create Security Group

Click on Create Security Group under EC2>Security Groups fill the details then click create security group

Select the created security group click on edit inbound & outbound rules

Add rules for Inbound:

Click on Add rule and fill the details then click on Save Rules

Add Rules for Outbound:

Click on Edit Outbound rule and fill the details then click save rules

7.Create Kubernetes control plane VMs

Step-1: Launch new instance from EC2 and select the image, click Next

Step-2: Choose Instance Type, click Next

Step-3: Configure Instance

Select no of instances 3, network with your created VPC, subnet IDs & Enable Auto-assign Public IP then click Next

Step-4: Add Storage:

Add disk storage as required then click Next

Step-5: Add Tags

Add Tags if needed, click Next

Step-6: Configure Security Group

Create new security group or select existing security group if already exists then click on Review and Launch

Step-7: Review and Launch

Note: While Launching select the Key Pair, check acknowledgement box and click Launch Instances

8. Apply Customization script for control plane VMs

Below is the Customization script, apply on all control plane VMs by running with “sudo <script.sh>”

#!/bin/bash

DOCKER_VERSION=18.09.5
sudo apt-get update
curl https://releases.rancher.com/install-docker/$DOCKER_VERSION.sh | sh
sudo mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/docker.conf << EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry=nexus3.onap.org:10001
EOF
sudo usermod -aG docker ubuntu
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo apt-mark hold docker-ce
IP_ADDR=`ip address |grep eth|grep inet|awk '{print $2}'| awk -F / '{print $1}'`
HOSTNAME=`hostname`
echo "$IP_ADDR $HOSTNAME" >> /etc/hosts
sudo docker login -u docker -p docker nexus3.onap.org:10001
sudo apt-get install make -y

#nfs server
sudo apt-get install nfs-kernel-server -y
sudo mkdir -p /nfs_share
sudo chown nobody:nogroup /nfs_share/
exit 0

9. Create Kubernetes Worker VMs

The number and size of Worker VMs is dependent on the size of the ONAP deployment. By default, all ONAP applications are deployed. It’s possible to customize the deployment and enable a subset of the ONAP applications. For the purpose of this guide, however, we will deploy 12 Kubernetes Workers that have been sized to handle the entire ONAP application workload.

Step-1: Launch new instance and select the required image

Step-2: Choose an Instance Type

Select required configuration and click Next

Step-3: Configure Instances

Select no of instances, network and subnet details then click Next

Step-4: Add Storage

Add required disk storage then click Next

Step-5: Add Tags

Select Add Tags if needed then click

Step-6: Configure Security Group

Create new or Select existing security group then click Review and Launch

Step-7: Review Instance Launch

Review the configuration details the click Launch

Note: While Launching select an existing key pair or create new, click acknowledgement checkbox then Launch

10. Apply Customization script for Kubernetes worker VMs

Below is the Customization script, apply on all worker VMs by running with “sudo <script.sh>”

#!/bin/bash

DOCKER_VERSION=18.09.5
sudo apt-get update
curl https://releases.rancher.com/install-docker/$DOCKER_VERSION.sh | sh
mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/docker.conf << EOF
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --insecure-registry=nexus3.onap.org:10001
EOF
sudo usermod -aG docker ubuntu
systemctl daemon-reload
systemctl restart docker
apt-mark hold docker-ce
IP_ADDR=`ip address |grep ens|grep inet|awk '{print $2}'| awk -F / '{print $1}'`
HOSTNAME=`hostname`
echo "$IP_ADDR $HOSTNAME" >> /etc/hosts
docker login -u docker -p docker nexus3.onap.org:10001
sudo apt-get install make -y
# install nfs
sudo apt-get install nfs-common -y
exit 0

11. Configure Rancher Kubernetes Engine (RKE)

Download and install RKE on a VM, desktop or laptop. Binaries can be found here for Linux and Mac:

https://github.com/rancher/rke/releases/tag/v1.0.6

Execute below once RKE installed:

RKE requires a cluster.yml as input. An example file is shown below that describes a Kubernetes cluster that will be mapped onto the AWS instances created earlier in this guide.

Below is an example of an HA Kubernetes cluster for ONAP

Prepare Cluster.yml

Before this configuration file can be used the external address and the internal_address must be mapped for each control and worker node in this file.

Run RKE:

From within the same directory as the cluster.yml file, simply execute:

12. Install Kubectl and Validate K8S cluster Deployment

Download and Install Kubectl. Binaries can be found here for Linux and Mac:

https://storage.googleapis.com/kubernetes-release/release/v1.15.11/bin/linux/amd64/kubectl
https://storage.googleapis.com/kubernetes-release/release/v1.15.11/bin/darwin/amd64/kubectl

Execute below after kubectl installation:

To Validate K8S cluster deployment, execute below:

Output will look something like this after successful Deployment: