Table of Contents

Tracking

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	OOM-1598

WIP - as of 20190225

NON-HA version in https://git.onap.org/oom/tree/kubernetes/contrib/tools/rke/rke_setup.sh

https://gerrit.onap.org/r/#/c/79067/

Move to https://onap.readthedocs.io/en/beijing/submodules/oom.git/docs/oom_cloud_setup_guide.html or similar when this documentation is released

https://lists.onap.org/g/onap-discuss/topic/rke_ha_collaboration_for/31313969?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,31313969

Automated Installation Video of RKE install

20190227 - VMware

View file

name	20190228_rke_install_preliminary.mp4
height	250

Quickstart

Get your public and private keys on the Ubuntu 16.04 VM.

Adjust authorized_keys with your public key if not already.

20190330 - AWS

View file

name	20190330_rke_install_nso_blue_zoom_0_5_min.mp4
height	250

Quickstart

Get your public and private keys on the Ubuntu 16.04 VM.

Adjust authorized_keys with your public key if not already - aws has it, openstack may not

get rke script from jira, gerrit or by cloning OOM when the review is done.

Code Block

theme	Midnight

sudo ./rke_install.sh -b master -s localhost -e onap -l ubuntu

Versions

Currently Docker 18.06, RKE 0.1.16, Kubernetes 1.11.6, Kubectl 1.11.6, Helm 2.12.3

TODO: verify later versions of helm and a way to get RKE to install Kubernetes 1.13

Prerequisites

Ubuntu 16.04 VM

Determine RKE and Docker versions

Don't just use the latest docker version - check the RKE release page to get the version pair - 0.1.15/17.03 and 0.1.16/18.06 - see https://github.com/docker/docker-ce/releases - currently https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce

Code Block

theme	Midnight

ubuntu@a-rke:~$ sudo curl https://releases.rancher.com/install-docker/18.06.sh | sh
ubuntu@a-rke:~$ sudo usermod -aG docker ubuntu
ubuntu@a-rke:~$ sudo docker version
Client:
 Version:           18.06.3-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        d7080c1
 Built:             Wed Feb 20 02:27:18 2019

# install RKE
sudo wget https://github.com/rancher/rke/releases/download/v0.1.16/rke_linux-amd64
mv rke_linux-amd64 rke
sudo mv ./rke /usr/local/bin/rke

ubuntu@a-rke:~$ rke --version
rke version v0.1.16

...

scp your public key to the box - ideally to ~/.ssh and chmod 400 it - make sure you add your key to authorized_keys

Elastic Reserved IP

get a VIP or EIP and assign this to your VM

generate cluster.yml - optional

cluster.yml will generated by the script rke_setup.sh

Code Block

theme	Midnight

azure config - no need to hand build the yml
Watch the path of your 2 keys
Also don't add an "addon" until you have one of the config job will fail

{noformat}
ubuntu@a-rke:~$ rke config --name cluster.yml
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/onap_rsa
[+] Number of Hosts [1]: 
[+] SSH Address of host (1) [none]: rke.onap.cloud
[+] SSH Port of host (1) [22]: 
[+] SSH Private Key Path of host (rke.onap.cloud) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (rke.onap.cloud) [ubuntu]: 
[+] Is host (rke.onap.cloud) a Control Plane host (y/n)? [y]: y
[+] Is host (rke.onap.cloud) a Worker host (y/n)? [n]: y
[+] Is host (rke.onap.cloud) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (rke.onap.cloud) [none]: 
[+] Internal IP of host (rke.onap.cloud) [none]: 
[+] Docker socket path on host (rke.onap.cloud) [/var/run/docker.sock]: 
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]: 
[+] Authentication Strategy [x509]: 
[+] Authorization Mode (rbac, none) [rbac]: 
[+] Kubernetes Docker image [rancher/hyperkube:v1.11.6-rancher1]: 
[+] Cluster domain [cluster.local]: 
[+] Service Cluster IP Range [10.43.0.0/16]: 
[+] Enable PodSecurityPolicy [n]: 
[+] Cluster Network CIDR [10.42.0.0/16]: 
[+] Cluster DNS Service IP [10.43.0.10]: 
[+] Add addon manifest URLs or YAML files [no]: no
ubuntu@a-rke:~$ sudo cat cluster.yml 
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: rke.onap.cloud
  port: "22"
  internal_address: ""
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: ""
  user: ubuntu
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/onap_rsa
  labels: {}
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    snapshot: null
    retention: ""
    creation: ""
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: canal
  options: {}
authentication:
  strategy: x509
  options: {}
  sans: []
system_images:
  etcd: rancher/coreos-etcd:v3.2.18
  alpine: rancher/rke-tools:v0.1.15
  nginx_proxy: rancher/rke-tools:v0.1.15
  cert_downloader: rancher/rke-tools:v0.1.15
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.15
  kubedns: rancher/k8s-dns-kube-dns-amd64:1.14.10
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny-amd64:1.14.10
  kubedns_sidecar: rancher/k8s-dns-sidecar-amd64:1.14.10
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler-amd64:1.0.0
  kubernetes: rancher/hyperkube:v1.11.6-rancher1
  flannel: rancher/coreos-flannel:v0.10.0
  flannel_cni: rancher/coreos-flannel-cni:v0.3.0
  calico_node: rancher/calico-node:v3.1.3
  calico_cni: rancher/calico-cni:v3.1.3
  calico_controllers: ""
  calico_ctl: rancher/calico-ctl:v2.0.0
  canal_node: rancher/calico-node:v3.1.3
  canal_cni: rancher/calico-cni:v3.1.3
  canal_flannel: rancher/coreos-flannel:v0.10.0
  wave_node: weaveworks/weave-kube:2.1.2
  weave_cni: weaveworks/weave-npc:2.1.2
  pod_infra_container: rancher/pause-amd64:3.1
  ingress: rancher/nginx-ingress-controller:0.16.2-rancher1
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.4
  metrics_server: rancher/metrics-server-amd64:v0.2.1
ssh_key_path: ~/.ssh/onap_rsa
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
  provider: ""
  options: {}
  node_selector: {}
  extra_args: {}
cluster_name: ""
cloud_provider:
  name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""
monitoring:
  provider: ""
  options: {}
{noformat}

Kubernetes Single Node Developer Installation

Code Block

theme	Midnight

sudo ./rke_install.sh -b master -s localhost -e onap -l ubuntu

Kubernetes HA Cluster Production Installation

Design Issues

DI 20190225-1: RKE/Docker version pair

As of 20190215 RKE 0.16 supports Docker 18.06-ce (and 18.09 non-ce) (up from 0.15 supporting 17.03)

https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce

https://github.com/rancher/rke/releases/tag/v0.1.16

Code Block

theme	Midnight

ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster                  
INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] 
FATA[0000] Unsupported Docker version found [18.06.3-ce], supported versions are [1.11.x 1.12.x 1.13.x 17.03.x]

DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working

Does rke remove, regenerate the yaml (or hand upgrade the versions) then rke up

Code Block

theme	Midnight

ubuntu@a-rke:~$ sudo rke remove
Are you sure you want to remove Kubernetes cluster [y/n]: y
INFO[0002] Tearing down Kubernetes cluster              
INFO[0002] [dialer] Setup tunnel for host [rke.onap.cloud] 
INFO[0002] [worker] Tearing down Worker Plane..         
INFO[0002] [remove/kubelet] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/kube-proxy] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [worker] Successfully tore down Worker Plane.. 
INFO[0003] [controlplane] Tearing down the Controller Plane.. 
INFO[0003] [remove/kube-apiserver] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/kube-controller-manager] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [remove/kube-scheduler] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [controlplane] Host [rke.onap.cloud] is already a worker host, skipping delete kubelet and kubeproxy. 
INFO[0004] [controlplane] Successfully tore down Controller Plane.. 
INFO[0004] [etcd] Tearing down etcd plane..# on your laptop/where your cert is
# chmod 777 your cert before you scp it over
obrienbiometrics:full michaelobrien$ scp ~/wse_onap/onap_rsa ubuntu@rke0.onap.info:~/

# on the host
sudo cp onap_rsa ~/.ssh
sudo chmod 400 ~/.ssh/onap_rsa
sudo chown ubuntu:ubuntu ~/.ssh/onap_rsa 
# just verify
sudo vi ~/.ssh/authorized_keys

git clone --recurse-submodules https://gerrit.onap.org/r/oom
sudo cp oom/kubernetes/contrib/tools/rke/rke_setup.sh .
sudo nohup ./rke_setup.sh -b master -s 104.209.161.210 -e onap -k onap_rsa -l ubuntu &

ubuntu@a-rke0-master:~$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                      READY     STATUS      RESTARTS   AGE
ingress-nginx   default-http-backend-797c5bc547-55fpn     1/1       Running     0          4m
ingress-nginx   nginx-ingress-controller-znhgz            1/1       Running     0          4m
kube-system     canal-dqt2m                               3/3       Running     0          5m
kube-system     kube-dns-7588d5b5f5-pzdfh                 3/3       Running     0          5m
kube-system     kube-dns-autoscaler-5db9bbb766-b7vvg      1/1       Running     0          5m
kube-system     metrics-server-97bc649d5-fmqjd            1/1       Running     0          4m
kube-system     rke-ingress-controller-deploy-job-dxmbd   0/1       Completed   0          4m
kube-system     rke-kubedns-addon-deploy-job-wqccp        0/1       Completed   0          5m
kube-system     rke-metrics-addon-deploy-job-ssrgp        0/1       Completed   0          4m
kube-system     rke-network-plugin-deploy-job-jkffq       0/1       Completed   0          5m
kube-system     tiller-deploy-759cb9df9-rlt7v             1/1       Running     0          2m
ubuntu@a-rke0-master:~$ helm list

Versions

Currently Docker 18.06, RKE 0.1.16, Kubernetes 1.11.6, Kubectl 1.11.6, Helm 2.12.3

TODO: verify later versions of helm and a way to get RKE to install Kubernetes 1.13

Prerequisites

Ubuntu 16.04 VM

Determine RKE and Docker versions

Don't just use the latest docker version - check the RKE release page to get the version pair - 0.1.15/17.03 and 0.1.16/18.06 - see https://github.com/docker/docker-ce/releases - currently https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce

Code Block

theme	Midnight

ubuntu@a-rke:~$ sudo curl https://releases.rancher.com/install-docker/18.06.sh | sh
ubuntu@a-rke:~$ sudo usermod -aG docker ubuntu
ubuntu@a-rke:~$ sudo docker version
Client:
 Version:           18.06.3-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        d7080c1
 Built:             Wed Feb 20 02:27:18 2019

# install RKE
sudo wget https://github.com/rancher/rke/releases/download/v0.1.16/rke_linux-amd64
mv rke_linux-amd64 rke
sudo mv ./rke /usr/local/bin/rke

ubuntu@a-rke:~$ rke --version
rke version v0.1.16

Private SSH key

scp your public key to the box - ideally to ~/.ssh and chmod 400 it - make sure you add your key to authorized_keys

Elastic Reserved IP

get a VIP or EIP and assign this to your VM

generate cluster.yml - optional

cluster.yml will generated by the script rke_setup.sh

Code Block

theme	Midnight

azure config - no need to hand build the yml
Watch the path of your 2 keys
Also don't add an "addon" until you have one of the config job will fail

{noformat}
ubuntu@a-rke:~$ rke config --name cluster.yml
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/onap_rsa
[+] Number of Hosts [1]: 
[+] SSH Address of host (1) [none]: rke.onap.cloud
[+] SSH Port of host (1) [22]: 
[+] SSH Private Key Path of host (rke.onap.cloud) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (rke.onap.cloud) [ubuntu]: 
[+] Is host (rke.onap.cloud) a Control Plane host (y/n)? [y]: y
[+] Is host (rke.onap.cloud) a Worker host (y/n)? [n]: y
[+] Is host (rke.onap.cloud) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (rke.onap.cloud) [none]: 
[+] Internal IP of host (rke.onap.cloud) [none]: 
[+] Docker socket path on host (rke.onap.cloud) [/var/run/docker.sock]: 
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]: 
[+] Authentication Strategy [x509]: 
[+] Authorization Mode (rbac, none) [rbac]: 
[+] Kubernetes Docker image [rancher/hyperkube:v1.11.6-rancher1]: 
[+] Cluster domain [cluster.local]: 
[+] Service Cluster IP Range [10.43.0.0/16]: 
[+] Enable PodSecurityPolicy [n]: 
[+] Cluster Network CIDR [10.42.0.0/16]: 
[+] Cluster DNS Service IP [10.43.0.10]: 
[+] Add addon manifest URLs or YAML files [no]: no
ubuntu@a-rke:~$ sudo cat cluster.yml 
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: rke.onap.cloud
  port: "22"
  internal_address: ""
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: ""
  user: ubuntu
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/onap_rsa
  labels: {}
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    snapshot: null
    retention: ""
    creation: ""
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: canal
  options: {}
authentication:
  strategy: x509
  options: {}
  sans: []
system_images:
  etcd: rancher/coreos-etcd:v3.2.18
  alpine: rancher/rke-tools:v0.1.15
  nginx_proxy: rancher/rke-tools:v0.1.15
  cert_downloader: rancher/rke-tools:v0.1.15
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.15
  kubedns: rancher/k8s-dns-kube-dns-amd64:1.14.10
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny-amd64:1.14.10
  kubedns_sidecar: rancher/k8s-dns-sidecar-amd64:1.14.10
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler-amd64:1.0.0
  kubernetes: rancher/hyperkube:v1.11.6-rancher1
  flannel: rancher/coreos-flannel:v0.10.0
  flannel_cni: rancher/coreos-flannel-cni:v0.3.0
  calico_node: rancher/calico-node:v3.1.3
  calico_cni: rancher/calico-cni:v3.1.3
  calico_controllers: ""
  calico_ctl: rancher/calico-ctl:v2.0.0
  canal_node: rancher/calico-node:v3.1.3
  canal_cni: rancher/calico-cni:v3.1.3
  canal_flannel: rancher/coreos-flannel:v0.10.0
  wave_node: weaveworks/weave-kube:2.1.2
  weave_cni: weaveworks/weave-npc:2.1.2
  pod_infra_container: rancher/pause-amd64:3.1
  ingress: rancher/nginx-ingress-controller:0.16.2-rancher1
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.4
  metrics_server: rancher/metrics-server-amd64:v0.2.1
ssh_key_path: ~/.ssh/onap_rsa
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
  provider: ""
  options: {}
  node_selector: {}
  extra_args: {}
cluster_name: ""
cloud_provider:
  name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""
monitoring:
  provider: ""
  options: {}
{noformat}

Kubernetes Single Node Developer Installation

Code Block

theme	Midnight

sudo ./rke_install.sh -b master -s localhost -e onap -l ubuntu

Kubernetes HA Cluster Production Installation

Design Issues

DI 20190225-1: RKE/Docker version pair

As of 20190215 RKE 0.16 supports Docker 18.06-ce (and 18.09 non-ce) (up from 0.15 supporting 17.03)

https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce

https://github.com/rancher/rke/releases/tag/v0.1.16

Code Block

theme	Midnight

ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster                  
INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] 
FATA[0000] Unsupported Docker version found [18.06.3-ce], supported versions are [1.11.x 1.12.x 1.13.x 17.03.x]

DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working

Does rke remove, regenerate the yaml (or hand upgrade the versions) then rke up

Code Block

theme	Midnight

ubuntu@a-rke:~$ sudo rke remove
Are you sure you want to remove Kubernetes cluster [y/n]: y
INFO[0002] Tearing down Kubernetes cluster              
INFO[0002] [dialer] Setup tunnel for host [rke.onap.cloud] 
INFO[0002] [worker] Tearing down Worker Plane..         
INFO[0002] [remove/kubelet] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/kube-proxy] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [worker] Successfully tore down Worker Plane.. 
INFO[0003] [controlplane] Tearing down the Controller Plane.. 
INFO[0003] [remove/kube-apiserver] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/kube-controller-manager] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [remove/kube-scheduler] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [controlplane] Host [rke.onap.cloud] is already a worker host, skipping delete kubelet and kubeproxy. 
INFO[0004] [controlplane] Successfully tore down Controller Plane.. 
INFO[0004] [etcd] Tearing down etcd plane..             
INFO[0004] [remove/etcd] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [etcd] Successfully tore down etcd plane..   
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0004] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0005] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0005] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0005] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0006] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0006] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0007] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0008] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0008] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0008] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0009] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0009] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0009] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0010] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0010] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0010] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0011] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0011] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0011] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0011] Removing local admin Kubeconfig: ./kube_config_cluster.yml 
INFO[0011] Local admin Kubeconfig removed successfully  
INFO[0011] Cluster removed successfully  

ubuntu@a-rke:~$ rke config --name cluster.ym
ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster                  
INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] 
INFO[0000] [network] Deploying port listener containers 
INFO[0001] [network] Successfully started [rke-etcd-port-listener] container on host [rke.onap.cloud] 
INFO[0001] [network] Successfully started [rke-cp-port-listener] container on host [rke.onap.cloud] 
INFO[0002] [network] Successfully started [rke-worker-port-listener] container on host [rke.onap.cloud] 
INFO[0002] [network] Port listener containers deployed successfully 
INFO[0002] [network] Running control plane -> etcd port checks 
INFO[0003] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0003] [network] Running control plane -> worker port checks 
INFO[0004] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0004] [network] Running workers -> control plane port checks 
INFO[0005] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0005] [network] Checking KubeAPI port Control Plane hosts 
INFO[0005] [network] Removing port listener containers  
INFO[0005] [remove/rke-etcd-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-cp-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-worker-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [network] Port listener containers removed successfully 
INFO[0006] [certificates] Attempting to recover certificates from backup on [etcd,controlPlane] hosts 
INFO[0007] [certificates] No Certificate backup found on [etcd,controlPlane] hosts 
INFO[0007] [certificates] Generating CA kubernetes certificates 
INFO[0007] [certificates] Generating Kubernetes API server certficates 
INFO[0008] [certificates] Generating Kube Controller certificates 
INFO[0008] [certificates] Generating Kube Scheduler certificates 
INFO[0008] [certificates] Generating Kube Proxy certificates 
INFO[0009] [certificates] Generating Node certificate   
INFO[0009] [certificates] Generating admin certificates and kubeconfig 
INFO[0009] [certificates] Generating etcd-rke.onap.cloud certificate and key 
INFO[0009] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0009] [certificates] Generating Kubernetes API server proxy client certificates 
INFO[0010] [certificates] Temporarily saving certs to [etcd,controlPlane] hosts 
INFO[0016] [certificates] Saved certs to [etcd,controlPlane] hosts 
INFO[0016] [reconcile] Reconciling cluster state        
INFO[0016] [reconcile] This is newly generated cluster  
INFO[0016] [certificates] Deploying kubernetes certificates to Cluster nodes 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes 
INFO[0022] Pre-pulling kubernetes images                
INFO[0022] Kubernetes images pulled successfully        
INFO[0022] [etcd] Building up etcd plane..              
INFO[0023] [etcd] Successfully started [etcd] container on host [rke.onap.cloud] 
INFO[0023] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [rke.onap.cloud] 
INFO[0028] [certificates] Successfully started [rke-bundle-cert] container on host [rke.onap.cloud] 
INFO[0029] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [rke.onap.cloud] 
INFO[0029] [etcd] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0030] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0030] [etcd] Successfully started etcd plane..     
INFO[0030] [controlplane] Building up Controller Plane.. 
INFO[0031] [controlplane] Successfully started [kube-apiserver] container on host [rke.onap.cloud] 
INFO[0031] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [rke.onap.cloud] 
INFO[0045] [healthcheck] service [kube-apiserver] on host [rke.onap.cloud] is healthy 
INFO[0046] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0046] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0047] [controlplane] Successfully started [kube-controller-manager] container on host [rke.onap.cloud] 
INFO[0047] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [rke.onap.cloud] 
INFO[0052] [healthcheck] service [kube-controller-manager] on host [rke.onap.cloud] is healthy 
INFO[0053] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0053] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0054] [controlplane] Successfully started [kube-scheduler] container on host [rke.onap.cloud] 
INFO[0054] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [rke.onap.cloud] 
INFO[0059] [healthcheck] service [kube-scheduler] on host [rke.onap.cloud] is healthy 
INFO[0060] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0060] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0060] [controlplane] Successfully started Controller Plane.. 
INFO[0060] [authz] Creating rke-job-deployer ServiceAccount 
INFO[0060] [authz] rke-job-deployer ServiceAccount created successfully 
INFO[0060] [authz] Creating system:node ClusterRoleBinding 
INFO[0060] [authz] system:node ClusterRoleBinding created successfully 
INFO[0060] [certificates] Save kubernetes certificates as secrets 
INFO[0060] [certificates] Successfully saved certificates as kubernetes secret [k8s-certs] 
INFO[0060] [state] Saving cluster state to Kubernetes   
INFO[0061] [state] Successfully Saved cluster state to Kubernetes ConfigMap: cluster-state 
INFO[0061] [state] Saving cluster state to cluster nodes 
INFO[0061] [state] Successfully started [cluster-state-deployer] container on host [rke.onap.cloud] 
INFO[0062] [remove/cluster-state-deployer] Successfully removed container on host [rke.onap.cloud] 
INFO[0062] [worker] Building up Worker Plane..          
INFO[0062] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] 
INFO[0063] [worker] Successfully started [kubelet] container on host [rke.onap.cloud] 
INFO[0063] [healthcheck] Start Healthcheck on service [kubelet] on host [rke.onap.cloud] 
INFO[0068] [healthcheck] service [kubelet] on host [rke.onap.cloud] is healthy 
INFO[0069] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0070] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0070] [worker] Successfully started [kube-proxy] container on host [rke.onap.cloud] 
INFO[0070] [healthcheck] Start Healthcheck on service [kube-proxy] on host [rke.onap.cloud] 
INFO[0076] [healthcheck] service [kube-proxy] on host [rke.onap.cloud] is healthy 
INFO[0076] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0077] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0077] [worker] Successfully started Worker Plane.. 
INFO[0077] [sync] Syncing nodes Labels and Taints       
INFO[0077] [sync] Successfully synced nodes Labels and Taints 
INFO[0077] [network] Setting up network plugin: canal   
INFO[0077] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0077] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-network-plugin 
INFO[0077] [addons] Executing deploy job..              
INFO[0082] [addons] Setting up KubeDNS                  
INFO[0082] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0082] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-kubedns-addon 
INFO[0082] [addons] Executing deploy job..              
INFO[0087] [addons] KubeDNS deployed successfully..     
INFO[0087] [addons] Setting up Metrics Server           
INFO[0087] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0087] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-metrics-addon 
INFO[0087] [addons] Executing deploy job..              
INFO[0092] [addons] KubeDNS deployed successfully..     
INFO[0092] [ingress] Setting up nginx ingress controller 
INFO[0092] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0092] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-ingress-controller 
INFO[0092] [addons] Executing deploy job..              
INFO[0097] [ingress] ingress controller nginx is successfully deployed 
INFO[0097] [addons] Setting up user addons              
INFO[0097] [addons] Checking for included user addons   
WARN[0097] [addons] Unable to determine if  is a file path or url, skipping 
INFO[0097] [addons] Deploying rke-user-includes-addons  
INFO[0097] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0097] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-user-includes-addons 
INFO[0097] [addons] Executing deploy job..              
WARN[0128] Failed to deploy addon execute job [rke-user-includes-addons]: Failed to get job complete status: <nil> 
INFO[0128] Finished building Kubernetes cluster successfully 

ubuntu@a-rke:~$ sudo docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS               NAMES
ec26c4bd24b5        846921f0fe0e                         "/server"                10 minutes ago      Up 10 minutes                           k8s_default-http-backend_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
f8d5db205e14        8a7739f672b4                         "/sidecar --v=2 --lo…"   10 minutes ago      Up 10 minutes                           k8s_sidecar_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
490461545ae4        rancher/metrics-server-amd64         "/metrics-server --s…"   10 minutes ago      Up 10 minutes                           k8s_metrics-server_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
aaf03b62bd41        6816817d9dce                         "/dnsmasq-nanny -v=2…"   10 minutes ago      Up 10 minutes                           k8s_dnsmasq_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
58ec007db72f        55ffe31ac578                         "/kube-dns --domain=…"   10 minutes ago      Up 10 minutes                           k8s_kubedns_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
0a95c06f6aa6        e183460c484d                         "/cluster-proportion…"   10 minutes ago      Up 10 minutes                           k8s_autoscaler_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
968a7c99b210        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
69969b331e49        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
baa5f03c16ff        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
82b2a9f640cb        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
953a4d4be0c1        df4469c42185                         "/usr/bin/dumb-init …"   10 minutes ago      Up 10 minutes                           k8s_nginx-ingress-controller_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
cce552840749        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
baa65f9c6f97        f0fad859c909                         "/opt/bin/flanneld -…"   10 minutes ago      Up 10 minutes                           k8s_kube-flannel_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1736ce68f41a        9f355e076ea7                         "/install-cni.sh"        10 minutes ago      Up 10 minutes                           k8s_install-cni_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
615d3f702ee7        7eca10056c8e                         "start_runit"            10 minutes ago      Up 10 minutes                           k8s_calico-node_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1c4a702f0f18        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
0da1cada08e1        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   10 minutes ago      Up 10 minutes                           kube-proxy
57f44998f34a        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kubelet
50f424c4daec        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kube-scheduler
502d327912d9        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kube-controller-manager
9fc706bbf3a5        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kube-apiserver
2e7630c2047c        rancher/coreos-etcd:v3.2.18          "/usr/local/bin/etcd…"   11 minutes ago      Up 11 minutes                           etcd
fef566337eb6        rancher/rke-tools:v0.1.15            "/opt/rke-tools/rke-…"   26 minutes ago      Up 26 minutes                           etcd-rolling-snapshots


amdocs@obriensystemsu0:~$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                      READY     STATUS      RESTARTS   AGE
ingress-nginx   default-http-backend-797c5bc547-m8hbx     1/1       Running     0          1h
ingress-nginx   nginx-ingress-controller-2v7w7            1/1       Running     0          1h
kube-system     canal-thmfg                               3/3       Running     0          1h
kube-system     kube-dns-7588d5b5f5-j66s8                 3/3       Running     0          1h
kube-system     kube-dns-autoscaler-5db9bbb766-rg5n8      1/1       Running     0          1h
kube-system     metrics-server-97bc649d5-jd2rr            1/1       Running     0          1h
kube-system     rke-ingress-controller-deploy-job-znp9n   0/1       Completed   0          1h
kube-system     rke-kubedns-addon-deploy-job-dzxsj        0/1       Completed   0          1h
kube-system     rke-metrics-addon-deploy-job-gpm4j        0/1       Completed   0          1h
kube-system     rke-network-plugin-deploy-job-kqdds       0/1       Completed   0          1h
kube-system     tiller-deploy-69458576b-khgr5             1/1       Running     0          1h

DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user

Code Block

theme	Midnight

amdocs@obriensystemsu0:~$ sudo rke up
Segmentation fault (core dumped)

# issue was I was using ubuntu as the yml user not amdocs in this case for a particular VM

DI 20190227-1: Verify no 110 pod limit per VM

https://forums.rancher.com/t/solved-setting-max-pods/11866

Code Block

theme	Midnight

kubelet:
    image: ""
    extra_args:
      max-pods: 900

DI 20190228-1: deploy casablanca MR to RKE under K8S 1.11.6, Docker 18.06, Helm 2.12.3

Code Block

theme	Midnight

sudo git clone https://gerrit.onap.org/r/logging-analytics
sudo wget https://git.onap.org/oom/plain/kubernetes/onap/resources/environments/dev.yaml
sudo cp dev.yaml dev0.yaml
sudo vi dev0.yaml
sudo cp dev0.yaml dev1.yaml
sudo cp logging-analytics/deploy/cd.sh .
sudo ./cd.sh -b casablanca -e onap -p false nexus3.onap.org:10001 -f true -s 300 -c true -d false -w false -r false


no good for helm 2.12.3 deployment - just using 2.9.1 for now - 
Error: Chart incompatible with Tiller v2.12.3


in the casablanca branch only - flip
https://git.onap.org/oom/tree/kubernetes/onap/Chart.yaml?h=casablanca#n24
tillerVersion: "~2.9.1"

DI 20190305-1: Azure 256G VM full ONAP Testing

Code Block

theme	Midnight

obrienbiometrics:oom michaelobrien$ ssh ubuntu@onap-dmz.onap.cloud

./oom_deployment.sh -b master -s rke.onap.cloud -e onap -r a_rke0_master -t _arm_deploy_onap_cd.json -p _arm_deploy_onap_rke_z_parameters.json

DI 20190425: HA RKE install Testing

Manual first for RC0, later retrofit the script in https://git.onap.org/oom/tree/kubernetes/contrib/tools/rke/rke_setup.sh and move/adjust the heat template in https://git.onap.org/logging-analytics/tree/deploy/heat/logging_openstack_13_16g.yaml

Installing on 6 nodes on AWS (windriver is having an issue right now).

We are good on RKE 0.2.1, Ubuntu 18.04 / Kubernetes/kubectl 1.13.5 / Helm 2.13.1 / Docker 18.09.5

https://github.com/rancher/rke/releases RKE 0.2.2 has experimental k8s 1.14 support - running with 0.2.1 for now

Just need to test onap deployments

I'll do the NFS/EFS later before deployment

Code Block

theme	Midnight

# on all VMs (control, etcd, worker)
# move the key to all vms
scp ~/wse_onap/onap_rsa ubuntu@rke0.onap.info:~/
sudo curl https://releases.rancher.com/install-docker/18.09.sh | sh
sudo usermod -aG docker ubuntu

# nfs server

# on control/etcd nodes only
# from script
sudo wget https://github.com/rancher/rke/releases/download/v0.2.1/rke_linux-amd64
mv rke_linux-amd64 rke
sudo chmod +x rke
sudo mv ./rke /usr/local/bin/rke
# one time setup of the yaml or use the generated one
ubuntu@ip-172-31-38-182:~$ sudo rke config
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/onap_rsa
[+] Number of Hosts [1]: 6
[+] SSH Address of host (1) [none]: 3.14.102.175
[+] SSH Port of host (1) [22]: 
[+] SSH Private Key Path of host (3.14.102.175) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (3.14.102.175) [ubuntu]: ubuntu
[+] Is host (3.14.102.175) a Control Plane host (y/n)? [y]: y
[+] Is host (3.14.102.175) a Worker host (y/n)? [n]: n
[+] Is host (3.14.102.175) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (3.14.102.175) [none]: 
[+] Internal IP of host (3.14.102.175) [none]: 
[+] Docker socket path on host (3.14.102.175) [/var/run/docker.sock]: 
[+] SSH Address of host (2) [none]: 18.220.62.6
[+] SSH Port of host (2) [22]: 
[+] SSH Private Key Path of host (18.220.62.6) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.220.62.6) [ubuntu]: 
[+] Is host (18.220.62.6) a Control Plane host (y/n)? [y]: y
[+] Is host (18.220.62.6) a Worker host (y/n)? [n]: n
[+] Is host (18.220.62.6) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (18.220.62.6) [none]: 
[+] Internal IP of host (18.220.62.6) [none]: 
[+] Docker socket path on host (18.220.62.6) [/var/run/docker.sock]: 
[+] SSH Address of host (3) [none]: 18.217.96.12
[+] SSH Port of host (3) [22]: 
[+] SSH Private Key Path of host (18.217.96.12) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.217.96.12) [ubuntu]: 
[+] Is host (18.217.96.12) a Control Plane host (y/n)? [y]: y
[+] Is host (18.217.96.12) a Worker host (y/n)? [n]: n
[+] Is host (18.217.96.12) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (18.217.96.12) [none]: 
[+] Internal IP of host (18.217.96.12) [none]: 
[+] Docker socket path on host (18.217.96.12) [/var/run/docker.sock]: 
[+] SSH Address of host (4) [none]: 18.188.214.137
[+] SSH Port of host (4) [22]: 
[+] SSH Private Key Path of host (18.188.214.137) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.188.214.137) [ubuntu]: 
[+] Is host (18.188.214.137) a Control Plane host (y/n)? [y]: n
[+] Is host (18.188.214.137) a Worker host (y/n)? [n]: y
[+] Is host (18.188.214.137) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (18.188.214.137) [none]: 
[+] Internal IP of host (18.188.214.137) [none]: 
[+] Docker socket path on host (18.188.214.137) [/var/run/docker.sock]: 
[+] SSH Address of host (5) [none]: 18.220.70.253
[+] SSH Port of host (5) [22]: 
[+] SSH Private Key Path of host (18.220.70.253) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.220.70.253) [ubuntu]: 
[+] Is host (18.220.70.253) a Control Plane host (y/n)? [y]: n
[+] Is host (18.220.70.253) a Worker host (y/n)? [n]: y
[+] Is host (18.220.70.253) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (18.220.70.253) [none]: 
[+] Internal IP of host (18.220.70.253) [none]: 
[+] Docker socket path on host (18.220.70.253) [/var/run/docker.sock]: 
[+] SSH Address of host (6) [none]: 3.17.76.33
[+] SSH Port of host (6) [22]: 
[+] SSH Private Key Path of host (3.17.76.33) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (3.17.76.33) [ubuntu]: 
[+] Is host (3.17.76.33) a Control Plane host (y/n)? [y]: n
[+] Is host (3.17.76.33) a Worker host (y/n)? [n]: y
[+] Is host (3.17.76.33) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (3.17.76.33) [none]: 
[+] Internal IP of host (3.17.76.33) [none]: 
[+] Docker socket path on host (3.17.76.33) [/var/run/docker.sock]: 
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]: 
[+] Authentication Strategy [x509]: 
[+] Authorization Mode (rbac, none) [rbac]: 
[+] Kubernetes Docker image [rancher/hyperkube:v1.13.5-rancher1]: 
[+] Cluster domain [cluster.local]: 
[+] Service Cluster IP Range [10.43.0.0/16]: 
[+] Enable PodSecurityPolicy [n]: 
[+] Cluster Network CIDR [10.42.0.0/16]: 
[+] Cluster DNS Service IP [10.43.0.10]: 
[+] Add addon manifest URLs or YAML files [no]:

# new
[+] Cluster domain [cluster.local]: 

ubuntu@ip-172-31-38-182:~$ sudo rke up
INFO[0000] Initiating Kubernetes cluster                
INFO[0000] [certificates] Generating CA kubernetes certificates 
INFO[0000] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0000] [certificates] Generating Kubernetes API server certificates 
INFO[0000] [certificates] Generating Service account token key 
INFO[0000] [certificates] Generating etcd-3.14.102.175 certificate and key 
INFO[0000] [certificates] Generating etcd-18.220.62.6 certificate and key 
INFO[0001] [certificates] Generating etcd-18.217.96.12 certificate and key 
INFO[0001] [certificates] Generating Kube Controller certificates 
INFO[0001] [certificates] Generating Kube Scheduler certificates 
INFO[0001] [certificates] Generating Kube Proxy certificates 
INFO[0001] [certificates] Generating Node certificate   
INFO[0001] [certificates] Generating admin certificates and kubeconfig 
INFO[0001] [certificates] Generating Kubernetes API server proxy client certificates 
INFO[0002] Successfully Deployed state file at [./cluster.rkestate] 
INFO[0002] Building Kubernetes cluster                  
INFO[0002] [dialer] Setup tunnel for host [3.14.102.175] 
INFO[0002] [dialer] Setup tunnel for host [18.188.214.137] 
INFO[0002] [dialer] Setup tunnel for host [18.220.70.253] 
INFO[0002] [dialer] Setup tunnel for host [18.220.62.6] 
INFO[0002] [dialer] Setup tunnel for host [18.217.96.12] 
INFO[0002] [dialer] Setup tunnel for host [3.17.76.33]  
INFO[0002] [network] Deploying port listener containers 
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.220.62.6] 
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [3.14.102.175] 
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.217.96.12] 
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.220.62.6] 
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.217.96.12] 
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [3.14.102.175] 
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [18.220.62.6] 
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [18.217.96.12] 
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [3.14.102.175] 
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [18.217.96.12] 
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [18.220.62.6] 
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [3.14.102.175] 
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.188.214.137] 
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.220.70.253] 
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [3.17.76.33] 
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [3.17.76.33] 
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.220.70.253] 
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.188.214.137] 
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [3.17.76.33] 
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [18.220.70.253] 
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [18.188.214.137] 
INFO[0013] [network] Port listener containers deployed successfully 
INFO[0013] [network] Running etcd <-> etcd port checks  
INFO[0013] [network] Successfully started [rke-port-checker] container on host [18.220.62.6] 
INFO[0013] [network] Successfully started [rke-port-checker] container on host [3.14.102.175] 
INFO[0013] [network] Successfully started [rke-port-checker] container on host [18.217.96.12] 
INFO[0014] [network] Running control plane -> etcd port checks 
INFO[0014] [network] Successfully started [rke-port-checker] container on host [18.220.62.6] 
INFO[0014] [network] Successfully started [rke-port-checker] container on host [3.14.102.175] 
INFO[0014] [network] Successfully started [rke-port-checker] container on host [18.217.96.12] 
INFO[0014] [network] Running control plane -> worker port checks 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.220.62.6] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.217.96.12] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [3.14.102.175] 
INFO[0015] [network] Running workers -> control plane port checks 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [3.17.76.33] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.220.70.253] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.188.214.137] 
INFO[0016] [network] Checking KubeAPI port Control Plane hosts 
INFO[0016] [network] Removing port listener containers  
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [18.220.62.6] 
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [3.14.102.175] 
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [18.217.96.12] 
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [18.217.96.12] 
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [3.14.102.175] 
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [18.220.62.6] 
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [18.220.70.253] 
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [3.17.76.33] 
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [18.188.214.137] 
INFO[0017] [network] Port listener containers removed successfully 
INFO[0017] [certificates] Deploying kubernetes certificates to Cluster nodes 
INFO[0022] [reconcile] Rebuilding and updating local kube config 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes 
INFO[0022] [reconcile] Reconciling cluster state        
INFO[0022] [reconcile] This is newly generated cluster  
INFO[0022] Pre-pulling kubernetes images                
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [3.14.102.175] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.188.214.137] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.70.253] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.217.96.12] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [3.17.76.33] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.62.6] 
INFO[0038] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.62.6] 
INFO[0038] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.70.253] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [3.17.76.33] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.217.96.12] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.188.214.137] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [3.14.102.175] 
INFO[0039] Kubernetes images pulled successfully        
INFO[0039] [etcd] Building up etcd plane..              
INFO[0039] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [3.14.102.175] 
INFO[0041] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [3.14.102.175] 
INFO[0051] [etcd] Successfully started [etcd] container on host [3.14.102.175] 
INFO[0051] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [3.14.102.175] 
INFO[0052] [etcd] Successfully started [etcd-rolling-snapshots] container on host [3.14.102.175] 
INFO[0057] [certificates] Successfully started [rke-bundle-cert] container on host [3.14.102.175] 
INFO[0058] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [3.14.102.175] 
INFO[0058] [etcd] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0058] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0058] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.220.62.6] 
INFO[0063] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.220.62.6] 
INFO[0069] [etcd] Successfully started [etcd] container on host [18.220.62.6] 
INFO[0069] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [18.220.62.6] 
INFO[0069] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.220.62.6] 
INFO[0075] [certificates] Successfully started [rke-bundle-cert] container on host [18.220.62.6] 
INFO[0075] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.220.62.6] 
INFO[0076] [etcd] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0076] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0076] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.217.96.12] 
INFO[0078] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.217.96.12] 
INFO[0078] [etcd] Successfully started [etcd] container on host [18.217.96.12] 
INFO[0078] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [18.217.96.12] 
INFO[0078] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.217.96.12] 
INFO[0084] [certificates] Successfully started [rke-bundle-cert] container on host [18.217.96.12] 
INFO[0084] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.217.96.12] 
INFO[0085] [etcd] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0085] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0085] [etcd] Successfully started etcd plane.. Checking etcd cluster health 
INFO[0086] [controlplane] Building up Controller Plane.. 
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [18.220.62.6] 
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.220.62.6] 
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [3.14.102.175] 
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [3.14.102.175] 
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [18.217.96.12] 
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.217.96.12] 
INFO[0098] [healthcheck] service [kube-apiserver] on host [18.220.62.6] is healthy 
INFO[0099] [healthcheck] service [kube-apiserver] on host [18.217.96.12] is healthy 
INFO[0099] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0099] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0099] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0099] [healthcheck] service [kube-apiserver] on host [3.14.102.175] is healthy 
INFO[0099] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0099] [controlplane] Successfully started [kube-controller-manager] container on host [18.220.62.6] 
INFO[0099] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.220.62.6] 
INFO[0100] [controlplane] Successfully started [kube-controller-manager] container on host [18.217.96.12] 
INFO[0100] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.217.96.12] 
INFO[0100] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0100] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0100] [healthcheck] service [kube-controller-manager] on host [18.220.62.6] is healthy 
INFO[0100] [healthcheck] service [kube-controller-manager] on host [18.217.96.12] is healthy 
INFO[0100] [controlplane] Successfully started [kube-controller-manager] container on host [3.14.102.175] 
INFO[0100] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [3.14.102.175] 
INFO[0101] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0101] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0101] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0101] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0101] [controlplane] Successfully started [kube-scheduler] container on host [18.220.62.6] 
INFO[0101] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.220.62.6] 
INFO[0101] [healthcheck] service [kube-controller-manager] on host [3.14.102.175] is healthy 
INFO[0101] [controlplane] Successfully started [kube-scheduler] container on host [18.217.96.12] 
INFO[0101] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.217.96.12] 
INFO[0102] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0102] [healthcheck] service [kube-scheduler] on host [18.220.62.6] is healthy 
INFO[0102] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0102] [healthcheck] service [kube-scheduler] on host [18.217.96.12] is healthy 
INFO[0103] [controlplane] Successfully started [kube-scheduler] container on host [3.14.102.175] 
INFO[0103] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [3.14.102.175] 
INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0103] [healthcheck] service [kube-scheduler] on host [3.14.102.175] is healthy 
INFO[0104] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0104] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0104] [controlplane] Successfully started Controller Plane.. 
INFO[0104] [authz] Creating rke-job-deployer ServiceAccount 
INFO[0104] [authz] rke-job-deployer ServiceAccount created successfully 
INFO[0104] [authz] Creating system:node ClusterRoleBinding 
INFO[0104] [authz] system:node ClusterRoleBinding created successfully 
INFO[0104] Successfully Deployed state file at [./cluster.rkestate] 
INFO[0104] [state] Saving full cluster state to Kubernetes 
INFO[0104] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: cluster-state 
INFO[0104] [worker] Building up Worker Plane..          
INFO[0104] [sidekick] Sidekick container already created on host [18.220.62.6] 
INFO[0104] [sidekick] Sidekick container already created on host [3.14.102.175] 
INFO[0104] [sidekick] Sidekick container already created on host [18.217.96.12] 
INFO[0105] [worker] Successfully started [kubelet] container on host [3.14.102.175] 
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [3.14.102.175] 
INFO[0105] [worker] Successfully started [kubelet] container on host [18.220.62.6] 
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [18.220.62.6] 
INFO[0105] [worker] Successfully started [kubelet] container on host [18.217.96.12] 
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [18.217.96.12] 
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [18.220.70.253] 
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [3.17.76.33] 
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [18.188.214.137] 
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253] 
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33] 
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137] 
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33] 
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253] 
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137] 
INFO[0106] [worker] Successfully started [kubelet] container on host [3.17.76.33] 
INFO[0106] [healthcheck] Start Healthcheck on service [kubelet] on host [3.17.76.33] 
INFO[0107] [worker] Successfully started [kubelet] container on host [18.220.70.253] 
INFO[0107] [healthcheck] Start Healthcheck on service [kubelet] on host [18.220.70.253] 
INFO[0107] [worker] Successfully started [kubelet] container on host [18.188.214.137] 
INFO[0107] [healthcheck] Start Healthcheck on service [kubelet] on host [18.188.214.137] 
INFO[0111] [healthcheck] service [kubelet] on host [18.220.62.6] is healthy 
INFO[0111] [healthcheck] service [kubelet] on host [18.217.96.12] is healthy 
INFO[0111] [healthcheck] service [kubelet] on host [3.14.102.175] is healthy 
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0112] [worker] Successfully started [kube-proxy] container on host [18.220.62.6] 
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.220.62.6] 
INFO[0112] [worker] Successfully started [kube-proxy] container on host [18.217.96.12] 
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.217.96.12] 
INFO[0112] [worker] Successfully started [kube-proxy] container on host [3.14.102.175] 
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [3.14.102.175] 
INFO[0113] [healthcheck] service [kube-proxy] on host [18.220.62.6] is healthy 
INFO[0113] [healthcheck] service [kube-proxy] on host [18.217.96.12] is healthy 
INFO[0113] [healthcheck] service [kube-proxy] on host [3.14.102.175] is healthy 
INFO[0113] [healthcheck] service [kubelet] on host [18.220.70.253] is healthy 
INFO[0113] [healthcheck] service [kubelet] on host [3.17.76.33] is healthy 
INFO[0113] [healthcheck] service [kubelet] on host [18.188.214.137] is healthy 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253] 
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137] 
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33] 
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253] 
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137] 
INFO[0114] [worker] Successfully started [kube-proxy] container on host [3.17.76.33] 
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [3.17.76.33] 
INFO[0114] [worker] Successfully started [kube-proxy] container on host [18.220.70.253] 
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.220.70.253] 
INFO[0114] [worker] Successfully started [kube-proxy] container on host [18.188.214.137] 
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.188.214.137] 
INFO[0114] [healthcheck] service [kube-proxy] on host [18.220.70.253] is healthy 
INFO[0114] [healthcheck] service [kube-proxy] on host [3.17.76.33] is healthy 
INFO[0115] [healthcheck] service [kube-proxy] on host [18.188.214.137] is healthy 
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253] 
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33] 
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137] 
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253] 
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33] 
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137] 
INFO[0115] [worker] Successfully started Worker Plane.. 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.188.214.137] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [3.17.76.33] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.220.70.253] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.220.62.6] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.217.96.12] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [3.14.102.175] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [3.17.76.33] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.188.214.137] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.220.62.6] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.220.70.253] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.217.96.12] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [3.14.102.175] 
INFO[0116] [sync] Syncing nodes Labels and Taints       
INFO[0117] [sync] Successfully synced nodes Labels and Taints 
INFO[0117] [network] Setting up network plugin: canal   
INFO[0117] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes 
INFO[0117] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes 
INFO[0117] [addons] Executing deploy job rke-network-plugin 
INFO[0122] [addons] Setting up kube-dns                 
INFO[0122] [addons] Saving ConfigMap for addon rke-kube-dns-addon to Kubernetes 
INFO[0122] [addons] Successfully saved ConfigMap for addon rke-kube-dns-addon to Kubernetes 
INFO[0122] [addons] Executing deploy job rke-kube-dns-addon 
INFO[0127] [addons] kube-dns deployed successfully      
INFO[0127] [dns] DNS provider kube-dns deployed successfully 
INFO[0127] [addons] Setting up Metrics Server           
INFO[0127] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes 
INFO[0127] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes 
INFO[0127] [addons] Executing deploy job rke-metrics-addon 
INFO[0132] [addons] Metrics Server deployed successfully 
INFO[0132] [ingress] Setting up nginx ingress controller 
INFO[0132] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0132] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0132] [addons] Executing deploy job rke-ingress-controller 
INFO[0137] [ingress] ingress controller nginx deployed successfully 
INFO[0137] [addons] Setting up user addons              
INFO[0137] [addons] no user addons defined              
INFO[0137] Finished building Kubernetes cluster successfully

# finish kubectl install
sudo curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.13.5/bin/linux/amd64/kubectl
sudo chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
sudo mkdir ~/.kube

# finish helm
#https://github.com/helm/helm/releases
# there is no helm 2.12.5 - last is 2.12.3 - trying 2.13.1
wget http://storage.googleapis.com/kubernetes-helm/helm-v2.13.1-linux-amd64.tar.gz
sudo tar -zxvf helm-v2.13.1-linux-amd64.tar.gz
sudo cp kube_config_cluster.yml ~/.kube/config
sudo chmod 777 ~/.kube/config

# test
ubuntu@ip-172-31-38-182:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE       NAME                                    READY   STATUS    RESTARTS   AGE   IP               NODE             NOMINATED NODE   READINESS GATES
ingress-nginx   default-http-backend-78fccfc5d9-f6z25   1/1     Running   0          17m   10.42.5.2        18.220.70.253    <none>           <none>
ingress-nginx   nginx-ingress-controller-2zxs7          1/1     Running   0          17m   18.188.214.137   18.188.214.137   <none>           <none>
ingress-nginx   nginx-ingress-controller-6b7gs          1/1     Running   0          17m   3.17.76.33       3.17.76.33       <none>           <none>
ingress-nginx   nginx-ingress-controller-nv4qg          1/1     Running   0          17m   18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-48579                             2/2     Running   0          17m   18.220.62.6      18.220.62.6      <none>           <none>
kube-system     canal-6skkm                             2/2     Running   0          17m   18.188.214.137   18.188.214.137   <none>           <none>
kube-system     canal-9xmxv                             2/2     Running   0          17m   18.217.96.12     18.217.96.12     <none>           <none>
kube-system     canal-c582x                             2/2     Running   0          17m   18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-whbck                             2/2     Running   0          17m   3.14.102.175     3.14.102.175     <none>           <none>
kube-system     canal-xbbnh                             2/2     Running   0          17m   3.17.76.33       3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-6mcm7               3/3     Running   0          17m   10.42.3.3        3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-cd5dg               3/3     Running   0          17m   10.42.4.2        18.188.214.137   <none>           <none>
kube-system     kube-dns-autoscaler-77bc5fd84-p4zfd     1/1     Running   0          17m   10.42.3.2        3.17.76.33       <none>           <none>
kube-system     metrics-server-58bd5dd8d7-kftjn         1/1     Running   0          17m   10.42.3.4        3.17.76.33       <none>           <none>


# install tiller

ubuntu@ip-172-31-38-182:~$ kubectl -n kube-system create serviceaccount tiller
serviceaccount/tiller created
ubuntu@ip-172-31-38-182:~$ kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller created
ubuntu@ip-172-31-38-182:~$ helm init --service-account tiller
Creating /home/ubuntu/.helm 
Creating /home/ubuntu/.helm/repository 
Creating /home/ubuntu/.helm/repository/cache 
Creating /home/ubuntu/.helm/repository/local 
Creating /home/ubuntu/.helm/plugins 
Creating /home/ubuntu/.helm/starters 
Creating /home/ubuntu/.helm/cache/archive 
Creating /home/ubuntu/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /home/ubuntu/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
ubuntu@ip-172-31-38-182:~$ kubectl -n kube-system  rollout status deploy/tiller-deploy
deployment "tiller-deploy" successfully rolled out
ubuntu@ip-172-31-38-182:~$ sudo helm init --upgrade
$HELM_HOME has been configured at /home/ubuntu/.helm.

Tiller (the Helm server-side component) has been upgraded to the current version.
Happy Helming!


ubuntu@ip-172-31-38-182:~$ sudo helm version
Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
ubuntu@ip-172-31-38-182:~$ sudo helm serve &
[1] 706
ubuntu@ip-172-31-38-182:~$ Regenerating index. This may take a moment.
Now serving you on 127.0.0.1:8879

ubuntu@ip-172-31-38-182:~$ sudo helm list
ubuntu@ip-172-31-38-182:~$ sudo helm repo add local http://127.0.0.1:8879
"local" has been added to your repositories
ubuntu@ip-172-31-38-182:~$  sudo helm repo list
NAME  	URL                                             
stable	https://kubernetes-charts.storage.googleapis.com
local 	http://127.0.0.1:8879                           

ubuntu@ip-172-31-38-182:~$ kubectl get nodes -o wide
NAME             STATUS   ROLES               AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
18.188.214.137   Ready    worker              22m   v1.13.5   18.188.214.137   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
18.217.96.12     Ready    controlplane,etcd   22m   v1.13.5   18.217.96.12     <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
18.220.62.6      Ready    controlplane,etcd   22m   v1.13.5   18.220.62.6      <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
18.220.70.253    Ready    worker              22m   v1.13.5   18.220.70.253    <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
3.14.102.175     Ready    controlplane,etcd   22m   v1.13.5   3.14.102.175     <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
3.17.76.33       Ready    worker              22m   v1.13.5   3.17.76.33       <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5


# install make
sudo apt-get install make -y

# install nfs/efs
ubuntu@ip-172-31-38-182:~$ sudo apt-get install nfs-common -y
ubuntu@ip-172-31-38-182:~$ sudo mkdir /dockerdata-nfs
ubuntu@ip-172-31-38-182:~$ sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-5fd6ab26.efs.us-east-2.amazonaws.com:/ /dockerdata-nfs



# check
sudo nohup ./cd.sh -b master -e onap -p false -n nexus3.onap.org:10001 -f false -s 600 -c false -d false -w false -r false &
ubuntu@ip-172-31-38-182:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE       NAME                                                     READY   STATUS             RESTARTS   AGE     IP               NODE             NOMINATED NODE   READINESS GATES
ingress-nginx   default-http-backend-78fccfc5d9-f6z25                    1/1     Running            0          103m    10.42.5.2        18.220.70.253    <none>           <none>
ingress-nginx   nginx-ingress-controller-2zxs7                           1/1     Running            0          103m    18.188.214.137   18.188.214.137   <none>           <none>
ingress-nginx   nginx-ingress-controller-6b7gs                           1/1     Running            0          103m    3.17.76.33       3.17.76.33       <none>           <none>
ingress-nginx   nginx-ingress-controller-nv4qg                           1/1     Running            0          103m    18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-48579                                              2/2     Running            0          103m    18.220.62.6      18.220.62.6      <none>           <none>
kube-system     canal-6skkm                                              2/2     Running            0          103m    18.188.214.137   18.188.214.137   <none>           <none>
kube-system     canal-9xmxv                                              2/2     Running            0          103m    18.217.96.12     18.217.96.12     <none>           <none>
kube-system     canal-c582x                                              2/2     Running            0          103m    18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-whbck                                              2/2     Running            0          103m    3.14.102.175     3.14.102.175     <none>           <none>
kube-system     canal-xbbnh                                              2/2     Running            0          103m    3.17.76.33       3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-6mcm7                                3/3     Running            0          103m    10.42.3.3        3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-cd5dg                                3/3     Running            0          103m    10.42.4.2        18.188.214.137   <none>           <none>
kube-system     kube-dns-autoscaler-77bc5fd84-p4zfd                      1/1     Running            0          103m    10.42.3.2        3.17.76.33       <none>           <none>
kube-system     metrics-server-58bd5dd8d7-kftjn                          1/1     Running            0          103m    10.42.3.4        3.17.76.33       <none>           <none>
kube-system     tiller-deploy-5f4fc5bcc6-gc4tc                           1/1     Running            0          84m     10.42.5.3        18.220.70.253    <none>           <none>
onap            onap-aai-aai-587cb79c6d-mzpbs                            0/1     Init:0/1           1          13m     10.42.5.17       18.220.70.253    <none>           <none>
onap            onap-aai-aai-babel-8c755bcfc-kmzdm                       2/2     Running            0          29m     10.42.5.8        18.220.70.253    <none>           <none>
onap            onap-aai-aai-champ-78b9d7f68b-98tm9                      0/2     Init:0/1           2          29m     10.42.3.6        3.17.76.33       <none>           <none>
onap            onap-aai-aai-data-router-64fcfbc5bb-wkkvz               INFO[0004] [remove/etcd] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [etcd] Successfully tore down etcd plane..   
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud] 1/2     CrashLoopBackOff   9          29m     10.42.5.7       INFO[0004] [hosts] Cleaning up host [rke.onap.cloud] 18.220.70.253    <none>     INFO[0004] [hosts] Running cleaner container on host<none>
[rke.onap.cloud]onap    INFO[0005] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0005] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0005] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0006] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0006] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0007] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0008] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0008] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0008] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0009] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0009] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0009] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0010] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0010] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0010] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0011] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0011] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0011] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0011] Removing local admin Kubeconfig: ./kube_config_cluster.yml 
INFO[0011] Local admin Kubeconfig removed successfully  
INFO[0011] Cluster removed successfully  

ubuntu@a-rke:~$ rke config --name cluster.ym
ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster onap-aai-aai-elasticsearch-6dcf5d9966-j7z67              1/1     Running            0          29m     10.42.4.4        18.188.214.137   <none>           <none>
onap            onap-aai-aai-gizmo-5bddb87589-zn8pl                      2/2     Running            0          29m     10.42.4.3        18.188.214.137   <none>           <none>
onap            onap-aai-aai-graphadmin-774f9d698f-f8lwv                 0/2     Init:0/1           2          29m     10.42.5.4        18.220.70.253    <none>           <none>
onap            onap-aai-aai-graphadmin-create-db-schema-94q4l           0/1     Init:Error  INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] 0 INFO[0000] [network] Deploying port listener containers  INFO[0001] [network] Successfully started [rke-etcd-port-listener] container on host [rke.onap.cloud] 
INFO[0001] [network] Successfully started [rke-cp-port-listener] container on host [rke.onap.cloud] 
INFO[0002] [network] Successfully started [rke-worker-port-listener] container on host [rke.onap.cloud] 
INFO[0002] [network] Port listener containers deployed successfully 
INFO[0002] [network] Running control plane -> etcd port checks 
INFO[0003] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0003] [network] Running control plane -> worker port checks 
INFO[0004] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0004] [network] Running workers -> control plane port checks 
INFO[0005] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0005] [network] Checking KubeAPI port Control Plane hosts 
INFO[0005] [network] Removing port listener containers  
INFO[0005] [remove/rke-etcd-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-cp-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-worker-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [network] Port listener containers removed successfully 
INFO[0006] [certificates] Attempting to recover certificates from backup on [etcd,controlPlane] hosts 
INFO[0007] [certificates] No Certificate backup found on [etcd,controlPlane] hosts 
INFO[0007] [certificates] Generating CA kubernetes certificates 
INFO[0007] [certificates] Generating Kubernetes API server certficates 
INFO[0008] [certificates] Generating Kube Controller certificates 
INFO[0008] [certificates] Generating Kube Scheduler certificates 
INFO[0008] [certificates] Generating Kube Proxy certificates 
INFO[0009] [certificates] Generating Node certificate   
INFO[0009] [certificates] Generating admin certificates and kubeconfig 
INFO[0009] [certificates] Generating etcd-rke.onap.cloud certificate and key 
INFO[0009] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0009] [certificates] Generating Kubernetes API server proxy client certificates 
INFO[0010] [certificates] Temporarily saving certs to [etcd,controlPlane] hosts 
INFO[0016] [certificates] Saved certs to [etcd,controlPlane] hosts 
INFO[0016] [reconcile] Reconciling cluster state18m     10.42.4.16       18.188.214.137   <none>           <none>
onap            onap-aai-aai-graphadmin-create-db-schema-s42pq           0/1     Init:0/1           0          7m54s   10.42.5.20       18.220.70.253    <none>           <none>
onap            onap-aai-aai-graphadmin-create-db-schema-tsvcw           0/1     Init:Error         0          29m     10.42.4.5        18.188.214.137   <none>           <none>
onap            onap-aai-aai-modelloader-845fc684bd-r7mdw                2/2     Running            0          29m     10.42.4.6        18.188.214.137   <none>           <none>
onap            onap-aai-aai-resources-67f8dfcbdb-kz6cp            INFO[0016] [reconcile] This is newly generated cluster0/2   INFO[0016] [certificates] Deploying kubernetes certificates to Cluster nodes 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes 
INFO[0022] Pre-pulling kubernetes images         Init:0/1           2          29m     10.42.5.11       18.220.70.253    <none>         INFO[0022] Kubernetes images<none>
pulledonap successfully         INFO[0022] [etcd] Building up etcd plane.. onap-aai-aai-schema-service-6c56b45b7c-7zlfz             2/2     INFO[0023] [etcd] Successfully started [etcd] container on host [rke.onap.cloud] 
INFO[0023] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [rke.onap.cloud] 
INFO[0028] [certificates] Successfully started [rke-bundle-cert] container on host [rke.onap.cloud] 
INFO[0029] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [rke.onap.cloud] 
INFO[0029] [etcd] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0030] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0030] [etcd] Successfully started etcd plane..     
INFO[0030] [controlplane] Building up Controller Plane.. 
INFO[0031] [controlplane] Successfully started [kube-apiserver] container on host [rke.onap.cloud] 
INFO[0031] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [rke.onap.cloud] 
INFO[0045] [healthcheck] service [kube-apiserver] on host [rke.onap.cloud] is healthy 
INFO[0046] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0046] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0047] [controlplane] Successfully started [kube-controller-manager] container on host [rke.onap.cloud] 
INFO[0047] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [rke.onap.cloud] 
INFO[0052] [healthcheck] service [kube-controller-manager] on host [rke.onap.cloud] is healthy 
INFO[0053] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0053] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0054] [controlplane] Successfully started [kube-scheduler] container on host [rke.onap.cloud] 
INFO[0054] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [rke.onap.cloud] 
INFO[0059] [healthcheck] service [kube-scheduler] on host [rke.onap.cloud] is healthy 
INFO[0060] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0060] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0060] [controlplane] Successfully started Controller Plane.. 
INFO[0060] [authz] Creating rke-job-deployer ServiceAccount 
INFO[0060] [authz] rke-job-deployer ServiceAccount created successfully 
INFO[0060] [authz] Creating system:node ClusterRoleBinding 
INFO[0060] [authz] system:node ClusterRoleBinding created successfully 
INFO[0060] [certificates] Save kubernetes certificates as secrets 
INFO[0060] [certificates] Successfully saved certificates as kubernetes secret [k8s-certs] 
INFO[0060] [state] Saving cluster state to Kubernetes   
INFO[0061] [state] Successfully Saved cluster state to Kubernetes ConfigMap: cluster-state 
INFO[0061] [state] Saving cluster state to cluster nodes 
INFO[0061] [state] Successfully started [cluster-state-deployer] container on host [rke.onap.cloud] 
INFO[0062] [remove/cluster-state-deployer] Successfully removed container on host [rke.onap.cloud] 
INFO[0062] [worker] Building up Worker Plane..  Running            0          29m     10.42.3.7        3.17.76.33       <none>           <none>
onap            onap-aai-aai-search-data-5d8d7759b8-flxwj                2/2     Running            0          29m     10.42.3.9        3.17.76.33       <none>           <none>
onap            onap-aai-aai-sparky-be-8444df749c-mzc2n                  0/2     Init:0/1           0          29m     10.42.4.11       18.188.214.137   <none>           <none>
onap            onap-aai-aai-spike-54ff77787f-d678x                      2/2     Running            0          29m     10.42.5.6        18.220.70.253    <none>           <none>
onap            onap-aai-aai-traversal-6ff868f477-lzv2f                  0/2     Init:0/1         INFO[0062] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] 
INFO[0063] [worker] Successfully started [kubelet] container on host [rke.onap.cloud] 
INFO[0063] [healthcheck] Start Healthcheck on service [kubelet] on host [rke.onap.cloud] 
INFO[0068] [healthcheck] service [kubelet] on host [rke.onap.cloud] is healthy 
INFO[0069] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0070] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0070] [worker] Successfully started [kube-proxy] container on host [rke.onap.cloud] 
INFO[0070] [healthcheck] Start Healthcheck on service [kube-proxy] on host [rke.onap.cloud] 
INFO[0076] [healthcheck] service [kube-proxy] on host [rke.onap.cloud] is healthy 
INFO[0076] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0077] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0077] [worker] Successfully started Worker Plane.. 
INFO[0077] [sync] Syncing nodes Labels and Taints 2          29m     10.42.3.8        3.17.76.33       <none>           <none>
onap            onap-aai-aai-traversal-update-query-data-9g2b8           0/1     Init:0/1           2          29m     10.42.5.12       18.220.70.253    <none>           <none>
onap           INFO[0077] [sync] Successfully synced nodes Labels and Taints 
INFO[0077] [network] Setting up network plugin: canal   
INFO[0077] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0077] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-network-plugin 
INFO[0077] [addons] Executing deploy job..    onap-dmaap-dbc-pg-0                                      1/1     Running            INFO[0082] [addons] Setting up KubeDNS0          29m     10.42.5.9        INFO[0082] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0082] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-kubedns-addon 
INFO[0082] [addons] Executing deploy job..18.220.70.253    <none>           <none>
onap            onap-dmaap-dbc-pg-1         INFO[0087] [addons] KubeDNS deployed successfully..      INFO[0087] [addons] Setting up Metrics Server            INFO[0087] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0087] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-metrics-addon 
INFO[0087] [addons] Executing deploy job.. 1/1     Running            0          26m     10.42.3.14   INFO[0092] [addons] KubeDNS deployed successfully3.17.76.33      INFO[0092] [ingress] Setting up nginx ingress controller 
INFO[0092] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0092] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-ingress-controller 
INFO[0092] [addons] Executing deploy job.. <none>           <none>
onap            onap-dmaap-dbc-pgpool-8666b57857-97zjc                   1/1   INFO[0097] [ingress] ingressRunning controller nginx is successfully deployed  INFO[0097] [addons] Setting up user addons0          29m     INFO[0097] [addons] Checking for included user addons   
WARN[0097] [addons] Unable to determine if  is a file path or url, skipping 
INFO[0097] [addons] Deploying rke-user-includes-addons  
INFO[0097] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0097] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-user-includes-addons 
INFO[0097] [addons] Executing deploy job..       10.42.5.5        18.220.70.253    <none>           <none>
onap            onap-dmaap-dbc-pgpool-8666b57857-vr8gk                   1/1     Running        WARN[0128] Failed to deploy addon0 execute job [rke-user-includes-addons]: Failed to get job complete status: <nil>29m  INFO[0128] Finished building Kubernetes cluster successfully 10.42.4.8     ubuntu@a-rke:~$ sudo docker ps
CONTAINER ID  18.188.214.137   <none>       IMAGE    <none>
onap            onap-dmaap-dmaap-bc-745995bf74-m6hhq               COMMAND      0/1     Init:0/2       CREATED    2         STATUS 29m     10.42.4.12        PORTS18.188.214.137   <none>           <none>
onap   NAMES ec26c4bd24b5        846921f0fe0eonap-dmaap-dmaap-bc-post-install-6ff4j                   1/1     Running "/server"           0      10 minutes ago  29m    Up 10.42.4.9 minutes       18.188.214.137   <none>           <none>
onap        k8s_default-http-backend_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
f8d5db205e14    onap-dmaap-dmaap-dr-db-0              8a7739f672b4                   1/1     Running "/sidecar --v=2 --lo…"   10 minutes ago    0  Up 10 minutes      29m     10.42.4.10       18.188.214.137   <none>      k8s_sidecar_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0 490461545ae4    <none>
onap   rancher/metrics-server-amd64         "/metrics-server --s…"onap-dmaap-dmaap-dr-db-1        10 minutes ago      Up 10 minutes               1/1     Running       k8s_metrics-server_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
aaf03b62bd41     1         6816817d9dce 24m     10.42.5.15       18.220.70.253    <none>        "/dnsmasq-nanny -v=2…"  <none>
10onap minutes ago      Up 10 minutes  onap-dmaap-dmaap-dr-node-0                         k8s_dnsmasq_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0 58ec007db72f     2/2   55ffe31ac578  Running            0          29m "/kube-dns --domain=…"   10 minutes ago.42.3.11       3.17.76.33  Up 10 minutes   <none>           <none>
onap            k8s_kubedns_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
0a95c06f6aa6onap-dmaap-dmaap-dr-prov-fbf6c94f5-v9bmq            e183460c484d     2/2     Running            0   "/cluster-proportion…"   10 minutes ago  29m    Up 10.42.5.10 minutes       18.220.70.253    <none>           <none>
onap            k8s_autoscaler_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
968a7c99b210onap-dmaap-message-router-0                rancher/pause-amd64:3.1              "/pause"1/1     Running            0      10 minutes ago  29m    Up 10 minutes.42.4.14       18.188.214.137   <none>           <none>
onap            k8s_POD_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
69969b331e49onap-dmaap-message-router-kafka-0          rancher/pause-amd64:3.1              "/pause"1/1     Running            1      10 minutes ago  29m    Up 10 minutes.42.5.13       18.220.70.253    <none>           <none>
onap            k8s_POD_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
baa5f03c16ffonap-dmaap-message-router-kafka-1                        rancher/pause-amd64:3.1/1     Running         "/pause"   1          29m     10 minutes ago.42.3.13       3.17.76.33  Up 10 minutes   <none>           <none>
onap            k8s_POD_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
82b2a9f640cbonap-dmaap-message-router-kafka-2          rancher/pause-amd64:3.1              "1/pause"1     Running            100 minutes ago      Up 10 minutes29m     10.42.4.15       18.188.214.137   <none>            k8s_POD_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
953a4d4be0c1<none>
onap         df4469c42185   onap-dmaap-message-router-mirrormaker-8587c4c9cf-lfnd8   0/1     CrashLoopBackOff   9           "/usr/bin/dumb-init …"29m     10 minutes ago.42.4.7       Up 10 minutes 18.188.214.137   <none>           <none>
onap             k8s_nginx-ingress-controller_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
cce552840749onap-dmaap-message-router-zookeeper-0                    rancher/pause-amd64:3.1/1     Running         "/pause"   0          29m     10.42.5.14 minutes ago      Up 10 minutes18.220.70.253    <none>           <none>
onap             k8s_POD_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
baa65f9c6f97onap-dmaap-message-router-zookeeper-1                    1/1   f0fad859c909  Running            0          29m "/opt/bin/flanneld -…"   10 minutes ago.42.4.13       18.188.214.137   <none>      Up 10 minutes   <none>
onap            onap-dmaap-message-router-zookeeper-2           k8s_kube-flannel_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0 1736ce68f41a        9f355e076ea71/1     Running            0        "/install-cni.sh"  29m     10.42.3.12 10 minutes ago    3.17.76.33  Up 10 minutes   <none>           <none>
onap            k8s_install-cni_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
615d3f702ee7   onap-nfs-provisioner-nfs-provisioner-57c999dc57-mdcw5    1/1     7eca10056c8eRunning            0          24m   "start_runit"  10.42.3.15        3.17.76.33  10 minutes ago   <none>   Up 10 minutes      <none>
onap                    k8s_calico-node_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1c4a702f0f18onap-robot-robot-677bdbb454-zj9jk          rancher/pause-amd64:3.1              "1/pause"1     Running            100 minutes ago      Up 10 minutes24m     10.42.5.16       18.220.70.253    <none>            k8s_POD_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
0da1cada08e1<none>
onap         rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"onap-so-so-8569947cbd-jn5x4   10 minutes ago      Up 10 minutes                 0/1     Init:0/1     kube-proxy 57f44998f34a     1   rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes13m ago      Up 11 minutes10.42.4.19       18.188.214.137   <none>           <none>
onap     kubelet 50f424c4daec        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes agoonap-so-so-bpmn-infra-78c8fd665d-b47qn                Up 11 minutes 0/1     Init:0/1           1          kube-scheduler13m 502d327912d9    10.42.3.16    rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"3.17.76.33   11 minutes ago  <none>    Up 11 minutes     <none>
onap            onap-so-so-catalog-db-adapter-565f9767ff-lvbgx         kube-controller-manager 9fc706bbf3a5 0/1     Init:0/1  rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago 1     Up 11 minutes   13m     10.42.3.17       3.17.76.33       <none>     kube-apiserver 2e7630c2047c     <none>
onap  rancher/coreos-etcd:v3.2.18          "/usr/local/bin/etcd…"onap-so-so-mariadb-config-job-d9sdb   11 minutes ago      Up 11 minutes         0/1     Init:0/2           0  etcd   fef566337eb6     3m37s   rancher/rke-tools:v0.1.1510.42.3.20       3.17.76.33      "/opt/rke-tools/rke-…"  <none>  26 minutes ago      Up 26<none>
minutesonap            onap-so-so-mariadb-config-job-rkqdl               etcd-rolling-snapshots   amdocs@obriensystemsu0:~$ kubectl get pods --all-namespaces
NAMESPACE 0/1     Init:Error   NAME      0          13m     10.42.5.19       18.220.70.253    <none>      READY     STATUS<none>
onap     RESTARTS   AGE ingress-nginx   defaultonap-so-httpso-backendmonitoring-797c5bc54769b9fdd94c-m8hbxdks4v     1/1       Running     0  1/1     Running   1h ingress-nginx   nginx-ingress-controller-2v7w7     0       1/1   13m    Running 10.42.4.18    0   18.188.214.137   <none>    1h kube-system     canal-thmfg <none>
onap            onap-so-so-openstack-adapter-5f9cf896d7-mgbdd            0/1     3/3Init:0/1           1   Running     0  13m     10.42.4.17   1h kube-system   18.188.214.137  kube-dns-7588d5b5f5-j66s8 <none>           <none>
onap     3/3       Runningonap-so-so-request-db-adapter-5c9bfd7b57-2krnp     0      0/1    1h
kube-system Init:0/1     kube-dns-autoscaler-5db9bbb766-rg5n8      1/1       Running   13m  0   10.42.3.18       1h
kube-system3.17.76.33       <none> metrics-server-97bc649d5-jd2rr          <none>
onap 1/1       Running     0onap-so-so-sdc-controller-6fb5cf5775-bsxhm           1h kube-system   0/1  rke-ingress-controller-deploy-job-znp9n   Init:0/1           1   Completed   0    13m     10.42.4.20 1h kube-system     rke-kubedns-addon-deploy-job-dzxsj  18.188.214.137   <none>      0/1     <none>
onap  Completed   0       onap-so-so-sdnc-adapter-8555689c75-r6vkb   1h kube-system     rke-metrics-addon-deploy-job-gpm4j        01/1     Running  Completed   0       0   1h kube-system     rke-network-plugin-deploy-job-kqdds  13m      0/110.42.5.18       18.220.70.253 Completed   0<none>          1h kube-system<none>
onap    tiller-deploy-69458576b-khgr5        onap-so-so-vfc-adapter-68fccc8bb8-c56t2     1/1        Running     0/1     Init:0/1     1h

DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user

Code Block

theme	Midnight

amdocs@obriensystemsu0:~$ sudo rke up Segmentation fault (core1 dumped)  # issue was I was using ubuntu as13m the yml user not amdocs in this case for a particular VM

DI 20190227-1: Verify no 110 pod limit per VM

DI 20190228-1: deploy casablanca MR to RKE under K8S 1.11.6, Docker 18.06, Helm 2.12.3

Code Block

theme	Midnight

sudo git clone https://gerrit.onap.org/r/logging-analytics
sudo wget https://git.onap.org/oom/plain/kubernetes/onap/resources/environments/dev.yaml
sudo cp dev.yaml dev0.yaml
sudo vi dev0.yaml
sudo cp dev0.yaml dev1.yaml
sudo cp logging-analytics/deploy/cd.sh .
sudo ./cd.sh -b casablanca -e onap -p false nexus3.onap.org:10001 -f true -s 300 -c true -d false -w false -r false


no good for helm 2.12.3 deployment - just using 2.9.1 for now - 
Error: Chart incompatible with Tiller v2.12.3


in the casablanca branch only - flip
https://git.onap.org/oom/tree/kubernetes/onap/Chart.yaml?h=casablanca#n24
tillerVersion: "~2.9.1"

DI 20190305-1: Azure 256G VM full ONAP Testing

Code Block

theme	Midnight

obrienbiometrics:oom michaelobrien$ ssh ubuntu@onap-dmz.onap.cloud

./oom_deployment.sh -b master -s rke.onap.cloud -e onap -r a_rke0_master -t _arm_deploy_onap_cd.json -p _arm_deploy_onap_rke_z_parameters.json  10.42.4.21       18.188.214.137   <none>           <none>
onap            onap-so-so-vnfm-adapter-65c4c5944b-72nlf                 1/1     Running            0          13m     10.42.3.19       3.17.76.33       <none>           <none>


# on worker nodes only
# nfs client

DI 20190507: ARM support using RKE 0.2.1 ARM friendly install

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	LOG-331

Instance type(s) a1.4xlarge $0.408

AMI ID ami-0b9bd0b532ebcf4c9

Notes

Pre-RKE installation details in Cloud Native Deployment

...

Version	Old Version 30	New Version Current
Changes made by	Michael O'Brien	Michael O'Brien
Saved on	Mar 05, 2019	May 07, 2019

Content Comparison

Versions Compared

Key

Automated Installation Video of RKE install

20190227 - VMware

Quickstart

20190330 - AWS

Quickstart

Versions

Prerequisites

Ubuntu 16.04 VM

Determine RKE and Docker versions

Elastic Reserved IP

generate cluster.yml - optional

Kubernetes Single Node Developer Installation

Kubernetes HA Cluster Production Installation

Design Issues

DI 20190225-1: RKE/Docker version pair

DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working

Versions

Prerequisites

Ubuntu 16.04 VM

Determine RKE and Docker versions

Private SSH key

Elastic Reserved IP

generate cluster.yml - optional

Kubernetes Single Node Developer Installation

Kubernetes HA Cluster Production Installation

Design Issues

DI 20190225-1: RKE/Docker version pair

DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working

DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user

DI 20190227-1: Verify no 110 pod limit per VM

DI 20190228-1: deploy casablanca MR to RKE under K8S 1.11.6, Docker 18.06, Helm 2.12.3

DI 20190305-1: Azure 256G VM full ONAP Testing

DI 20190425: HA RKE install Testing

DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user

DI 20190227-1: Verify no 110 pod limit per VM

DI 20190228-1: deploy casablanca MR to RKE under K8S 1.11.6, Docker 18.06, Helm 2.12.3

DI 20190507: ARM support using RKE 0.2.1 ARM friendly install

Notes