Table of Contents

Tracking

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	OOM-1598

WIP - as of 20190225

NON-HA version in https://git.onap.org/oom/tree/kubernetes/contrib/tools/rke/rke_setup.sh

https://gerrit.onap.org/r/#/c/79067/

Move to https://onap.readthedocs.io/en/beijing/submodules/oom.git/docs/oom_cloud_setup_guide.html or similar when this documentation is released

https://lists.onap.org/g/onap-discuss/topic/rke_ha_collaboration_for/31313969?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,31313969

Automated Installation Video of RKE install

20190227 - VMware

View file

name	20190228_rke_install_preliminary.mp4
height	250

20190330 - AWS

View file

name	2019022720190330_rke_install_016nso_blue_preliminaryzoom_1click0_install_oom-1598-6min5_min.mp4
height	250

Versions

Currently Docker 18.06, RKE 0.16, Kubernetes 1.11.6, Kubectl 1.11.6, Helm 2.9.1

TODO: verify later versions of helm and a way to get RKE to install Kubernetes 1.13

Prerequisites

Ubuntu 16.04 VM

Determine RKE and Docker versions

Don't just use the latest docker version - check the RKE release page to get the version pair - 0.15/17.03 and 0.16/18.06 - see https://github.com/docker/docker-ce/releases - currently https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce

...

theme	Midnight

...

Quickstart

Get your public and private keys on the Ubuntu 16.04 VM.

Adjust authorized_keys with your public key if not already - aws has it, openstack may not

get rke script from jira, gerrit or by cloning OOM when the review is done.

Code Block

theme	Midnight

# on your laptop/where your cert is
# chmod 777 your cert before you scp it over
obrienbiometrics:full michaelobrien$ scp ~/wse_onap/onap_rsa ubuntu@rke0.onap.info:~/

# on the host
sudo cp onap_rsa ~/.ssh
sudo chmod 400 ~/.ssh/onap_rsa
sudo chown ubuntu:ubuntu ~/.ssh/onap_rsa 
# just verify
sudo vi ~/.ssh/authorized_keys

git clone --recurse-submodules https://gerrit.onap.org/r/oom
sudo cp oom/kubernetes/contrib/tools/rke/rke_setup.sh .
sudo nohup ./rke_setup.sh -b master -s 104.209.161.210 -e onap -k onap_rsa -l ubuntu &

ubuntu@a-rke0-master:~$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                      READY     STATUS      RESTARTS   AGE
ingress-nginx   default-http-backend-797c5bc547-55fpn     1/1       Running     0          4m
ingress-nginx   nginx-ingress-controller-znhgz            1/1       Running     0          4m
kube-system     canal-dqt2m                               3/3       Running     0          5m
kube-system     kube-dns-7588d5b5f5-pzdfh                 3/3       Running     0          5m
kube-system     kube-dns-autoscaler-5db9bbb766-b7vvg      1/1       Running     0          5m
kube-system     metrics-server-97bc649d5-fmqjd            1/1       Running     0          4m
kube-system     rke-ingress-controller-deploy-job-dxmbd   0/1       Completed   0          4m
kube-system     rke-kubedns-addon-deploy-job-wqccp        0/1       Completed   0          5m
kube-system     rke-metrics-addon-deploy-job-ssrgp        0/1       Completed   0          4m
kube-system     rke-network-plugin-deploy-job-jkffq       0/1       Completed   0          5m
kube-system     tiller-deploy-759cb9df9-rlt7v             1/1       Running     0          2m
ubuntu@a-rke0-master:~$ helm list

Versions

Currently Docker 18.06, RKE 0.1.16, Kubernetes 1.11.6, Kubectl 1.11.6, Helm 2.12.3

TODO: verify later versions of helm and a way to get RKE to install Kubernetes 1.13

Prerequisites

Ubuntu 16.04 VM

Determine RKE and Docker versions

Don't just use the latest docker version - check the RKE release page to get the version pair - 0.1.15/17.03 and 0.1.16/18.06 - see https://github.com/docker/docker-ce/releases - currently https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce

Code Block

theme	Midnight

ubuntu@a-rke:~$ sudo curl https://releases.rancher.com/install-docker/18.06.sh | sh
ubuntu@a-rke:~$ sudo usermod -aG docker ubuntu
ubuntu@a-rke:~$ sudo docker version
Client:
 Version:           18.06.3-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        d7080c1
 Built:             Wed Feb 20 02:27:18 2019

# install RKE
sudo wget https://github.com/rancher/rke/releases/download/v0.1.16/rke_linux-amd64
mv rke_linux-amd64 rke
sudo mv ./rke /usr/local/bin/rke

ubuntu@a-rke:~$ rke --version
rke version v0.1.16

Private SSH key

scp your public key to the box - ideally to ~/.ssh and chmod 400 it - make sure you add your key to authorized_keys

Elastic Reserved IP

get a VIP or EIP and assign this to your VM

generate cluster.yml - optional

cluster.yml will generated by the script rke_setup.sh

Code Block

theme	Midnight

azure config - no need to hand build the yml
Watch the path of your 2 keys
Also don't add an "addon" until you have one of the config job will fail

{noformat}
ubuntu@a-rke:~$ rke config --name cluster.yml
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/onap_rsa
[+] Number of Hosts [1]: 
[+] SSH Address of host (1) [none]: rke.onap.cloud
[+] SSH Port of host (1) [22]: 
[+] SSH Private Key Path of host (rke.onap.cloud) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (rke.onap.cloud) [ubuntu]: 
[+] Is host (rke.onap.cloud) a Control Plane host (y/n)? [y]: y
[+] Is host (rke.onap.cloud) a Worker host (y/n)? [n]: y
[+] Is host (rke.onap.cloud) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (rke.onap.cloud) [none]: 
[+] Internal IP of host (rke.onap.cloud) [none]: 
[+] Docker socket path on host (rke.onap.cloud) [/var/run/docker.sock]: 
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]: 
[+] Authentication Strategy [x509]: 
[+] Authorization Mode (rbac, none) [rbac]: 
[+] Kubernetes Docker image [rancher/hyperkube:v1.11.6-rancher1]: 
[+] Cluster domain [cluster.local]: 
[+] Service Cluster IP Range [10.43.0.0/16]: 
[+] Enable PodSecurityPolicy [n]: 
[+] Cluster Network CIDR [10.42.0.0/16]: 
[+] Cluster DNS Service IP [10.43.0.10]: 
[+] Add addon manifest URLs or YAML files [no]: no
ubuntu@a-rke:~$ sudo cat cluster.yml 
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: rke.onap.cloud
  port: "22"
  internal_address: ""
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: ""
  user: ubuntu
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/onap_rsa
  labels: {}
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    snapshot: null
    retention: ""
    creation: ""
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: canal
  options: {}
authentication:
  strategy: x509
  options: {}
  sans: []
system_images:
  etcd: rancher/coreos-etcd:v3.2.18
  alpine: rancher/rke-tools:v0.1.15
  nginx_proxy: rancher/rke-tools:v0.1.15
  cert_downloader: rancher/rke-tools:v0.1.15
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.15
  kubedns: rancher/k8s-dns-kube-dns-amd64:1.14.10
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny-amd64:1.14.10
  kubedns_sidecar: rancher/k8s-dns-sidecar-amd64:1.14.10
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler-amd64:1.0.0
  kubernetes: rancher/hyperkube:v1.11.6-rancher1
  flannel: rancher/coreos-flannel:v0.10.0
  flannel_cni: rancher/coreos-flannel-cni:v0.3.0
  calico_node: rancher/calico-node:v3.1.3
  calico_cni: rancher/calico-cni:v3.1.3
  calico_controllers: ""
  calico_ctl: rancher/calico-ctl:v2.0.0
  canal_node: rancher/calico-node:v3.1.3
  canal_cni: rancher/calico-cni:v3.1.3
  canal_flannel: rancher/coreos-flannel:v0.10.0
  wave_node: weaveworks/weave-kube:2.1.2
  weave_cni: weaveworks/weave-npc:2.1.2
  pod_infra_container: rancher/pause-amd64:3.1
  ingress: rancher/nginx-ingress-controller:0.16.2-rancher1
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.4
  metrics_server: rancher/metrics-server-amd64:v0.2.1
ssh_key_path: ~/.ssh/onap_rsa
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
  provider: ""
  options: {}
  node_selector: {}
  extra_args: {}
cluster_name: ""
cloud_provider:
  name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""
monitoring:
  provider: ""
  options: {}
{noformat}

Kubernetes Single Node Developer Installation

Code Block

theme	Midnight

sudo ./rke_install.sh -b master -s localhost -e onap -l ubuntu

Kubernetes HA Cluster Production Installation

Design Issues

DI 20190225-1: RKE/Docker version pair

As of 20190215 RKE 0.16 supports Docker 18.06-ce (and 18.09 non-ce) (up from 0.15 supporting 17.03)

https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce

https://github.com/rancher/rke/releases/tag/v0.1.16

Code Block

theme	Midnight

ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster                  
INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] 
FATA[0000] Unsupported Docker version found [18.06.3-ce], supported versions are [1.11.x 1.12.x 1.13.x 17.03.x]

DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working

Does rke remove, regenerate the yaml (or hand upgrade the versions) then rke up

Code Block

theme	Midnight

ubuntu@a-rke:~$ sudo rke remove
Are you sure you want to remove Kubernetes cluster [y/n]: y
INFO[0002] Tearing down Kubernetes cluster              
INFO[0002] [dialer] Setup tunnel for host [rke.onap.cloud] 
INFO[0002] [worker] Tearing down Worker Plane..         
INFO[0002] [remove/kubelet] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/kube-proxy] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [worker] Successfully tore down Worker Plane.. 
INFO[0003] [controlplane] Tearing down the Controller Plane.. 
INFO[0003] [remove/kube-apiserver] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/kube-controller-manager] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [remove/kube-scheduler] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [controlplane] Host [rke.onap.cloud] is already a worker host, skipping delete kubelet and kubeproxy. 
INFO[0004] [controlplane] Successfully tore down Controller Plane.. 
INFO[0004] [etcd] Tearing down etcd plane..             
INFO[0004] [remove/etcd] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [etcd] Successfully tore down etcd plane..   
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0004] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0005] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0005] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0005] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0006] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0006] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0007] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0008] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0008] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0008] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0009] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0009] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0009] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0010] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0010] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0010] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0011] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0011] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0011] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0011] Removing local admin Kubeconfig: ./kube_config_cluster.yml 
INFO[0011] Local admin Kubeconfig removed successfully  
INFO[0011] Cluster removed successfully  

ubuntu@a-rke:~$ rke config --name cluster.ym
ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster                  
INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] 
INFO[0000] [network] Deploying port listener containers 
INFO[0001] [network] Successfully started [rke-etcd-port-listener] container on host [rke.onap.cloud] 
INFO[0001] [network] Successfully started [rke-cp-port-listener] container on host [rke.onap.cloud] 
INFO[0002] [network] Successfully started [rke-worker-port-listener] container on host [rke.onap.cloud] 
INFO[0002] [network] Port listener containers deployed successfully 
INFO[0002] [network] Running control plane -> etcd port checks 
INFO[0003] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0003] [network] Running control plane -> worker port checks 
INFO[0004] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0004] [network] Running workers -> control plane port checks 
INFO[0005] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0005] [network] Checking KubeAPI port Control Plane hosts 
INFO[0005] [network] Removing port listener containers  
INFO[0005] [remove/rke-etcd-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-cp-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-worker-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [network] Port listener containers removed successfully 
INFO[0006] [certificates] Attempting to recover certificates from backup on [etcd,controlPlane] hosts 
INFO[0007] [certificates] No Certificate backup found on [etcd,controlPlane] hosts 
INFO[0007] [certificates] Generating CA kubernetes certificates 
INFO[0007] [certificates] Generating Kubernetes API server certficates 
INFO[0008] [certificates] Generating Kube Controller certificates 
INFO[0008] [certificates] Generating Kube Scheduler certificates 
INFO[0008] [certificates] Generating Kube Proxy certificates 
INFO[0009] [certificates] Generating Node certificate   
INFO[0009] [certificates] Generating admin certificates and kubeconfig 
INFO[0009] [certificates] Generating etcd-rke.onap.cloud certificate and key 
INFO[0009] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0009] [certificates] Generating Kubernetes API server proxy client certificates 
INFO[0010] [certificates] Temporarily saving certs to [etcd,controlPlane] hosts 
INFO[0016] [certificates] Saved certs to [etcd,controlPlane] hosts 
INFO[0016] [reconcile] Reconciling cluster state        
INFO[0016] [reconcile] This is newly generated cluster  
INFO[0016] [certificates] Deploying kubernetes certificates to Cluster nodes 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes 
INFO[0022] Pre-pulling kubernetes images                
INFO[0022] Kubernetes images pulled successfully        
INFO[0022] [etcd] Building up etcd plane..              
INFO[0023] [etcd] Successfully started [etcd] container on host [rke.onap.cloud] 
INFO[0023] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [rke.onap.cloud] 
INFO[0028] [certificates] Successfully started [rke-bundle-cert] container on host [rke.onap.cloud] 
INFO[0029] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [rke.onap.cloud] 
INFO[0029] [etcd] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0030] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0030] [etcd] Successfully started etcd plane..     
INFO[0030] [controlplane] Building up Controller Plane.. 
INFO[0031] [controlplane] Successfully started [kube-apiserver] container on host [rke.onap.cloud] 
INFO[0031] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [rke.onap.cloud] 
INFO[0045] [healthcheck] service [kube-apiserver] on host [rke.onap.cloud] is healthy 
INFO[0046] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0046] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0047] [controlplane] Successfully started [kube-controller-manager] container on host [rke.onap.cloud] 
INFO[0047] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [rke.onap.cloud] 
INFO[0052] [healthcheck] service [kube-controller-manager] on host [rke.onap.cloud] is healthy 
INFO[0053] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0053] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0054] [controlplane] Successfully started [kube-scheduler] container on host [rke.onap.cloud] 
INFO[0054] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [rke.onap.cloud] 
INFO[0059] [healthcheck] service [kube-scheduler] on host [rke.onap.cloud] is healthy 
INFO[0060] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0060] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0060] [controlplane] Successfully started Controller Plane.. 
INFO[0060] [authz] Creating rke-job-deployer ServiceAccount 
INFO[0060] [authz] rke-job-deployer ServiceAccount created successfully 
INFO[0060] [authz] Creating system:node ClusterRoleBinding 
INFO[0060] [authz] system:node ClusterRoleBinding created successfully 
INFO[0060] [certificates] Save kubernetes certificates as secrets 
INFO[0060] [certificates] Successfully saved certificates as kubernetes secret [k8s-certs] 
INFO[0060] [state] Saving cluster state to Kubernetes   
INFO[0061] [state] Successfully Saved cluster state to Kubernetes ConfigMap: cluster-state 
INFO[0061] [state] Saving cluster state to cluster nodes 
INFO[0061] [state] Successfully started [cluster-state-deployer] container on host [rke.onap.cloud] 
INFO[0062] [remove/cluster-state-deployer] Successfully removed container on host [rke.onap.cloud] 
INFO[0062] [worker] Building up Worker Plane..          
INFO[0062] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] 
INFO[0063] [worker] Successfully started [kubelet] container on host [rke.onap.cloud] 
INFO[0063] [healthcheck] Start Healthcheck on service [kubelet] on host [rke.onap.cloud] 
INFO[0068] [healthcheck] service [kubelet] on host [rke.onap.cloud] is healthy 
INFO[0069] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0070] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0070] [worker] Successfully started [kube-proxy] container on host [rke.onap.cloud] 
INFO[0070] [healthcheck] Start Healthcheck on service [kube-proxy] on host [rke.onap.cloud] 
INFO[0076] [healthcheck] service [kube-proxy] on host [rke.onap.cloud] is healthy 
INFO[0076] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0077] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0077] [worker] Successfully started Worker Plane.. 
INFO[0077] [sync] Syncing nodes Labels and Taints       
INFO[0077] [sync] Successfully synced nodes Labels and Taints 
INFO[0077] [network] Setting up network plugin: canal   
INFO[0077] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0077] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-network-plugin 
INFO[0077] [addons] Executing deploy job..              
INFO[0082] [addons] Setting up KubeDNS                  
INFO[0082] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0082] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-kubedns-addon 
INFO[0082] [addons] Executing deploy job..              
INFO[0087] [addons] KubeDNS deployed successfully..     
INFO[0087] [addons] Setting up Metrics Server           
INFO[0087] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0087] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-metrics-addon 
INFO[0087] [addons] Executing deploy job..              
INFO[0092] [addons] KubeDNS deployed successfully..     
INFO[0092] [ingress] Setting up nginx ingress controller 
INFO[0092] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0092] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-ingress-controller 
INFO[0092] [addons] Executing deploy job..              
INFO[0097] [ingress] ingress controller nginx is successfully deployed 
INFO[0097] [addons] Setting up user addons              
INFO[0097] [addons] Checking for included user addons   
WARN[0097] [addons] Unable to determine if  is a file path or url, skipping 
INFO[0097] [addons] Deploying rke-user-includes-addons  
INFO[0097] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0097] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-user-includes-addons 
INFO[0097] [addons] Executing deploy job..              
WARN[0128] Failed to deploy addon execute job [rke-user-includes-addons]: Failed to get job complete status: <nil> 
INFO[0128] Finished building Kubernetes cluster successfully 

ubuntu@a-rke:~$ sudo docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS               NAMES
ec26c4bd24b5        846921f0fe0e                         "/server"                10 minutes ago      Up 10 minutes                           k8s_default-http-backend_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
f8d5db205e14        8a7739f672b4                         "/sidecar --v=2 --lo…"   10 minutes ago      Up 10 minutes                           k8s_sidecar_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
490461545ae4        rancher/metrics-server-amd64         "/metrics-server --s…"   10 minutes ago      Up 10 minutes                           k8s_metrics-server_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
aaf03b62bd41        6816817d9dce                         "/dnsmasq-nanny -v=2…"   10 minutes ago      Up 10 minutes                           k8s_dnsmasq_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
58ec007db72f        55ffe31ac578                         "/kube-dns --domain=…"   10 minutes ago      Up 10 minutes                           k8s_kubedns_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
0a95c06f6aa6        e183460c484d                         "/cluster-proportion…"   10 minutes ago      Up 10 minutes                           k8s_autoscaler_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
968a7c99b210        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
69969b331e49        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
baa5f03c16ff        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
82b2a9f640cb        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
953a4d4be0c1        df4469c42185                         "/usr/bin/dumb-init …"   10 minutes ago      Up 10 minutes                           k8s_nginx-ingress-controller_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
cce552840749        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
baa65f9c6f97        f0fad859c909                         "/opt/bin/flanneld -…"   10 minutes ago      Up 10 minutes                           k8s_kube-flannel_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1736ce68f41a        9f355e076ea7                         "/install-cni.sh"        10 minutes ago      Up 10 minutes                           k8s_install-cni_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
615d3f702ee7        7eca10056c8e                         "start_runit"            10 minutes ago      Up 10 minutes                           k8s_calico-node_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1c4a702f0f18        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
0da1cada08e1        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   10 minutes ago      Up 10 minutes                           kube-proxy
57f44998f34a        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kubelet
50f424c4daec        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kube-scheduler
502d327912d9        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kube-controller-manager
9fc706bbf3a5        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kube-apiserver
2e7630c2047c        rancher/coreos-etcd:v3.2.18          "/usr/local/bin/etcd…"   11 minutes ago      Up 11 minutes                           etcd
fef566337eb6        rancher/rke-tools:v0.1.15            "/opt/rke-tools/rke-…"   26 minutes ago      Up 26 minutes                           etcd-rolling-snapshots


amdocs@obriensystemsu0:~$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                      READY     STATUS      RESTARTS   AGE
ingress-nginx   default-http-backend-797c5bc547-m8hbx     1/1       Running     0          1h
ingress-nginx   nginx-ingress-controller-2v7w7            1/1       Running     0          1h
kube-system     canal-thmfg                               3/3       Running     0          1h
kube-system     kube-dns-7588d5b5f5-j66s8                 3/3       Running     0          1h
kube-system     kube-dns-autoscaler-5db9bbb766-rg5n8      1/1       Running     0          1h
kube-system     metrics-server-97bc649d5-jd2rr            1/1       Running     0          1h
kube-system     rke-ingress-controller-deploy-job-znp9n   0/1       Completed   0          1h
kube-system     rke-kubedns-addon-deploy-job-dzxsj        0/1       Completed   0          1h
kube-system     rke-metrics-addon-deploy-job-gpm4j        0/1       Completed   0          1h
kube-system     rke-network-plugin-deploy-job-kqdds       0/1       Completed   0          1h
kube-system     tiller-deploy-69458576b-khgr5             1/1       Running     0          1h

DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user

Code Block

theme	Midnight

amdocs@obriensystemsu0:~$ sudo rke up
Segmentation fault (core dumped)

# issue was I was using ubuntu as the yml user not amdocs in this case for a particular VM

DI 20190227-1: Verify no 110 pod limit per VM

https://forums.rancher.com/t/solved-setting-max-pods/11866

Code Block

theme	Midnight

kubelet:
    image: ""
    extra_args:
      max-pods: 900

DI 20190228-1: deploy casablanca MR to RKE under K8S 1.11.6, Docker 18.06, Helm 2.12.3

Code Block

theme	Midnight

sudo git clone https://gerrit.onap.org/r/logging-analytics
sudo wget https://git.onap.org/oom/plain/kubernetes/onap/resources/environments/dev.yaml
sudo cp dev.yaml dev0.yaml
sudo vi dev0.yaml
sudo cp dev0.yaml dev1.yaml
sudo cp logging-analytics/deploy/cd.sh .
sudo ./cd.sh -b casablanca -e onap -p false nexus3.onap.org:10001 -f true -s 300 -c true -d false -w false -r false


no good for helm 2.12.3 deployment - just using 2.9.1 for now - 
Error: Chart incompatible with Tiller v2.12.3


in the casablanca branch only - flip
https://git.onap.org/oom/tree/kubernetes/onap/Chart.yaml?h=casablanca#n24
tillerVersion: "~2.9.1"

DI 20190305-1: Azure 256G VM full ONAP Testing

Code Block

theme	Midnight

obrienbiometrics:oom michaelobrien$ ssh ubuntu@onap-dmz.onap.cloud

./oom_deployment.sh -b master -s rke.onap.cloud -e onap -r a_rke0_master -t _arm_deploy_onap_cd.json -p _arm_deploy_onap_rke_z_parameters.json

DI 20190425: HA RKE install Testing

Manual first for RC0, later retrofit the script in https://git.onap.org/oom/tree/kubernetes/contrib/tools/rke/rke_setup.sh and move/adjust the heat template in https://git.onap.org/logging-analytics/tree/deploy/heat/logging_openstack_13_16g.yaml

Installing on 6 nodes on AWS (windriver is having an issue right now).

We are good on RKE 0.2.1, Ubuntu 18.04 / Kubernetes/kubectl 1.13.5 / Helm 2.13.1 / Docker 18.09.5

https://github.com/rancher/rke/releases RKE 0.2.2 has experimental k8s 1.14 support - running with 0.2.1 for now

Just need to test onap deployments

I'll do the NFS/EFS later before deployment

Code Block

theme	Midnight

# on all VMs (control, etcd, worker)
# move the key to all vms
scp ~/wse_onap/onap_rsa ubuntu@rke0.onap.info:~/
sudo curl https://releases.rancher.com/install-docker/18.09.sh | sh
sudo usermod -aG docker ubuntu

# nfs server

# on control/etcd nodes only
# from script
sudo wget https://github.com/rancher/rke/releases/download/v0.2.1/rke_linux-amd64
mv rke_linux-amd64 rke
sudo chmod +x rke
sudo mv ./rke /usr/local/bin/rke
# one time setup of the yaml or use the generated one
ubuntu@ip-172-31-38-182:~$ sudo rke config
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/onap_rsa
[+] Number of Hosts [1]: 6
[+] SSH Address of host (1) [none]: 3.14.102.175
[+] SSH Port of host (1) [22]: 
[+] SSH Private Key Path of host (3.14.102.175) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (3.14.102.175) [ubuntu]: ubuntu
[+] Is host (3.14.102.175) a Control Plane host (y/n)? [y]: y
[+] Is host (3.14.102.175) a Worker host (y/n)? [n]: n
[+] Is host (3.14.102.175) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (3.14.102.175) [none]: 
[+] Internal IP of host (3.14.102.175) [none]: 
[+] Docker socket path on host (3.14.102.175) [/var/run/docker.sock]: 
[+] SSH Address of host (2) [none]: 18.220.62.6
[+] SSH Port of host (2) [22]: 
[+] SSH Private Key Path of host (18.220.62.6) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.220.62.6) [ubuntu]: 
[+] Is host (18.220.62.6) a Control Plane host (y/n)? [y]: y
[+] Is host (18.220.62.6) a Worker host (y/n)? [n]: n
[+] Is host (18.220.62.6) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (18.220.62.6) [none]: 
[+] Internal IP of host (18.220.62.6) [none]: 
[+] Docker socket path on host (18.220.62.6) [/var/run/docker.sock]: 
[+] SSH Address of host (3) [none]: 18.217.96.12
[+] SSH Port of host (3) [22]: 
[+] SSH Private Key Path of host (18.217.96.12) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.217.96.12) [ubuntu]: 
[+] Is host (18.217.96.12) a Control Plane host (y/n)? [y]: y
[+] Is host (18.217.96.12) a Worker host (y/n)? [n]: n
[+] Is host (18.217.96.12) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (18.217.96.12) [none]: 
[+] Internal IP of host (18.217.96.12) [none]: 
[+] Docker socket path on host (18.217.96.12) [/var/run/docker.sock]: 
[+] SSH Address of host (4) [none]: 18.188.214.137
[+] SSH Port of host (4) [22]: 
[+] SSH Private Key Path of host (18.188.214.137) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.188.214.137) [ubuntu]: 
[+] Is host (18.188.214.137) a Control Plane host (y/n)? [y]: n
[+] Is host (18.188.214.137) a Worker host (y/n)? [n]: y
[+] Is host (18.188.214.137) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (18.188.214.137) [none]: 
[+] Internal IP of host (18.188.214.137) [none]: 
[+] Docker socket path on host (18.188.214.137) [/var/run/docker.sock]: 
[+] SSH Address of host (5) [none]: 18.220.70.253
[+] SSH Port of host (5) [22]: 
[+] SSH Private Key Path of host (18.220.70.253) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.220.70.253) [ubuntu]: 
[+] Is host (18.220.70.253) a Control Plane host (y/n)? [y]: n
[+] Is host (18.220.70.253) a Worker host (y/n)? [n]: y
[+] Is host (18.220.70.253) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (18.220.70.253) [none]: 
[+] Internal IP of host (18.220.70.253) [none]: 
[+] Docker socket path on host (18.220.70.253) [/var/run/docker.sock]: 
[+] SSH Address of host (6) [none]: 3.17.76.33
[+] SSH Port of host (6) [22]: 
[+] SSH Private Key Path of host (3.17.76.33) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (3.17.76.33) [ubuntu]: 
[+] Is host (3.17.76.33) a Control Plane host (y/n)? [y]: n
[+] Is host (3.17.76.33) a Worker host (y/n)? [n]: y
[+] Is host (3.17.76.33) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (3.17.76.33) [none]: 
[+] Internal IP of host (3.17.76.33) [none]: 
[+] Docker socket path on host (3.17.76.33) [/var/run/docker.sock]: 
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]: 
[+] Authentication Strategy [x509]: 
[+] Authorization Mode (rbac, none) [rbac]: 
[+] Kubernetes Docker image [rancher/hyperkube:v1.13.5-rancher1]: 
[+] Cluster domain [cluster.local]: 
[+] Service Cluster IP Range [10.43.0.0/16]: 
[+] Enable PodSecurityPolicy [n]: 
[+] Cluster Network CIDR [10.42.0.0/16]: 
[+] Cluster DNS Service IP [10.43.0.10]: 
[+] Add addon manifest URLs or YAML files [no]:

# new
[+] Cluster domain [cluster.local]: 

ubuntu@ip-172-31-38-182:~$ sudo rke up
INFO[0000] Initiating Kubernetes cluster                
INFO[0000] [certificates] Generating CA kubernetes certificates 
INFO[0000] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0000] [certificates] Generating Kubernetes API server certificates 
INFO[0000] [certificates] Generating Service account token key 
INFO[0000] [certificates] Generating etcd-3.14.102.175 certificate and key 
INFO[0000] [certificates] Generating etcd-18.220.62.6 certificate and key 
INFO[0001] [certificates] Generating etcd-18.217.96.12 certificate and key 
INFO[0001] [certificates] Generating Kube Controller certificates 
INFO[0001] [certificates] Generating Kube Scheduler certificates 
INFO[0001] [certificates] Generating Kube Proxy certificates 
INFO[0001] [certificates] Generating Node certificate   
INFO[0001] [certificates] Generating admin certificates and kubeconfig 
INFO[0001] [certificates] Generating Kubernetes API server proxy client certificates 
INFO[0002] Successfully Deployed state file at [./cluster.rkestate] 
INFO[0002] Building Kubernetes cluster                  
INFO[0002] [dialer] Setup tunnel for host [3.14.102.175] 
INFO[0002] [dialer] Setup tunnel for host [18.188.214.137] 
INFO[0002] [dialer] Setup tunnel for host [18.220.70.253] 
INFO[0002] [dialer] Setup tunnel for host [18.220.62.6] 
INFO[0002] [dialer] Setup tunnel for host [18.217.96.12] 
INFO[0002] [dialer] Setup tunnel for host [3.17.76.33]  
INFO[0002] [network] Deploying port listener containers 
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.220.62.6] 
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [3.14.102.175] 
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.217.96.12] 
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.220.62.6] 
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.217.96.12] 
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [3.14.102.175] 
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [18.220.62.6] 
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [18.217.96.12] 
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [3.14.102.175] 
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [18.217.96.12] 
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [18.220.62.6] 
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [3.14.102.175] 
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.188.214.137] 
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.220.70.253] 
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [3.17.76.33] 
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [3.17.76.33] 
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.220.70.253] 
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.188.214.137] 
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [3.17.76.33] 
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [18.220.70.253] 
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [18.188.214.137] 
INFO[0013] [network] Port listener containers deployed successfully 
INFO[0013] [network] Running etcd <-> etcd port checks  
INFO[0013] [network] Successfully started [rke-port-checker] container on host [18.220.62.6] 
INFO[0013] [network] Successfully started [rke-port-checker] container on host [3.14.102.175] 
INFO[0013] [network] Successfully started [rke-port-checker] container on host [18.217.96.12] 
INFO[0014] [network] Running control plane -> etcd port checks 
INFO[0014] [network] Successfully started [rke-port-checker] container on host [18.220.62.6] 
INFO[0014] [network] Successfully started [rke-port-checker] container on host [3.14.102.175] 
INFO[0014] [network] Successfully started [rke-port-checker] container on host [18.217.96.12] 
INFO[0014] [network] Running control plane -> worker port checks 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.220.62.6] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.217.96.12] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [3.14.102.175] 
INFO[0015] [network] Running workers -> control plane port checks 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [3.17.76.33] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.220.70.253] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.188.214.137] 
INFO[0016] [network] Checking KubeAPI port Control Plane hosts 
INFO[0016] [network] Removing port listener containers  
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [18.220.62.6] 
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [3.14.102.175] 
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [18.217.96.12] 
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [18.217.96.12] 
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [3.14.102.175] 
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [18.220.62.6] 
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [18.220.70.253] 
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [3.17.76.33] 
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [18.188.214.137] 
INFO[0017] [network] Port listener containers removed successfully 
INFO[0017] [certificates] Deploying kubernetes certificates to Cluster nodes 
INFO[0022] [reconcile] Rebuilding and updating local kube config 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes 
INFO[0022] [reconcile] Reconciling cluster state        
INFO[0022] [reconcile] This is newly generated cluster  
INFO[0022] Pre-pulling kubernetes images                
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [3.14.102.175] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.188.214.137] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.70.253] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.217.96.12] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [3.17.76.33] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.62.6] 
INFO[0038] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.62.6] 
INFO[0038] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.70.253] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [3.17.76.33] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.217.96.12] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.188.214.137] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [3.14.102.175] 
INFO[0039] Kubernetes images pulled successfully        
INFO[0039] [etcd] Building up etcd plane..              
INFO[0039] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [3.14.102.175] 
INFO[0041] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [3.14.102.175] 
INFO[0051] [etcd] Successfully started [etcd] container on host [3.14.102.175] 
INFO[0051] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [3.14.102.175] 
INFO[0052] [etcd] Successfully started [etcd-rolling-snapshots] container on host [3.14.102.175] 
INFO[0057] [certificates] Successfully started [rke-bundle-cert] container on host [3.14.102.175] 
INFO[0058] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [3.14.102.175] 
INFO[0058] [etcd] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0058] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0058] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.220.62.6] 
INFO[0063] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.220.62.6] 
INFO[0069] [etcd] Successfully started [etcd] container on host [18.220.62.6] 
INFO[0069] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [18.220.62.6] 
INFO[0069] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.220.62.6] 
INFO[0075] [certificates] Successfully started [rke-bundle-cert] container on host [18.220.62.6] 
INFO[0075] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.220.62.6] 
INFO[0076] [etcd] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0076] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0076] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.217.96.12] 
INFO[0078] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.217.96.12] 
INFO[0078] [etcd] Successfully started [etcd] container on host [18.217.96.12] 
INFO[0078] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [18.217.96.12] 
INFO[0078] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.217.96.12] 
INFO[0084] [certificates] Successfully started [rke-bundle-cert] container on host [18.217.96.12] 
INFO[0084] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.217.96.12] 
INFO[0085] [etcd] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0085] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0085] [etcd] Successfully started etcd plane.. Checking etcd cluster health 
INFO[0086] [controlplane] Building up Controller Plane.. 
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [18.220.62.6] 
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.220.62.6] 
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [3.14.102.175] 
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [3.14.102.175] 
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [18.217.96.12] 
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.217.96.12] 
INFO[0098] [healthcheck] service [kube-apiserver] on host [18.220.62.6] is healthy 
INFO[0099] [healthcheck] service [kube-apiserver] on host [18.217.96.12] is healthy 
INFO[0099] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0099] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0099] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0099] [healthcheck] service [kube-apiserver] on host [3.14.102.175] is healthy 
INFO[0099] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0099] [controlplane] Successfully started [kube-controller-manager] container on host [18.220.62.6] 
INFO[0099] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.220.62.6] 
INFO[0100] [controlplane] Successfully started [kube-controller-manager] container on host [18.217.96.12] 
INFO[0100] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.217.96.12] 
INFO[0100] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0100] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0100] [healthcheck] service [kube-controller-manager] on host [18.220.62.6] is healthy 
INFO[0100] [healthcheck] service [kube-controller-manager] on host [18.217.96.12] is healthy 
INFO[0100] [controlplane] Successfully started [kube-controller-manager] container on host [3.14.102.175] 
INFO[0100] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [3.14.102.175] 
INFO[0101] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0101] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0101] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0101] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0101] [controlplane] Successfully started [kube-scheduler] container on host [18.220.62.6] 
INFO[0101] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.220.62.6] 
INFO[0101] [healthcheck] service [kube-controller-manager] on host [3.14.102.175] is healthy 
INFO[0101] [controlplane] Successfully started [kube-scheduler] container on host [18.217.96.12] 
INFO[0101] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.217.96.12] 
INFO[0102] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0102] [healthcheck] service [kube-scheduler] on host [18.220.62.6] is healthy 
INFO[0102] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0102] [healthcheck] service [kube-scheduler] on host [18.217.96.12] is healthy 
INFO[0103] [controlplane] Successfully started [kube-scheduler] container on host [3.14.102.175] 
INFO[0103] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [3.14.102.175] 
INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0103] [healthcheck] service [kube-scheduler] on host [3.14.102.175] is healthy 
INFO[0104] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0104] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0104] [controlplane] Successfully started Controller Plane.. 
INFO[0104] [authz] Creating rke-job-deployer ServiceAccount 
INFO[0104] [authz] rke-job-deployer ServiceAccount created successfully 
INFO[0104] [authz] Creating system:node ClusterRoleBinding 
INFO[0104] [authz] system:node ClusterRoleBinding created successfully 
INFO[0104] Successfully Deployed state file at [./cluster.rkestate] 
INFO[0104] [state] Saving full cluster state to Kubernetes 
INFO[0104] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: cluster-state 
INFO[0104] [worker] Building up Worker Plane..          
INFO[0104] [sidekick] Sidekick container already created on host [18.220.62.6] 
INFO[0104] [sidekick] Sidekick container already created on host [3.14.102.175] 
INFO[0104] [sidekick] Sidekick container already created on host [18.217.96.12] 
INFO[0105] [worker] Successfully started [kubelet] container on host [3.14.102.175] 
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [3.14.102.175] 
INFO[0105] [worker] Successfully started [kubelet] container on host [18.220.62.6] 
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [18.220.62.6] 
INFO[0105] [worker] Successfully started [kubelet] container on host [18.217.96.12] 
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [18.217.96.12] 
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [18.220.70.253] 
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [3.17.76.33] 
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [18.188.214.137] 
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253] 
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33] 
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137] 
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33] 
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253] 
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137] 
INFO[0106] [worker] Successfully started [kubelet] container on host [3.17.76.33] 
INFO[0106] [healthcheck] Start Healthcheck on service [kubelet] on host [3.17.76.33] 
INFO[0107] [worker] Successfully started [kubelet] container on host [18.220.70.253] 
INFO[0107] [healthcheck] Start Healthcheck on service [kubelet] on host [18.220.70.253] 
INFO[0107] [worker] Successfully started [kubelet] container on host [18.188.214.137] 
INFO[0107] [healthcheck] Start Healthcheck on service [kubelet] on host [18.188.214.137] 
INFO[0111] [healthcheck] service [kubelet] on host [18.220.62.6] is healthy 
INFO[0111] [healthcheck] service [kubelet] on host [18.217.96.12] is healthy 
INFO[0111] [healthcheck] service [kubelet] on host [3.14.102.175] is healthy 
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0112] [worker] Successfully started [kube-proxy] container on host [18.220.62.6] 
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.220.62.6] 
INFO[0112] [worker] Successfully started [kube-proxy] container on host [18.217.96.12] 
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.217.96.12] 
INFO[0112] [worker] Successfully started [kube-proxy] container on host [3.14.102.175] 
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [3.14.102.175] 
INFO[0113] [healthcheck] service [kube-proxy] on host [18.220.62.6] is healthy 
INFO[0113] [healthcheck] service [kube-proxy] on host [18.217.96.12] is healthy 
INFO[0113] [healthcheck] service [kube-proxy] on host [3.14.102.175] is healthy 
INFO[0113] [healthcheck] service [kubelet] on host [18.220.70.253] is healthy 
INFO[0113] [healthcheck] service [kubelet] on host [3.17.76.33] is healthy 
INFO[0113] [healthcheck] service [kubelet] on host [18.188.214.137] is healthy 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253] 
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137] 
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33] 
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253] 
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137] 
INFO[0114] [worker] Successfully started [kube-proxy] container on host [3.17.76.33] 
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [3.17.76.33] 
INFO[0114] [worker] Successfully started [kube-proxy] container on host [18.220.70.253] 
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.220.70.253] 
INFO[0114] [worker] Successfully started [kube-proxy] container on host [18.188.214.137] 
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.188.214.137] 
INFO[0114] [healthcheck] service [kube-proxy] on host [18.220.70.253] is healthy 
INFO[0114] [healthcheck] service [kube-proxy] on host [3.17.76.33] is healthy 
INFO[0115] [healthcheck] service [kube-proxy] on host [18.188.214.137] is healthy 
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253] 
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33] 
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137] 
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253] 
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33] 
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137] 
INFO[0115] [worker] Successfully started Worker Plane.. 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.188.214.137] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [3.17.76.33] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.220.70.253] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.220.62.6] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.217.96.12] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [3.14.102.175] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [3.17.76.33] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.188.214.137] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.220.62.6] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.220.70.253] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.217.96.12] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [3.14.102.175] 
INFO[0116] [sync] Syncing nodes Labels and Taints       
INFO[0117] [sync] Successfully synced nodes Labels and Taints 
INFO[0117] [network] Setting up network plugin: canal   
INFO[0117] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes 
INFO[0117] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes 
INFO[0117] [addons] Executing deploy job rke-network-plugin 
INFO[0122] [addons] Setting up kube-dns                 
INFO[0122] [addons] Saving ConfigMap for addon rke-kube-dns-addon to Kubernetes 
INFO[0122] [addons] Successfully saved ConfigMap for addon rke-kube-dns-addon to Kubernetes 
INFO[0122] [addons] Executing deploy job rke-kube-dns-addon 
INFO[0127] [addons] kube-dns deployed successfully      
INFO[0127] [dns] DNS provider kube-dns deployed successfully 
INFO[0127] [addons] Setting up Metrics Server           
INFO[0127] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes 
INFO[0127] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes 
INFO[0127] [addons] Executing deploy job rke-metrics-addon 
INFO[0132] [addons] Metrics Server deployed successfully 
INFO[0132] [ingress] Setting up nginx ingress controller 
INFO[0132] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0132] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0132] [addons] Executing deploy job rke-ingress-controller 
INFO[0137] [ingress] ingress controller nginx deployed successfully 
INFO[0137] [addons] Setting up user addons              
INFO[0137] [addons] no user addons defined              
INFO[0137] Finished building Kubernetes cluster successfully

# finish kubectl install
sudo curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.13.5/bin/linux/amd64/kubectl
sudo chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
sudo mkdir ~/.kube

# finish helm
#https://github.com/helm/helm/releases
# there is no helm 2.12.5 - last is 2.12.3 - trying 2.13.1
wget http://storage.googleapis.com/kubernetes-helm/helm-v2.13.1-linux-amd64.tar.gz
sudo tar -zxvf helm-v2.13.1-linux-amd64.tar.gz
sudo cp kube_config_cluster.yml ~/.kube/config
sudo chmod 777 ~/.kube/config

# test
ubuntu@ip-172-31-38-182:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE       NAME                                    READY   STATUS    RESTARTS   AGE   IP               NODE             NOMINATED NODE   READINESS GATES
ingress-nginx   default-http-backend-78fccfc5d9-f6z25   1/1     Running   0          17m   10.42.5.2        18.220.70.253    <none>           <none>
ingress-nginx   nginx-ingress-controller-2zxs7          1/1     Running   0          17m   18.188.214.137   18.188.214.137   <none>           <none>
ingress-nginx   nginx-ingress-controller-6b7gs          1/1     Running   0          17m   3.17.76.33       3.17.76.33       <none>           <none>
ingress-nginx   nginx-ingress-controller-nv4qg          1/1     Running   0          17m   18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-48579                             2/2     Running   0          17m   18.220.62.6      18.220.62.6      <none>           <none>
kube-system     canal-6skkm                             2/2     Running   0          17m   18.188.214.137   18.188.214.137   <none>           <none>
kube-system     canal-9xmxv                             2/2     Running   0          17m   18.217.96.12     18.217.96.12     <none>           <none>
kube-system     canal-c582x                             2/2     Running   0          17m   18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-whbck                             2/2     Running   0          17m   3.14.102.175     3.14.102.175     <none>           <none>
kube-system     canal-xbbnh                             2/2     Running   0          17m   3.17.76.33       3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-6mcm7               3/3     Running   0          17m   10.42.3.3        3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-cd5dg               3/3     Running   0          17m   10.42.4.2        18.188.214.137   <none>           <none>
kube-system     kube-dns-autoscaler-77bc5fd84-p4zfd     1/1     Running   0          17m   10.42.3.2        3.17.76.33       <none>           <none>
kube-system     metrics-server-58bd5dd8d7-kftjn         1/1     Running   0          17m   10.42.3.4        3.17.76.33       <none>           <none>


# install tiller

ubuntu@ip-172-31-38-182:~$ kubectl -n kube-system create serviceaccount tiller
serviceaccount/tiller created
ubuntu@ip-172-31-38-182:~$ kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller created
ubuntu@ip-172-31-38-182:~$ helm init --service-account tiller
Creating /home/ubuntu/.helm 
Creating /home/ubuntu/.helm/repository 
Creating /home/ubuntu/.helm/repository/cache 
Creating /home/ubuntu/.helm/repository/local 
Creating /home/ubuntu/.helm/plugins 
Creating /home/ubuntu/.helm/starters 
Creating /home/ubuntu/.helm/cache/archive 
Creating /home/ubuntu/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /home/ubuntu/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
ubuntu@ip-172-31-38-182:~$ kubectl -n kube-system  rollout status deploy/tiller-deploy
deployment "tiller-deploy" successfully rolled out
ubuntu@ip-172-31-38-182:~$ sudo helm init --upgrade
$HELM_HOME has been configured at /home/ubuntu/.helm.

Tiller (the Helm server-side component) has been upgraded to the current version.
Happy Helming!


ubuntu@ip-172-31-38-182:~$ sudo helm version
Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
ubuntu@ip-172-31-38-182:~$ sudo helm serve &
[1] 706
ubuntu@ip-172-31-38-182:~$ Regenerating index. This may take a moment.
Now serving you on 127.0.0.1:8879

ubuntu@ip-172-31-38-182:~$ sudo helm list
ubuntu@ip-172-31-38-182:~$ sudo helm repo add local http://127.0.0.1:8879
"local" has been added to your repositories
ubuntu@ip-172-31-38-182:~$  sudo helm repo list
NAME  	URL                                             
stable	https://kubernetes-charts.storage.googleapis.com
local 	http://127.0.0.1:8879                           

ubuntu@ip-172-31-38-182:~$ kubectl get nodes -o wide
NAME             STATUS   ROLES               AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
18.188.214.137   Ready    worker              22m   v1.13.5   18.188.214.137   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
18.217.96.12     Ready    controlplane,etcd   22m   v1.13.5   18.217.96.12     <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
18.220.62.6      Ready    controlplane,etcd   22m   v1.13.5   18.220.62.6      <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
18.220.70.253    Ready    worker              22m   v1.13.5   18.220.70.253    <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
3.14.102.175     Ready    controlplane,etcd   22m   v1.13.5   3.14.102.175     <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
3.17.76.33       Ready    worker              22m   v1.13.5   3.17.76.33       <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5


# install make
sudo apt-get install make -y

# install nfs/efs
ubuntu@ip-172-31-38-182:~$ sudo apt-get install nfs-common -y
ubuntu@ip-172-31-38-182:~$ sudo mkdir /dockerdata-nfs
ubuntu@ip-172-31-38-182:~$ sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-5fd6ab26.efs.us-east-2.amazonaws.com:/ /dockerdata-nfs



# check
sudo nohup ./cd.sh -b master -e onap -p false -n nexus3.onap.org:10001 -f false -s 600 -c false -d false -w false -r false &
ubuntu@ip-172-31-38-182:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE       NAME                                                     READY   STATUS             RESTARTS   AGE     IP               NODE             NOMINATED NODE   READINESS GATES
ingress-nginx   default-http-backend-78fccfc5d9-f6z25                    1/1     Running            0          103m    10.42.5.2        18.220.70.253    <none>           <none>
ingress-nginx   nginx-ingress-controller-2zxs7                           1/1     Running            0          103m    18.188.214.137   18.188.214.137   <none>           <none>
ingress-nginx   nginx-ingress-controller-6b7gs                           1/1     Running            0          103m    3.17.76.33       3.17.76.33       <none>           <none>
ingress-nginx   nginx-ingress-controller-nv4qg                           1/1     Running            0          103m    18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-48579                                              2/2     Running            0          103m    18.220.62.6      18.220.62.6      <none>           <none>
kube-system     canal-6skkm                                              2/2     Running            0          103m    18.188.214.137   18.188.214.137   <none>           <none>
kube-system     canal-9xmxv                                              2/2     Running            0          103m    18.217.96.12     18.217.96.12     <none>           <none>
kube-system     canal-c582x                      18.06.3-ce  API version:       1.38  Go version:        go1.10.3  Git commit:2/2     Running   d7080c1  Built:       0      Wed Feb 20 02:27:18 2019103m   # install RKE
sudo wget https://github.com/rancher/rke/releases/download/v0.1.16/rke_linux-amd64
mv rke_linux-amd64 rke
sudo mv ./rke /usr/local/bin/rke

ubuntu@a-rke:~$ rke --version
rke version v0.1.16

...

scp your key to the box - ideally to ~/.ssh and chmod 400 it

Elastic Reserved IP

get a VIP or EIP

generate cluster.yaml

Code Block

theme	Midnight

azure config - no need to hand build the yaml
Watch the path of your 2 keys
Also don't add an "addon" until you have one of the config job will fail

{noformat}
ubuntu@a-rke:~$ rke config --name cluster.yml
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/onap_rsa
[+] Number of Hosts [1]: 
[+] SSH Address of host (1) [none]: rke.onap.cloud
[+] SSH Port of host (1) [22]: 
[+] SSH Private Key Path of host (rke.onap.cloud) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (rke.onap.cloud) [ubuntu]: 
[+] Is host (rke.onap.cloud) a Control Plane host (y/n)? [y]: y
[+] Is host (rke.onap.cloud) a Worker host (y/n)? [n]: y
[+] Is host (rke.onap.cloud) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (rke.onap.cloud) [none]: 
[+] Internal IP of host (rke.onap.cloud) [none]: 
[+] Docker socket path on host (rke.onap.cloud) [/var/run/docker.sock]: 
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]: 
[+] Authentication Strategy [x509]: 
[+] Authorization Mode (rbac, none) [rbac]: 
[+] Kubernetes Docker image [rancher/hyperkube:v1.11.6-rancher1]: 
[+] Cluster domain [cluster.local]: 
[+] Service Cluster IP Range [10.43.0.0/16]: 
[+] Enable PodSecurityPolicy [n]: 
[+] Cluster Network CIDR [10.42.0.0/16]: 
[+] Cluster DNS Service IP [10.43.0.10]: 
[+] Add addon manifest URLs or YAML files [no]: no
ubuntu@a-rke:~$ sudo cat cluster.yml 
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: rke.onap.cloud
  port: "22"
  internal_address: ""
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: ""
  user: ubuntu
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/onap_rsa
  labels: {}
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    snapshot: null
    retention: ""
    creation: ""
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: canal
  options: {}
authentication:
  strategy: x509
  options: {}
  sans: []
system_images:
  etcd: rancher/coreos-etcd:v3.2.18
  alpine: rancher/rke-tools:v0.1.15
  nginx_proxy: rancher/rke-tools:v0.1.15
  cert_downloader: rancher/rke-tools:v0.1.15
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.15
  kubedns: rancher/k8s-dns-kube-dns-amd64:1.14.10
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny-amd64:1.14.10
  kubedns_sidecar: rancher/k8s-dns-sidecar-amd64:1.14.10
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler-amd64:1.0.0
  kubernetes: rancher/hyperkube:v1.11.6-rancher1
  flannel: rancher/coreos-flannel:v0.10.0
  flannel_cni: rancher/coreos-flannel-cni:v0.3.0
  calico_node: rancher/calico-node:v3.1.3
  calico_cni: rancher/calico-cni:v3.1.3
  calico_controllers: ""
  calico_ctl: rancher/calico-ctl:v2.0.0
  canal_node: rancher/calico-node:v3.1.3
  canal_cni: rancher/calico-cni:v3.1.3
  canal_flannel: rancher/coreos-flannel:v0.10.0
  wave_node: weaveworks/weave-kube:2.1.2
  weave_cni: weaveworks/weave-npc:2.1.2
  pod_infra_container: rancher/pause-amd64:3.1
  ingress: rancher/nginx-ingress-controller:0.16.2-rancher1
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.4
  metrics_server: rancher/metrics-server-amd64:v0.2.1
ssh_key_path: ~/.ssh/onap_rsa
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
  provider: ""
  options: {}
  node_selector: {}
  extra_args: {}
cluster_name: ""
cloud_provider:
  name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""
monitoring:
  provider: ""
  options: {}
{noformat}

Kubernetes Single Node Developer Installation

Code Block

theme	Midnight

sudo chmod 777 cluster.yml
rke up
# install kubectl
sudo curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.11.6/bin/linux/amd64/kubectl
sudo chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
sudo mkdir ~/.kube
sudo cp kube_config_cluster.yml ~/.kube/config
sudo chmod 777 ~/.kube/config 
kubectl get pods --all-namespaces
NAMESPACE       NAME    18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-whbck                                              2/2     Running            0          103m    3.14.102.175     3.14.102.175     <none>           <none>
kube-system     canal-xbbnh                                              2/2     Running            0          103m    3.17.76.33       3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-6mcm7                                3/3     Running            0          103m    10.42.3.3        3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-cd5dg                                3/3     Running            0          103m    10.42.4.2        18.188.214.137   <none>           <none>
kube-system     kube-dns-autoscaler-77bc5fd84-p4zfd                      1/1     Running            0          103m    10.42.3.2        3.17.76.33       <none>           <none>
kube-system     metrics-server-58bd5dd8d7-kftjn                          1/1     Running            0          103m    10.42.3.4        3.17.76.33       <none>           <none>
kube-system     tiller-deploy-5f4fc5bcc6-gc4tc                           1/1     Running            0          84m     10.42.5.3        18.220.70.253    <none>           <none>
onap            onap-aai-aai-587cb79c6d-mzpbs                            0/1     Init:0/1           1          13m     10.42.5.17       18.220.70.253    <none>           <none>
onap            onap-aai-aai-babel-8c755bcfc-kmzdm                       2/2     Running            0          29m     10.42.5.8        18.220.70.253    <none>           <none>
onap            onap-aai-aai-champ-78b9d7f68b-98tm9             READY     STATUS    0/2  RESTARTS   AGE
ingress-nginxInit:0/1    default-http-backend-797c5bc547-45msr       1/12       Running   29m  0   10.42.3.6       17m
ingress-nginx 3.17.76.33   nginx-ingress-controller-dfhp8    <none>          1/1 <none>
onap     Running     0  onap-aai-aai-data-router-64fcfbc5bb-wkkvz        17m kube-system     canal-lc6g6  1/2     CrashLoopBackOff   9          29m     10.42.5.7        3/318.220.70.253    <none>   Running     0   <none>
onap      17m kube-system     kube-dns-7588d5b5f5-6k286onap-aai-aai-elasticsearch-6dcf5d9966-j7z67              1/1     3/3Running       Running     0          17m29m kube-system     kube-dns-autoscaler-5db9bbb766-6slz710.42.4.4         1/118.188.214.137   <none>    Running     0   <none>
onap      17m kube-system     metrics-server-97bc649d5-q84tzonap-aai-aai-gizmo-5bddb87589-zn8pl              1/1       Running 2/2    0 Running         17m kube-system   0  rke-ingress-controller-deploy-job-5q2w7     0/1   29m    Completed 10.42.4.3  0      18.188.214.137    17m
kube-system <none>    rke-kubedns-addon-deploy-job-7vq49       <none>
onap  0/1       Completed   0onap-aai-aai-graphadmin-774f9d698f-f8lwv          17m kube-system     rke-metrics-addon-deploy-job-2hnbl  0/2     Init:0/1    0/1       Completed2   0       29m   17m kube-system  10.42.5.4    rke-network-plugin-deploy-job-6fzt2     18.220.70.253    <none>    0/1       Completed<none>
onap  0          17m

Kubernetes HA Cluster Production Installation

Design Issues

DI 20190225-1: RKE/Docker version pair

As of 20190215 RKE 0.16 supports Docker 18.06-ce (and 18.09 non-ce) (up from 0.15 supporting 17.03)

https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce

https://github.com/rancher/rke/releases/tag/v0.1.16

Code Block

theme	Midnight

ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster onap-aai-aai-graphadmin-create-db-schema-94q4l           0/1     Init:Error         0         INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] 
FATA[0000] Unsupported Docker version found [18.06.3-ce], supported versions are [1.11.x 1.12.x 1.13.x 17.03.x]

DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working

Does rke remove, regenerate the yaml (or hand upgrade the versions) then rke up

Code Block

theme	Midnight

ubuntu@a-rke:~$ sudo rke remove
Are you sure you want to remove Kubernetes cluster [y/n]: y
INFO[0002] Tearing down Kubernetes cluster 18m     10.42.4.16       18.188.214.137   <none>           <none>
onap            onap-aai-aai-graphadmin-create-db-schema-s42pq           0/1     Init:0/1           0       INFO[0002] [dialer] Setup tunnel7m54s for host [rke.onap.cloud] 
INFO[0002] [worker] Tearing down Worker Plane..10.42.5.20       18.220.70.253    <none>           <none>
onap   INFO[0002] [remove/kubelet] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/kube-proxy] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [worker] Successfully tore down Worker Plane.. 
INFO[0003] [controlplane] Tearing down the Controller Plane.. 
INFO[0003] [remove/kube-apiserver] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/kube-controller-manager] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [remove/kube-scheduler] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [controlplane] Host [rke.onap.cloud] is already a worker host, skipping delete kubelet and kubeproxy. 
INFO[0004] [controlplane] Successfully tore down Controller Plane.. 
INFO[0004] [etcd] Tearing down etcd plane..               onap-aai-aai-graphadmin-create-db-schema-tsvcw           0/1     Init:Error         0          29m     10.42.4.5        18.188.214.137   <none>           <none>
onap            onap-aai-aai-modelloader-845fc684bd-r7mdw                2/2     Running            0        INFO[0004] [remove/etcd] Successfully29m removed container on host [rke.onap.cloud] 
INFO[0004] [etcd] Successfully tore down etcd plane..   
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]10.42.4.6        18.188.214.137   <none>           <none>
onap       INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]onap-aai-aai-resources-67f8dfcbdb-kz6cp             INFO[0004] [hosts] Running cleaner container on0/2 host [rke.onap.cloud]  INFO[0005] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0005] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0005] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0006] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0006] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0007] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0008] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0008] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0008] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0009] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0009] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0009] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0010] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0010] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0010] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0011] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0011] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0011] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0011] Removing local admin Kubeconfig: ./kube_config_cluster.yml 
INFO[0011] Local admin Kubeconfig removed successfully  
INFO[0011] Cluster removed successfully  

ubuntu@a-rke:~$ rke config --name cluster.ym
ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster Init:0/1           2          29m     10.42.5.11       18.220.70.253    <none>           <none>
onap            onap-aai-aai-schema-service-6c56b45b7c-7zlfz             2/2     Running            0          29m     10.42.3.7        3.17.76.33       <none>           <none>
onap            onap-aai-aai-search-data-5d8d7759b8-flxwj                2/2     Running            0          29m     10.42.3.9        3.17.76.33       <none>           <none>
onap            onap-aai-aai-sparky-be-8444df749c-mzc2n                  0/2     Init:0/1           0          29m     10.42.4.11     INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] 
INFO[0000] [network] Deploying port listener containers 
INFO[0001] [network] Successfully started [rke-etcd-port-listener] container on host [rke.onap.cloud] 
INFO[0001] [network] Successfully started [rke-cp-port-listener] container on host [rke.onap.cloud] 
INFO[0002] [network] Successfully started [rke-worker-port-listener] container on host [rke.onap.cloud] 
INFO[0002] [network] Port listener containers deployed successfully 
INFO[0002] [network] Running control plane -> etcd port checks 
INFO[0003] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0003] [network] Running control plane -> worker port checks 
INFO[0004] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0004] [network] Running workers -> control plane port checks 
INFO[0005] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0005] [network] Checking KubeAPI port Control Plane hosts 
INFO[0005] [network] Removing port listener containers  
INFO[0005] [remove/rke-etcd-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-cp-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-worker-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [network] Port listener containers removed successfully 
INFO[0006] [certificates] Attempting to recover certificates from backup on [etcd,controlPlane] hosts 
INFO[0007] [certificates] No Certificate backup found on [etcd,controlPlane] hosts 
INFO[0007] [certificates] Generating CA kubernetes certificates 
INFO[0007] [certificates] Generating Kubernetes API server certficates 
INFO[0008] [certificates] Generating Kube Controller certificates 
INFO[0008] [certificates] Generating Kube Scheduler certificates 
INFO[0008] [certificates] Generating Kube Proxy certificates 
INFO[0009] [certificates] Generating Node certificate   
INFO[0009] [certificates] Generating admin certificates and kubeconfig 
INFO[0009] [certificates] Generating etcd-rke.onap.cloud certificate and key 
INFO[0009] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0009] [certificates] Generating Kubernetes API server proxy client certificates 
INFO[0010] [certificates] Temporarily saving certs to [etcd,controlPlane] hosts 
INFO[0016] [certificates] Saved certs to [etcd,controlPlane] hosts 
INFO[0016] [reconcile] Reconciling cluster state 18.188.214.137   <none>           <none>
onap            onap-aai-aai-spike-54ff77787f-d678x                      2/2     Running            0          29m     10.42.5.6        18.220.70.253    <none>           <none>
onap            onap-aai-aai-traversal-6ff868f477-lzv2f                  0/2     Init:0/1           2          29m     10.42.3.8        3.17.76.33       <none>           <none>
onap            onap-aai-aai-traversal-update-query-data-9g2b8           0/1     Init:0/1           2          29m     10.42.5.12       18.220.70.253    <none>           <none>
onap            onap-dmaap-dbc-pg-0                   INFO[0016] [reconcile] This is newly generated cluster   INFO[0016] [certificates] Deploying kubernetes certificates to Cluster nodes  INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes 
INFO[0022] Pre-pulling kubernetes images 1/1     Running            0          29m     10.42.5.9      INFO[0022] Kubernetes images pulled successfully   18.220.70.253    <none>       INFO[0022] [etcd] Building up etcd<none>
plane..onap            onap-dmaap-dbc-pg-1   INFO[0023] [etcd] Successfully started [etcd] container on host [rke.onap.cloud]  INFO[0023] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [rke.onap.cloud]  INFO[0028] [certificates] Successfully started [rke-bundle-cert] container on host [rke.onap.cloud]  INFO[0029] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [rke.onap.cloud] 
INFO[0029] [etcd] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0030] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0030] [etcd] Successfully started etcd plane..     
INFO[0030] [controlplane] Building up Controller Plane.. 
INFO[0031] [controlplane] Successfully started [kube-apiserver] container on host [rke.onap.cloud] 
INFO[0031] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [rke.onap.cloud] 
INFO[0045] [healthcheck] service [kube-apiserver] on host [rke.onap.cloud] is healthy 
INFO[0046] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0046] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0047] [controlplane] Successfully started [kube-controller-manager] container on host [rke.onap.cloud] 
INFO[0047] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [rke.onap.cloud] 
INFO[0052] [healthcheck] service [kube-controller-manager] on host [rke.onap.cloud] is healthy 
INFO[0053] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0053] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0054] [controlplane] Successfully started [kube-scheduler] container on host [rke.onap.cloud] 
INFO[0054] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [rke.onap.cloud] 
INFO[0059] [healthcheck] service [kube-scheduler] on host [rke.onap.cloud] is healthy 
INFO[0060] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0060] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0060] [controlplane] Successfully started Controller Plane.. 
INFO[0060] [authz] Creating rke-job-deployer ServiceAccount 
INFO[0060] [authz] rke-job-deployer ServiceAccount created successfully 
INFO[0060] [authz] Creating system:node ClusterRoleBinding 
INFO[0060] [authz] system:node ClusterRoleBinding created successfully 
INFO[0060] [certificates] Save kubernetes certificates as secrets 
INFO[0060] [certificates] Successfully saved certificates as kubernetes secret [k8s-certs] 
INFO[0060] [state] Saving cluster state to Kubernetes   
INFO[0061] [state] Successfully Saved cluster state to Kubernetes ConfigMap: cluster-state 
INFO[0061] [state] Saving cluster state to cluster nodes 
INFO[0061] [state] Successfully started [cluster-state-deployer] container on host [rke.onap.cloud] 
INFO[0062] [remove/cluster-state-deployer] Successfully removed container on host [rke.onap.cloud] 
INFO[0062] [worker] Building up Worker Plane..          
INFO[0062] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] 
INFO[0063] [worker] Successfully started [kubelet] container on host [rke.onap.cloud] 
INFO[0063] [healthcheck] Start Healthcheck on service [kubelet] on host [rke.onap.cloud] 
INFO[0068] [healthcheck] service [kubelet] on host [rke.onap.cloud] is healthy 
INFO[0069] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0070] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0070] [worker] Successfully started [kube-proxy] container on host [rke.onap.cloud] 
INFO[0070] [healthcheck] Start Healthcheck on service [kube-proxy] on host [rke.onap.cloud] 
INFO[0076] [healthcheck] service [kube-proxy] on host [rke.onap.cloud] is healthy 
INFO[0076] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0077] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0077] [worker] Successfully started Worker Plane.. 
INFO[0077] [sync] Syncing nodes Labels and Taints       
INFO[0077] [sync] Successfully synced nodes Labels and Taints 
INFO[0077] [network] Setting up network plugin: canal   
INFO[0077] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0077] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-network-plugin 
INFO[0077] [addons] Executing deploy job..      1/1     Running            0          26m     10.42.3.14       3.17.76.33       <none>           <none>
onap            onap-dmaap-dbc-pgpool-8666b57857-97zjc                   1/1     Running            0          29m     10.42.5.5        18.220.70.253    <none>           <none>
onap            onap-dmaap-dbc-pgpool-8666b57857-vr8gk                   1/1     Running            0          29m     10.42.4.8        18.188.214.137   <none>           <none>
onap            onap-dmaap-dmaap-bc-745995bf74-m6hhq                     0/1     Init:0/2           2          29m     10.42.4.12       18.188.214.137   <none>           <none>
onap            onap-dmaap-dmaap-bc-post-install-6ff4j                   1/1     Running            0          29m     10.42.4.9        18.188.214.137   <none>           <none>
onap            onap-dmaap-dmaap-dr-db-0                                 1/1     Running            0          29m     10.42.4.10       18.188.214.137   <none>           <none>
onap         INFO[0082] [addons] Setting up KubeDNS onap-dmaap-dmaap-dr-db-1                    INFO[0082] [addons] Saving addon ConfigMap to Kubernetes  INFO[0082] [addons] Successfully Saved addon to1/1 Kubernetes ConfigMap: rke-kubedns-addon  INFO[0082] [addons] Executing deploy job..Running            1       INFO[0087] [addons] KubeDNS deployed24m successfully..    10.42.5.15  INFO[0087] [addons] Setting up Metrics Server 18.220.70.253    <none>        INFO[0087] [addons] Saving addon<none>
ConfigMaponap to Kubernetes  INFO[0087] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-metrics-addon 
INFO[0087] [addons] Executing deploy job.. onap-dmaap-dmaap-dr-node-0                       INFO[0092] [addons] KubeDNS deployed successfully..    2/2  INFO[0092] [ingress] Setting upRunning nginx ingress controller  INFO[0092] [addons] Saving addon ConfigMap to Kubernetes 0 INFO[0092] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-ingress-controller 29m INFO[0092] [addons] Executing deploy job10.42.3.11       3.17.76.33       <none>    INFO[0097] [ingress] ingress controller nginx is successfully deployed<none>
onap INFO[0097] [addons] Setting up user addons      onap-dmaap-dmaap-dr-prov-fbf6c94f5-v9bmq         INFO[0097] [addons] Checking for included user addons  2/2  WARN[0097] [addons] Unable toRunning determine if  is a file path or url, skipping  INFO[0097] [addons] Deploying rke-user-includes-addons0      INFO[0097] [addons] Saving addon ConfigMap29m to Kubernetes  INFO[0097] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-user-includes-addons 
INFO[0097] [addons] Executing deploy job.. 10.42.5.10       18.220.70.253    <none>           <none>
onap     WARN[0128] Failed to deploy addon execute job [rke-user-includes-addons]: Failed to get job complete status: <nil> 
INFO[0128] Finished building Kubernetes cluster successfully 

ubuntu@a-rke:~$ sudo docker ps
CONTAINER ID    onap-dmaap-message-router-0                              1/1     Running     IMAGE       0          29m     10.42.4.14       18.188.214.137   COMMAND<none>           <none>
onap      CREATED      onap-dmaap-message-router-kafka-0       STATUS              PORTS   1/1     Running       NAMES ec26c4bd24b5    1    846921f0fe0e      29m     10.42.5.13       18.220.70.253    <none>   "/server"        <none>
onap       10 minutes ago   onap-dmaap-message-router-kafka-1   Up 10 minutes                   1/1     Running   k8s_default-http-backend_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0 f8d5db205e14        8a7739f672b41          29m     10.42.3.13       3.17.76.33   "/sidecar --v=2 --lo…"   10 minutes ago    <none>          Up 10<none>
minutesonap            onap-dmaap-message-router-kafka-2               k8s_sidecar_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0 490461545ae4        rancher/metrics-server-amd64 1/1     Running        "/metrics-server --s…"   100 minutes ago      Up 10 minutes29m     10.42.4.15       18.188.214.137   <none>            k8s_metrics-server_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
aaf03b62bd41<none>
onap         6816817d9dce   onap-dmaap-message-router-mirrormaker-8587c4c9cf-lfnd8   0/1     CrashLoopBackOff   9          29m "/dnsmasq-nanny -v=2…"   10 minutes ago.42.4.7       Up 10 minutes 18.188.214.137   <none>           <none>
onap             k8s_dnsmasq_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
58ec007db72fonap-dmaap-message-router-zookeeper-0              55ffe31ac578      1/1     Running            0  "/kube-dns --domain=…"   10 minutes ago  29m    Up 10 minutes.42.5.14       18.220.70.253    <none>           <none>
onap            k8s_kubedns_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
0a95c06f6aa6onap-dmaap-message-router-zookeeper-1               e183460c484d     1/1     Running            0   "/cluster-proportion…"   10 minutes ago  29m    Up 10 minutes.42.4.13       18.188.214.137   <none>           <none>
onap            k8s_autoscaler_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
968a7c99b210onap-dmaap-message-router-zookeeper-2            rancher/pause-amd64:3.1        1/1     Running "/pause"           0      10 minutes ago  29m    Up 10 minutes.42.3.12       3.17.76.33       <none>           <none>
onap     k8s_POD_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
69969b331e49       onap-nfs-provisioner-nfs-provisioner-57c999dc57-mdcw5    rancher/pause-amd64:3.1/1     Running         "/pause"   0          24m     10 minutes ago.42.3.15       3.17.76.33  Up 10 minutes   <none>           <none>
onap            k8s_POD_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
baa5f03c16ffonap-robot-robot-677bdbb454-zj9jk          rancher/pause-amd64:3.1              "/pause"1/1     Running            0      10 minutes ago  24m    Up 10 minutes.42.5.16       18.220.70.253    <none>           <none>
onap            k8s_POD_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
82b2a9f640cbonap-so-so-8569947cbd-jn5x4                     rancher/pause-amd64:3.1         0/1     "/pause"Init:0/1           1      10 minutes ago  13m    Up 10 minutes.42.4.19       18.188.214.137   <none>           <none>
onap            k8s_POD_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
953a4d4be0c1onap-so-so-bpmn-infra-78c8fd665d-b47qn                   0/1  df4469c42185   Init:0/1           1           "/usr/bin/dumb-init …"13m     10.42.3.16 minutes ago      Up 10 minutes3.17.76.33       <none>           <none>
onap            k8s_nginxonap-ingressso-controller_nginxso-ingresscatalog-controllerdb-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
cce552840749adapter-565f9767ff-lvbgx           rancher/pause-amd64:3.0/1     Init:0/1         "/pause"  1          13m     10.42.3.17 minutes ago     3.17.76.33 Up 10 minutes    <none>           <none>
onap            k8s_POD_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
baa65f9c6f97onap-so-so-mariadb-config-job-d9sdb                      f0fad859c9090/1     Init:0/2           0         "/opt/bin/flanneld -…" 3m37s   10 minutes ago.42.3.20       3.17.76.33   Up 10 minutes  <none>           <none>
onap             k8s_kube-flannel_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1736ce68f41aonap-so-so-mariadb-config-job-rkqdl         9f355e076ea7             0/1     Init:Error       "/install-cni.sh"  0      10 minutes ago  13m    Up 10 minutes.42.5.19       18.220.70.253    <none>           <none>
onap            k8s_install-cni_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
615d3f702ee7onap-so-so-monitoring-69b9fdd94c-dks4v           7eca10056c8e        1/1     Running            "start_runit"0          13m  10 minutes ago 10.42.4.18     Up 10 minutes 18.188.214.137   <none>           <none>
onap            k8s_calico-node_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1c4a702f0f18onap-so-so-openstack-adapter-5f9cf896d7-mgbdd            rancher/pause-amd64:3.0/1     Init:0/1         "/pause"  1          13m     10.42.4.17 minutes ago     18.188.214.137 Up 10 minutes<none>           <none>
onap               k8s_POD_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
0da1cada08e1onap-so-so-request-db-adapter-5c9bfd7b57-2krnp         rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"0/1   10 minutes agoInit:0/1      Up 10 minutes   1          13m     10.42.3.18         kube-proxy
57f44998f34a3.17.76.33         rancher/hyperkube:v1.11.6-rancher1 <none>   "/opt/rke-tools/entr…"   11 minutes ago   <none>
onap  Up 11 minutes        onap-so-so-sdc-controller-6fb5cf5775-bsxhm               0/1    kubelet 50f424c4daecInit:0/1        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"1   11 minutes ago     13m Up 11 minutes  10.42.4.20       18.188.214.137   <none>           <none>
onap   kube-scheduler 502d327912d9        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes agoonap-so-so-sdnc-adapter-8555689c75-r6vkb                 1/1 Up  11 minutes Running            0          13m    kube-controller-manager
9fc706bbf3a5 10.42.5.18        rancher/hyperkube:v1.11.6-rancher118.220.70.253   "/opt/rke-tools/entr…"  <none>  11 minutes ago      Up 11<none>
minutesonap            onap-so-so-vfc-adapter-68fccc8bb8-c56t2               kube-apiserver 2e7630c2047c  0/1      rancher/coreos-etcd:v3.2.18Init:0/1           "/usr/local/bin/etcd…"1   11 minutes ago     13m Up 11 minutes  10.42.4.21       18.188.214.137   <none>           <none>
onap   etcd fef566337eb6        rancher/rke-tools:v0.1.15onap-so-so-vnfm-adapter-65c4c5944b-72nlf                 "/opt/rke-tools/rke-…"1/1   26 minutes agoRunning      Up 26 minutes     0          13m     10.42.3.19       etcd-rolling-snapshots

DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user

Code Block

theme	Midnight

amdocs@obriensystemsu0:~$ sudo rke up
Segmentation fault (core dumped)


# issue was I was using ubuntu as the yml user not amdocs in this case for a particular VM

...

3.17.76.33       <none>           <none>


# on worker nodes only
# nfs client

DI 20190507: ARM support using RKE 0.2.1 ARM friendly install

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	LOG-331

Instance type(s) a1.4xlarge $0.408

AMI ID ami-0b9bd0b532ebcf4c9

Notes

Pre-RKE installation details in Cloud Native Deployment

...

Version	Old Version 22	New Version Current
Changes made by	Michael O'Brien	Michael O'Brien
Saved on	Feb 28, 2019	May 07, 2019

Content Comparison

Versions Compared

Key

Automated Installation Video of RKE install

20190227 - VMware

20190330 - AWS

Versions

Prerequisites

Ubuntu 16.04 VM

Determine RKE and Docker versions

Quickstart

Versions

Prerequisites

Ubuntu 16.04 VM

Determine RKE and Docker versions

Private SSH key

Elastic Reserved IP

generate cluster.yml - optional

Kubernetes Single Node Developer Installation

Kubernetes HA Cluster Production Installation

Design Issues

DI 20190225-1: RKE/Docker version pair

DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working

DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user

DI 20190227-1: Verify no 110 pod limit per VM

DI 20190228-1: deploy casablanca MR to RKE under K8S 1.11.6, Docker 18.06, Helm 2.12.3

DI 20190305-1: Azure 256G VM full ONAP Testing

DI 20190425: HA RKE install Testing

Elastic Reserved IP

generate cluster.yaml

Kubernetes Single Node Developer Installation

Kubernetes HA Cluster Production Installation

Design Issues

DI 20190225-1: RKE/Docker version pair

DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working

DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user

DI 20190507: ARM support using RKE 0.2.1 ARM friendly install

Notes