Tracking OOM-1598 - Getting issue details... STATUS

NON-HA version in https://git.onap.org/oom/tree/kubernetes/contrib/tools/rke/rke_setup.sh

https://gerrit.onap.org/r/#/c/79067/

Move to https://onap.readthedocs.io/en/beijing/submodules/oom.git/docs/oom_cloud_setup_guide.html or similar when this documentation is released

https://lists.onap.org/g/onap-discuss/topic/rke_ha_collaboration_for/31313969?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,31313969

Automated Installation Video of RKE install

20190227 - VMware

20190330 - AWS

Quickstart

Get your public and private keys on the Ubuntu 16.04 VM.

Adjust authorized_keys with your public key if not already - aws has it, openstack may not

get rke script from jira, gerrit or by cloning OOM when the review is done.

# on your laptop/where your cert is
# chmod 777 your cert before you scp it over
obrienbiometrics:full michaelobrien$ scp ~/wse_onap/onap_rsa ubuntu@rke0.onap.info:~/

# on the host
sudo cp onap_rsa ~/.ssh
sudo chmod 400 ~/.ssh/onap_rsa
sudo chown ubuntu:ubuntu ~/.ssh/onap_rsa 
# just verify
sudo vi ~/.ssh/authorized_keys

git clone --recurse-submodules https://gerrit.onap.org/r/oom
sudo cp oom/kubernetes/contrib/tools/rke/rke_setup.sh .
sudo nohup ./rke_setup.sh -b master -s 104.209.161.210 -e onap -k onap_rsa -l ubuntu &

ubuntu@a-rke0-master:~$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                      READY     STATUS      RESTARTS   AGE
ingress-nginx   default-http-backend-797c5bc547-55fpn     1/1       Running     0          4m
ingress-nginx   nginx-ingress-controller-znhgz            1/1       Running     0          4m
kube-system     canal-dqt2m                               3/3       Running     0          5m
kube-system     kube-dns-7588d5b5f5-pzdfh                 3/3       Running     0          5m
kube-system     kube-dns-autoscaler-5db9bbb766-b7vvg      1/1       Running     0          5m
kube-system     metrics-server-97bc649d5-fmqjd            1/1       Running     0          4m
kube-system     rke-ingress-controller-deploy-job-dxmbd   0/1       Completed   0          4m
kube-system     rke-kubedns-addon-deploy-job-wqccp        0/1       Completed   0          5m
kube-system     rke-metrics-addon-deploy-job-ssrgp        0/1       Completed   0          4m
kube-system     rke-network-plugin-deploy-job-jkffq       0/1       Completed   0          5m
kube-system     tiller-deploy-759cb9df9-rlt7v             1/1       Running     0          2m
ubuntu@a-rke0-master:~$ helm list

Versions

Currently Docker 18.06, RKE 0.1.16, Kubernetes 1.11.6, Kubectl 1.11.6, Helm 2.12.3

TODO: verify later versions of helm and a way to get RKE to install Kubernetes 1.13

Prerequisites

Ubuntu 16.04 VM

Determine RKE and Docker versions

Don't just use the latest docker version - check the RKE release page to get the version pair - 0.1.15/17.03 and 0.1.16/18.06 - see https://github.com/docker/docker-ce/releases - currently https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce

ubuntu@a-rke:~$ sudo curl https://releases.rancher.com/install-docker/18.06.sh | sh
ubuntu@a-rke:~$ sudo usermod -aG docker ubuntu
ubuntu@a-rke:~$ sudo docker version
Client:
 Version:           18.06.3-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        d7080c1
 Built:             Wed Feb 20 02:27:18 2019

# install RKE
sudo wget https://github.com/rancher/rke/releases/download/v0.1.16/rke_linux-amd64
mv rke_linux-amd64 rke
sudo mv ./rke /usr/local/bin/rke

ubuntu@a-rke:~$ rke --version
rke version v0.1.16

Private SSH key

scp your public key to the box - ideally to ~/.ssh and chmod 400 it - make sure you add your key to authorized_keys

Elastic Reserved IP

get a VIP or EIP and assign this to your VM

generate cluster.yml - optional

cluster.yml will generated by the script rke_setup.sh

azure config - no need to hand build the yml
Watch the path of your 2 keys
Also don't add an "addon" until you have one of the config job will fail

{noformat}
ubuntu@a-rke:~$ rke config --name cluster.yml
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/onap_rsa
[+] Number of Hosts [1]: 
[+] SSH Address of host (1) [none]: rke.onap.cloud
[+] SSH Port of host (1) [22]: 
[+] SSH Private Key Path of host (rke.onap.cloud) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (rke.onap.cloud) [ubuntu]: 
[+] Is host (rke.onap.cloud) a Control Plane host (y/n)? [y]: y
[+] Is host (rke.onap.cloud) a Worker host (y/n)? [n]: y
[+] Is host (rke.onap.cloud) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (rke.onap.cloud) [none]: 
[+] Internal IP of host (rke.onap.cloud) [none]: 
[+] Docker socket path on host (rke.onap.cloud) [/var/run/docker.sock]: 
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]: 
[+] Authentication Strategy [x509]: 
[+] Authorization Mode (rbac, none) [rbac]: 
[+] Kubernetes Docker image [rancher/hyperkube:v1.11.6-rancher1]: 
[+] Cluster domain [cluster.local]: 
[+] Service Cluster IP Range [10.43.0.0/16]: 
[+] Enable PodSecurityPolicy [n]: 
[+] Cluster Network CIDR [10.42.0.0/16]: 
[+] Cluster DNS Service IP [10.43.0.10]: 
[+] Add addon manifest URLs or YAML files [no]: no
ubuntu@a-rke:~$ sudo cat cluster.yml 
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: rke.onap.cloud
  port: "22"
  internal_address: ""
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: ""
  user: ubuntu
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/onap_rsa
  labels: {}
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    snapshot: null
    retention: ""
    creation: ""
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: canal
  options: {}
authentication:
  strategy: x509
  options: {}
  sans: []
system_images:
  etcd: rancher/coreos-etcd:v3.2.18
  alpine: rancher/rke-tools:v0.1.15
  nginx_proxy: rancher/rke-tools:v0.1.15
  cert_downloader: rancher/rke-tools:v0.1.15
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.15
  kubedns: rancher/k8s-dns-kube-dns-amd64:1.14.10
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny-amd64:1.14.10
  kubedns_sidecar: rancher/k8s-dns-sidecar-amd64:1.14.10
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler-amd64:1.0.0
  kubernetes: rancher/hyperkube:v1.11.6-rancher1
  flannel: rancher/coreos-flannel:v0.10.0
  flannel_cni: rancher/coreos-flannel-cni:v0.3.0
  calico_node: rancher/calico-node:v3.1.3
  calico_cni: rancher/calico-cni:v3.1.3
  calico_controllers: ""
  calico_ctl: rancher/calico-ctl:v2.0.0
  canal_node: rancher/calico-node:v3.1.3
  canal_cni: rancher/calico-cni:v3.1.3
  canal_flannel: rancher/coreos-flannel:v0.10.0
  wave_node: weaveworks/weave-kube:2.1.2
  weave_cni: weaveworks/weave-npc:2.1.2
  pod_infra_container: rancher/pause-amd64:3.1
  ingress: rancher/nginx-ingress-controller:0.16.2-rancher1
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.4
  metrics_server: rancher/metrics-server-amd64:v0.2.1
ssh_key_path: ~/.ssh/onap_rsa
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
  provider: ""
  options: {}
  node_selector: {}
  extra_args: {}
cluster_name: ""
cloud_provider:
  name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""
monitoring:
  provider: ""
  options: {}
{noformat}

Kubernetes Single Node Developer Installation

sudo ./rke_install.sh -b master -s localhost -e onap -l ubuntu

Kubernetes HA Cluster Production Installation

Design Issues

DI 20190225-1: RKE/Docker version pair

As of 20190215 RKE 0.16 supports Docker 18.06-ce (and 18.09 non-ce) (up from 0.15 supporting 17.03)

https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce

https://github.com/rancher/rke/releases/tag/v0.1.16

ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster                  
INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] 
FATA[0000] Unsupported Docker version found [18.06.3-ce], supported versions are [1.11.x 1.12.x 1.13.x 17.03.x]

DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working

Does rke remove, regenerate the yaml (or hand upgrade the versions) then rke up

ubuntu@a-rke:~$ sudo rke remove
Are you sure you want to remove Kubernetes cluster [y/n]: y
INFO[0002] Tearing down Kubernetes cluster              
INFO[0002] [dialer] Setup tunnel for host [rke.onap.cloud] 
INFO[0002] [worker] Tearing down Worker Plane..         
INFO[0002] [remove/kubelet] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/kube-proxy] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [worker] Successfully tore down Worker Plane.. 
INFO[0003] [controlplane] Tearing down the Controller Plane.. 
INFO[0003] [remove/kube-apiserver] Successfully removed container on host [rke.onap.cloud] 
INFO[0003] [remove/kube-controller-manager] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [remove/kube-scheduler] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [controlplane] Host [rke.onap.cloud] is already a worker host, skipping delete kubelet and kubeproxy. 
INFO[0004] [controlplane] Successfully tore down Controller Plane.. 
INFO[0004] [etcd] Tearing down etcd plane..             
INFO[0004] [remove/etcd] Successfully removed container on host [rke.onap.cloud] 
INFO[0004] [etcd] Successfully tore down etcd plane..   
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0004] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0004] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0005] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0005] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0005] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0006] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0006] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0006] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0007] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0008] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0008] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0008] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0009] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0009] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0009] [hosts] Cleaning up host [rke.onap.cloud]    
INFO[0009] [hosts] Running cleaner container on host [rke.onap.cloud] 
INFO[0010] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] 
INFO[0010] [hosts] Removing cleaner container on host [rke.onap.cloud] 
INFO[0010] [hosts] Removing dead container logs on host [rke.onap.cloud] 
INFO[0011] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] 
INFO[0011] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] 
INFO[0011] [hosts] Successfully cleaned up host [rke.onap.cloud] 
INFO[0011] Removing local admin Kubeconfig: ./kube_config_cluster.yml 
INFO[0011] Local admin Kubeconfig removed successfully  
INFO[0011] Cluster removed successfully  

ubuntu@a-rke:~$ rke config --name cluster.ym
ubuntu@a-rke:~$ sudo rke up
INFO[0000] Building Kubernetes cluster                  
INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] 
INFO[0000] [network] Deploying port listener containers 
INFO[0001] [network] Successfully started [rke-etcd-port-listener] container on host [rke.onap.cloud] 
INFO[0001] [network] Successfully started [rke-cp-port-listener] container on host [rke.onap.cloud] 
INFO[0002] [network] Successfully started [rke-worker-port-listener] container on host [rke.onap.cloud] 
INFO[0002] [network] Port listener containers deployed successfully 
INFO[0002] [network] Running control plane -> etcd port checks 
INFO[0003] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0003] [network] Running control plane -> worker port checks 
INFO[0004] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0004] [network] Running workers -> control plane port checks 
INFO[0005] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] 
INFO[0005] [network] Checking KubeAPI port Control Plane hosts 
INFO[0005] [network] Removing port listener containers  
INFO[0005] [remove/rke-etcd-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-cp-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [remove/rke-worker-port-listener] Successfully removed container on host [rke.onap.cloud] 
INFO[0006] [network] Port listener containers removed successfully 
INFO[0006] [certificates] Attempting to recover certificates from backup on [etcd,controlPlane] hosts 
INFO[0007] [certificates] No Certificate backup found on [etcd,controlPlane] hosts 
INFO[0007] [certificates] Generating CA kubernetes certificates 
INFO[0007] [certificates] Generating Kubernetes API server certficates 
INFO[0008] [certificates] Generating Kube Controller certificates 
INFO[0008] [certificates] Generating Kube Scheduler certificates 
INFO[0008] [certificates] Generating Kube Proxy certificates 
INFO[0009] [certificates] Generating Node certificate   
INFO[0009] [certificates] Generating admin certificates and kubeconfig 
INFO[0009] [certificates] Generating etcd-rke.onap.cloud certificate and key 
INFO[0009] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0009] [certificates] Generating Kubernetes API server proxy client certificates 
INFO[0010] [certificates] Temporarily saving certs to [etcd,controlPlane] hosts 
INFO[0016] [certificates] Saved certs to [etcd,controlPlane] hosts 
INFO[0016] [reconcile] Reconciling cluster state        
INFO[0016] [reconcile] This is newly generated cluster  
INFO[0016] [certificates] Deploying kubernetes certificates to Cluster nodes 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes 
INFO[0022] Pre-pulling kubernetes images                
INFO[0022] Kubernetes images pulled successfully        
INFO[0022] [etcd] Building up etcd plane..              
INFO[0023] [etcd] Successfully started [etcd] container on host [rke.onap.cloud] 
INFO[0023] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [rke.onap.cloud] 
INFO[0028] [certificates] Successfully started [rke-bundle-cert] container on host [rke.onap.cloud] 
INFO[0029] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [rke.onap.cloud] 
INFO[0029] [etcd] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0030] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0030] [etcd] Successfully started etcd plane..     
INFO[0030] [controlplane] Building up Controller Plane.. 
INFO[0031] [controlplane] Successfully started [kube-apiserver] container on host [rke.onap.cloud] 
INFO[0031] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [rke.onap.cloud] 
INFO[0045] [healthcheck] service [kube-apiserver] on host [rke.onap.cloud] is healthy 
INFO[0046] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0046] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0047] [controlplane] Successfully started [kube-controller-manager] container on host [rke.onap.cloud] 
INFO[0047] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [rke.onap.cloud] 
INFO[0052] [healthcheck] service [kube-controller-manager] on host [rke.onap.cloud] is healthy 
INFO[0053] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0053] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0054] [controlplane] Successfully started [kube-scheduler] container on host [rke.onap.cloud] 
INFO[0054] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [rke.onap.cloud] 
INFO[0059] [healthcheck] service [kube-scheduler] on host [rke.onap.cloud] is healthy 
INFO[0060] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0060] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0060] [controlplane] Successfully started Controller Plane.. 
INFO[0060] [authz] Creating rke-job-deployer ServiceAccount 
INFO[0060] [authz] rke-job-deployer ServiceAccount created successfully 
INFO[0060] [authz] Creating system:node ClusterRoleBinding 
INFO[0060] [authz] system:node ClusterRoleBinding created successfully 
INFO[0060] [certificates] Save kubernetes certificates as secrets 
INFO[0060] [certificates] Successfully saved certificates as kubernetes secret [k8s-certs] 
INFO[0060] [state] Saving cluster state to Kubernetes   
INFO[0061] [state] Successfully Saved cluster state to Kubernetes ConfigMap: cluster-state 
INFO[0061] [state] Saving cluster state to cluster nodes 
INFO[0061] [state] Successfully started [cluster-state-deployer] container on host [rke.onap.cloud] 
INFO[0062] [remove/cluster-state-deployer] Successfully removed container on host [rke.onap.cloud] 
INFO[0062] [worker] Building up Worker Plane..          
INFO[0062] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] 
INFO[0063] [worker] Successfully started [kubelet] container on host [rke.onap.cloud] 
INFO[0063] [healthcheck] Start Healthcheck on service [kubelet] on host [rke.onap.cloud] 
INFO[0068] [healthcheck] service [kubelet] on host [rke.onap.cloud] is healthy 
INFO[0069] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0070] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0070] [worker] Successfully started [kube-proxy] container on host [rke.onap.cloud] 
INFO[0070] [healthcheck] Start Healthcheck on service [kube-proxy] on host [rke.onap.cloud] 
INFO[0076] [healthcheck] service [kube-proxy] on host [rke.onap.cloud] is healthy 
INFO[0076] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] 
INFO[0077] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] 
INFO[0077] [worker] Successfully started Worker Plane.. 
INFO[0077] [sync] Syncing nodes Labels and Taints       
INFO[0077] [sync] Successfully synced nodes Labels and Taints 
INFO[0077] [network] Setting up network plugin: canal   
INFO[0077] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0077] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-network-plugin 
INFO[0077] [addons] Executing deploy job..              
INFO[0082] [addons] Setting up KubeDNS                  
INFO[0082] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0082] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-kubedns-addon 
INFO[0082] [addons] Executing deploy job..              
INFO[0087] [addons] KubeDNS deployed successfully..     
INFO[0087] [addons] Setting up Metrics Server           
INFO[0087] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0087] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-metrics-addon 
INFO[0087] [addons] Executing deploy job..              
INFO[0092] [addons] KubeDNS deployed successfully..     
INFO[0092] [ingress] Setting up nginx ingress controller 
INFO[0092] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0092] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-ingress-controller 
INFO[0092] [addons] Executing deploy job..              
INFO[0097] [ingress] ingress controller nginx is successfully deployed 
INFO[0097] [addons] Setting up user addons              
INFO[0097] [addons] Checking for included user addons   
WARN[0097] [addons] Unable to determine if  is a file path or url, skipping 
INFO[0097] [addons] Deploying rke-user-includes-addons  
INFO[0097] [addons] Saving addon ConfigMap to Kubernetes 
INFO[0097] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-user-includes-addons 
INFO[0097] [addons] Executing deploy job..              
WARN[0128] Failed to deploy addon execute job [rke-user-includes-addons]: Failed to get job complete status: <nil> 
INFO[0128] Finished building Kubernetes cluster successfully 

ubuntu@a-rke:~$ sudo docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS               NAMES
ec26c4bd24b5        846921f0fe0e                         "/server"                10 minutes ago      Up 10 minutes                           k8s_default-http-backend_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
f8d5db205e14        8a7739f672b4                         "/sidecar --v=2 --lo…"   10 minutes ago      Up 10 minutes                           k8s_sidecar_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
490461545ae4        rancher/metrics-server-amd64         "/metrics-server --s…"   10 minutes ago      Up 10 minutes                           k8s_metrics-server_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
aaf03b62bd41        6816817d9dce                         "/dnsmasq-nanny -v=2…"   10 minutes ago      Up 10 minutes                           k8s_dnsmasq_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
58ec007db72f        55ffe31ac578                         "/kube-dns --domain=…"   10 minutes ago      Up 10 minutes                           k8s_kubedns_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
0a95c06f6aa6        e183460c484d                         "/cluster-proportion…"   10 minutes ago      Up 10 minutes                           k8s_autoscaler_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
968a7c99b210        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0
69969b331e49        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0
baa5f03c16ff        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0
82b2a9f640cb        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0
953a4d4be0c1        df4469c42185                         "/usr/bin/dumb-init …"   10 minutes ago      Up 10 minutes                           k8s_nginx-ingress-controller_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
cce552840749        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0
baa65f9c6f97        f0fad859c909                         "/opt/bin/flanneld -…"   10 minutes ago      Up 10 minutes                           k8s_kube-flannel_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1736ce68f41a        9f355e076ea7                         "/install-cni.sh"        10 minutes ago      Up 10 minutes                           k8s_install-cni_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
615d3f702ee7        7eca10056c8e                         "start_runit"            10 minutes ago      Up 10 minutes                           k8s_calico-node_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
1c4a702f0f18        rancher/pause-amd64:3.1              "/pause"                 10 minutes ago      Up 10 minutes                           k8s_POD_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0
0da1cada08e1        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   10 minutes ago      Up 10 minutes                           kube-proxy
57f44998f34a        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kubelet
50f424c4daec        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kube-scheduler
502d327912d9        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kube-controller-manager
9fc706bbf3a5        rancher/hyperkube:v1.11.6-rancher1   "/opt/rke-tools/entr…"   11 minutes ago      Up 11 minutes                           kube-apiserver
2e7630c2047c        rancher/coreos-etcd:v3.2.18          "/usr/local/bin/etcd…"   11 minutes ago      Up 11 minutes                           etcd
fef566337eb6        rancher/rke-tools:v0.1.15            "/opt/rke-tools/rke-…"   26 minutes ago      Up 26 minutes                           etcd-rolling-snapshots


amdocs@obriensystemsu0:~$ kubectl get pods --all-namespaces
NAMESPACE       NAME                                      READY     STATUS      RESTARTS   AGE
ingress-nginx   default-http-backend-797c5bc547-m8hbx     1/1       Running     0          1h
ingress-nginx   nginx-ingress-controller-2v7w7            1/1       Running     0          1h
kube-system     canal-thmfg                               3/3       Running     0          1h
kube-system     kube-dns-7588d5b5f5-j66s8                 3/3       Running     0          1h
kube-system     kube-dns-autoscaler-5db9bbb766-rg5n8      1/1       Running     0          1h
kube-system     metrics-server-97bc649d5-jd2rr            1/1       Running     0          1h
kube-system     rke-ingress-controller-deploy-job-znp9n   0/1       Completed   0          1h
kube-system     rke-kubedns-addon-deploy-job-dzxsj        0/1       Completed   0          1h
kube-system     rke-metrics-addon-deploy-job-gpm4j        0/1       Completed   0          1h
kube-system     rke-network-plugin-deploy-job-kqdds       0/1       Completed   0          1h
kube-system     tiller-deploy-69458576b-khgr5             1/1       Running     0          1h

DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user

amdocs@obriensystemsu0:~$ sudo rke up
Segmentation fault (core dumped)

# issue was I was using ubuntu as the yml user not amdocs in this case for a particular VM

DI 20190227-1: Verify no 110 pod limit per VM

https://forums.rancher.com/t/solved-setting-max-pods/11866

kubelet:
    image: ""
    extra_args:
      max-pods: 900

DI 20190228-1: deploy casablanca MR to RKE under K8S 1.11.6, Docker 18.06, Helm 2.12.3

sudo git clone https://gerrit.onap.org/r/logging-analytics
sudo wget https://git.onap.org/oom/plain/kubernetes/onap/resources/environments/dev.yaml
sudo cp dev.yaml dev0.yaml
sudo vi dev0.yaml
sudo cp dev0.yaml dev1.yaml
sudo cp logging-analytics/deploy/cd.sh .
sudo ./cd.sh -b casablanca -e onap -p false nexus3.onap.org:10001 -f true -s 300 -c true -d false -w false -r false


no good for helm 2.12.3 deployment - just using 2.9.1 for now - 
Error: Chart incompatible with Tiller v2.12.3


in the casablanca branch only - flip
https://git.onap.org/oom/tree/kubernetes/onap/Chart.yaml?h=casablanca#n24
tillerVersion: "~2.9.1"

DI 20190305-1: Azure 256G VM full ONAP Testing

obrienbiometrics:oom michaelobrien$ ssh ubuntu@onap-dmz.onap.cloud

./oom_deployment.sh -b master -s rke.onap.cloud -e onap -r a_rke0_master -t _arm_deploy_onap_cd.json -p _arm_deploy_onap_rke_z_parameters.json

DI 20190425: HA RKE install Testing

Manual first for RC0, later retrofit the script in https://git.onap.org/oom/tree/kubernetes/contrib/tools/rke/rke_setup.sh and move/adjust the heat template in https://git.onap.org/logging-analytics/tree/deploy/heat/logging_openstack_13_16g.yaml

Installing on 6 nodes on AWS (windriver is having an issue right now).

We are good on RKE 0.2.1, Ubuntu 18.04 / Kubernetes/kubectl 1.13.5 / Helm 2.13.1 / Docker 18.09.5

https://github.com/rancher/rke/releases RKE 0.2.2 has experimental k8s 1.14 support - running with 0.2.1 for now

Just need to test onap deployments

I'll do the NFS/EFS later before deployment

# on all VMs (control, etcd, worker)
# move the key to all vms
scp ~/wse_onap/onap_rsa ubuntu@rke0.onap.info:~/
sudo curl https://releases.rancher.com/install-docker/18.09.sh | sh
sudo usermod -aG docker ubuntu

# nfs server

# on control/etcd nodes only
# from script
sudo wget https://github.com/rancher/rke/releases/download/v0.2.1/rke_linux-amd64
mv rke_linux-amd64 rke
sudo chmod +x rke
sudo mv ./rke /usr/local/bin/rke
# one time setup of the yaml or use the generated one
ubuntu@ip-172-31-38-182:~$ sudo rke config
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/onap_rsa
[+] Number of Hosts [1]: 6
[+] SSH Address of host (1) [none]: 3.14.102.175
[+] SSH Port of host (1) [22]: 
[+] SSH Private Key Path of host (3.14.102.175) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (3.14.102.175) [ubuntu]: ubuntu
[+] Is host (3.14.102.175) a Control Plane host (y/n)? [y]: y
[+] Is host (3.14.102.175) a Worker host (y/n)? [n]: n
[+] Is host (3.14.102.175) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (3.14.102.175) [none]: 
[+] Internal IP of host (3.14.102.175) [none]: 
[+] Docker socket path on host (3.14.102.175) [/var/run/docker.sock]: 
[+] SSH Address of host (2) [none]: 18.220.62.6
[+] SSH Port of host (2) [22]: 
[+] SSH Private Key Path of host (18.220.62.6) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.220.62.6) [ubuntu]: 
[+] Is host (18.220.62.6) a Control Plane host (y/n)? [y]: y
[+] Is host (18.220.62.6) a Worker host (y/n)? [n]: n
[+] Is host (18.220.62.6) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (18.220.62.6) [none]: 
[+] Internal IP of host (18.220.62.6) [none]: 
[+] Docker socket path on host (18.220.62.6) [/var/run/docker.sock]: 
[+] SSH Address of host (3) [none]: 18.217.96.12
[+] SSH Port of host (3) [22]: 
[+] SSH Private Key Path of host (18.217.96.12) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.217.96.12) [ubuntu]: 
[+] Is host (18.217.96.12) a Control Plane host (y/n)? [y]: y
[+] Is host (18.217.96.12) a Worker host (y/n)? [n]: n
[+] Is host (18.217.96.12) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (18.217.96.12) [none]: 
[+] Internal IP of host (18.217.96.12) [none]: 
[+] Docker socket path on host (18.217.96.12) [/var/run/docker.sock]: 
[+] SSH Address of host (4) [none]: 18.188.214.137
[+] SSH Port of host (4) [22]: 
[+] SSH Private Key Path of host (18.188.214.137) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.188.214.137) [ubuntu]: 
[+] Is host (18.188.214.137) a Control Plane host (y/n)? [y]: n
[+] Is host (18.188.214.137) a Worker host (y/n)? [n]: y
[+] Is host (18.188.214.137) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (18.188.214.137) [none]: 
[+] Internal IP of host (18.188.214.137) [none]: 
[+] Docker socket path on host (18.188.214.137) [/var/run/docker.sock]: 
[+] SSH Address of host (5) [none]: 18.220.70.253
[+] SSH Port of host (5) [22]: 
[+] SSH Private Key Path of host (18.220.70.253) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.220.70.253) [ubuntu]: 
[+] Is host (18.220.70.253) a Control Plane host (y/n)? [y]: n
[+] Is host (18.220.70.253) a Worker host (y/n)? [n]: y
[+] Is host (18.220.70.253) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (18.220.70.253) [none]: 
[+] Internal IP of host (18.220.70.253) [none]: 
[+] Docker socket path on host (18.220.70.253) [/var/run/docker.sock]: 
[+] SSH Address of host (6) [none]: 3.17.76.33
[+] SSH Port of host (6) [22]: 
[+] SSH Private Key Path of host (3.17.76.33) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (3.17.76.33) [ubuntu]: 
[+] Is host (3.17.76.33) a Control Plane host (y/n)? [y]: n
[+] Is host (3.17.76.33) a Worker host (y/n)? [n]: y
[+] Is host (3.17.76.33) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (3.17.76.33) [none]: 
[+] Internal IP of host (3.17.76.33) [none]: 
[+] Docker socket path on host (3.17.76.33) [/var/run/docker.sock]: 
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]: 
[+] Authentication Strategy [x509]: 
[+] Authorization Mode (rbac, none) [rbac]: 
[+] Kubernetes Docker image [rancher/hyperkube:v1.13.5-rancher1]: 
[+] Cluster domain [cluster.local]: 
[+] Service Cluster IP Range [10.43.0.0/16]: 
[+] Enable PodSecurityPolicy [n]: 
[+] Cluster Network CIDR [10.42.0.0/16]: 
[+] Cluster DNS Service IP [10.43.0.10]: 
[+] Add addon manifest URLs or YAML files [no]:

# new
[+] Cluster domain [cluster.local]: 

ubuntu@ip-172-31-38-182:~$ sudo rke up
INFO[0000] Initiating Kubernetes cluster                
INFO[0000] [certificates] Generating CA kubernetes certificates 
INFO[0000] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates 
INFO[0000] [certificates] Generating Kubernetes API server certificates 
INFO[0000] [certificates] Generating Service account token key 
INFO[0000] [certificates] Generating etcd-3.14.102.175 certificate and key 
INFO[0000] [certificates] Generating etcd-18.220.62.6 certificate and key 
INFO[0001] [certificates] Generating etcd-18.217.96.12 certificate and key 
INFO[0001] [certificates] Generating Kube Controller certificates 
INFO[0001] [certificates] Generating Kube Scheduler certificates 
INFO[0001] [certificates] Generating Kube Proxy certificates 
INFO[0001] [certificates] Generating Node certificate   
INFO[0001] [certificates] Generating admin certificates and kubeconfig 
INFO[0001] [certificates] Generating Kubernetes API server proxy client certificates 
INFO[0002] Successfully Deployed state file at [./cluster.rkestate] 
INFO[0002] Building Kubernetes cluster                  
INFO[0002] [dialer] Setup tunnel for host [3.14.102.175] 
INFO[0002] [dialer] Setup tunnel for host [18.188.214.137] 
INFO[0002] [dialer] Setup tunnel for host [18.220.70.253] 
INFO[0002] [dialer] Setup tunnel for host [18.220.62.6] 
INFO[0002] [dialer] Setup tunnel for host [18.217.96.12] 
INFO[0002] [dialer] Setup tunnel for host [3.17.76.33]  
INFO[0002] [network] Deploying port listener containers 
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.220.62.6] 
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [3.14.102.175] 
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.217.96.12] 
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.220.62.6] 
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.217.96.12] 
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [3.14.102.175] 
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [18.220.62.6] 
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [18.217.96.12] 
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [3.14.102.175] 
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [18.217.96.12] 
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [18.220.62.6] 
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [3.14.102.175] 
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.188.214.137] 
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.220.70.253] 
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [3.17.76.33] 
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [3.17.76.33] 
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.220.70.253] 
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.188.214.137] 
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [3.17.76.33] 
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [18.220.70.253] 
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [18.188.214.137] 
INFO[0013] [network] Port listener containers deployed successfully 
INFO[0013] [network] Running etcd <-> etcd port checks  
INFO[0013] [network] Successfully started [rke-port-checker] container on host [18.220.62.6] 
INFO[0013] [network] Successfully started [rke-port-checker] container on host [3.14.102.175] 
INFO[0013] [network] Successfully started [rke-port-checker] container on host [18.217.96.12] 
INFO[0014] [network] Running control plane -> etcd port checks 
INFO[0014] [network] Successfully started [rke-port-checker] container on host [18.220.62.6] 
INFO[0014] [network] Successfully started [rke-port-checker] container on host [3.14.102.175] 
INFO[0014] [network] Successfully started [rke-port-checker] container on host [18.217.96.12] 
INFO[0014] [network] Running control plane -> worker port checks 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.220.62.6] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.217.96.12] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [3.14.102.175] 
INFO[0015] [network] Running workers -> control plane port checks 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [3.17.76.33] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.220.70.253] 
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.188.214.137] 
INFO[0016] [network] Checking KubeAPI port Control Plane hosts 
INFO[0016] [network] Removing port listener containers  
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [18.220.62.6] 
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [3.14.102.175] 
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [18.217.96.12] 
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [18.217.96.12] 
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [3.14.102.175] 
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [18.220.62.6] 
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [18.220.70.253] 
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [3.17.76.33] 
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [18.188.214.137] 
INFO[0017] [network] Port listener containers removed successfully 
INFO[0017] [certificates] Deploying kubernetes certificates to Cluster nodes 
INFO[0022] [reconcile] Rebuilding and updating local kube config 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] 
INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes 
INFO[0022] [reconcile] Reconciling cluster state        
INFO[0022] [reconcile] This is newly generated cluster  
INFO[0022] Pre-pulling kubernetes images                
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [3.14.102.175] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.188.214.137] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.70.253] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.217.96.12] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [3.17.76.33] 
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.62.6] 
INFO[0038] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.62.6] 
INFO[0038] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.70.253] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [3.17.76.33] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.217.96.12] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.188.214.137] 
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [3.14.102.175] 
INFO[0039] Kubernetes images pulled successfully        
INFO[0039] [etcd] Building up etcd plane..              
INFO[0039] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [3.14.102.175] 
INFO[0041] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [3.14.102.175] 
INFO[0051] [etcd] Successfully started [etcd] container on host [3.14.102.175] 
INFO[0051] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [3.14.102.175] 
INFO[0052] [etcd] Successfully started [etcd-rolling-snapshots] container on host [3.14.102.175] 
INFO[0057] [certificates] Successfully started [rke-bundle-cert] container on host [3.14.102.175] 
INFO[0058] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [3.14.102.175] 
INFO[0058] [etcd] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0058] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0058] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.220.62.6] 
INFO[0063] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.220.62.6] 
INFO[0069] [etcd] Successfully started [etcd] container on host [18.220.62.6] 
INFO[0069] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [18.220.62.6] 
INFO[0069] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.220.62.6] 
INFO[0075] [certificates] Successfully started [rke-bundle-cert] container on host [18.220.62.6] 
INFO[0075] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.220.62.6] 
INFO[0076] [etcd] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0076] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0076] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.217.96.12] 
INFO[0078] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.217.96.12] 
INFO[0078] [etcd] Successfully started [etcd] container on host [18.217.96.12] 
INFO[0078] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [18.217.96.12] 
INFO[0078] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.217.96.12] 
INFO[0084] [certificates] Successfully started [rke-bundle-cert] container on host [18.217.96.12] 
INFO[0084] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.217.96.12] 
INFO[0085] [etcd] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0085] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0085] [etcd] Successfully started etcd plane.. Checking etcd cluster health 
INFO[0086] [controlplane] Building up Controller Plane.. 
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [18.220.62.6] 
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.220.62.6] 
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [3.14.102.175] 
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [3.14.102.175] 
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [18.217.96.12] 
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.217.96.12] 
INFO[0098] [healthcheck] service [kube-apiserver] on host [18.220.62.6] is healthy 
INFO[0099] [healthcheck] service [kube-apiserver] on host [18.217.96.12] is healthy 
INFO[0099] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0099] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0099] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0099] [healthcheck] service [kube-apiserver] on host [3.14.102.175] is healthy 
INFO[0099] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0099] [controlplane] Successfully started [kube-controller-manager] container on host [18.220.62.6] 
INFO[0099] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.220.62.6] 
INFO[0100] [controlplane] Successfully started [kube-controller-manager] container on host [18.217.96.12] 
INFO[0100] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.217.96.12] 
INFO[0100] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0100] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0100] [healthcheck] service [kube-controller-manager] on host [18.220.62.6] is healthy 
INFO[0100] [healthcheck] service [kube-controller-manager] on host [18.217.96.12] is healthy 
INFO[0100] [controlplane] Successfully started [kube-controller-manager] container on host [3.14.102.175] 
INFO[0100] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [3.14.102.175] 
INFO[0101] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0101] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0101] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0101] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0101] [controlplane] Successfully started [kube-scheduler] container on host [18.220.62.6] 
INFO[0101] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.220.62.6] 
INFO[0101] [healthcheck] service [kube-controller-manager] on host [3.14.102.175] is healthy 
INFO[0101] [controlplane] Successfully started [kube-scheduler] container on host [18.217.96.12] 
INFO[0101] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.217.96.12] 
INFO[0102] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0102] [healthcheck] service [kube-scheduler] on host [18.220.62.6] is healthy 
INFO[0102] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0102] [healthcheck] service [kube-scheduler] on host [18.217.96.12] is healthy 
INFO[0103] [controlplane] Successfully started [kube-scheduler] container on host [3.14.102.175] 
INFO[0103] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [3.14.102.175] 
INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0103] [healthcheck] service [kube-scheduler] on host [3.14.102.175] is healthy 
INFO[0104] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0104] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0104] [controlplane] Successfully started Controller Plane.. 
INFO[0104] [authz] Creating rke-job-deployer ServiceAccount 
INFO[0104] [authz] rke-job-deployer ServiceAccount created successfully 
INFO[0104] [authz] Creating system:node ClusterRoleBinding 
INFO[0104] [authz] system:node ClusterRoleBinding created successfully 
INFO[0104] Successfully Deployed state file at [./cluster.rkestate] 
INFO[0104] [state] Saving full cluster state to Kubernetes 
INFO[0104] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: cluster-state 
INFO[0104] [worker] Building up Worker Plane..          
INFO[0104] [sidekick] Sidekick container already created on host [18.220.62.6] 
INFO[0104] [sidekick] Sidekick container already created on host [3.14.102.175] 
INFO[0104] [sidekick] Sidekick container already created on host [18.217.96.12] 
INFO[0105] [worker] Successfully started [kubelet] container on host [3.14.102.175] 
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [3.14.102.175] 
INFO[0105] [worker] Successfully started [kubelet] container on host [18.220.62.6] 
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [18.220.62.6] 
INFO[0105] [worker] Successfully started [kubelet] container on host [18.217.96.12] 
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [18.217.96.12] 
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [18.220.70.253] 
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [3.17.76.33] 
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [18.188.214.137] 
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253] 
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33] 
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137] 
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33] 
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253] 
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137] 
INFO[0106] [worker] Successfully started [kubelet] container on host [3.17.76.33] 
INFO[0106] [healthcheck] Start Healthcheck on service [kubelet] on host [3.17.76.33] 
INFO[0107] [worker] Successfully started [kubelet] container on host [18.220.70.253] 
INFO[0107] [healthcheck] Start Healthcheck on service [kubelet] on host [18.220.70.253] 
INFO[0107] [worker] Successfully started [kubelet] container on host [18.188.214.137] 
INFO[0107] [healthcheck] Start Healthcheck on service [kubelet] on host [18.188.214.137] 
INFO[0111] [healthcheck] service [kubelet] on host [18.220.62.6] is healthy 
INFO[0111] [healthcheck] service [kubelet] on host [18.217.96.12] is healthy 
INFO[0111] [healthcheck] service [kubelet] on host [3.14.102.175] is healthy 
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0112] [worker] Successfully started [kube-proxy] container on host [18.220.62.6] 
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.220.62.6] 
INFO[0112] [worker] Successfully started [kube-proxy] container on host [18.217.96.12] 
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.217.96.12] 
INFO[0112] [worker] Successfully started [kube-proxy] container on host [3.14.102.175] 
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [3.14.102.175] 
INFO[0113] [healthcheck] service [kube-proxy] on host [18.220.62.6] is healthy 
INFO[0113] [healthcheck] service [kube-proxy] on host [18.217.96.12] is healthy 
INFO[0113] [healthcheck] service [kube-proxy] on host [3.14.102.175] is healthy 
INFO[0113] [healthcheck] service [kubelet] on host [18.220.70.253] is healthy 
INFO[0113] [healthcheck] service [kubelet] on host [3.17.76.33] is healthy 
INFO[0113] [healthcheck] service [kubelet] on host [18.188.214.137] is healthy 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.220.62.6] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.217.96.12] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [3.14.102.175] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253] 
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6] 
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137] 
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12] 
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175] 
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33] 
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253] 
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137] 
INFO[0114] [worker] Successfully started [kube-proxy] container on host [3.17.76.33] 
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [3.17.76.33] 
INFO[0114] [worker] Successfully started [kube-proxy] container on host [18.220.70.253] 
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.220.70.253] 
INFO[0114] [worker] Successfully started [kube-proxy] container on host [18.188.214.137] 
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.188.214.137] 
INFO[0114] [healthcheck] service [kube-proxy] on host [18.220.70.253] is healthy 
INFO[0114] [healthcheck] service [kube-proxy] on host [3.17.76.33] is healthy 
INFO[0115] [healthcheck] service [kube-proxy] on host [18.188.214.137] is healthy 
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253] 
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33] 
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137] 
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253] 
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33] 
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137] 
INFO[0115] [worker] Successfully started Worker Plane.. 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.188.214.137] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [3.17.76.33] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.220.70.253] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.220.62.6] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.217.96.12] 
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [3.14.102.175] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [3.17.76.33] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.188.214.137] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.220.62.6] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.220.70.253] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.217.96.12] 
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [3.14.102.175] 
INFO[0116] [sync] Syncing nodes Labels and Taints       
INFO[0117] [sync] Successfully synced nodes Labels and Taints 
INFO[0117] [network] Setting up network plugin: canal   
INFO[0117] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes 
INFO[0117] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes 
INFO[0117] [addons] Executing deploy job rke-network-plugin 
INFO[0122] [addons] Setting up kube-dns                 
INFO[0122] [addons] Saving ConfigMap for addon rke-kube-dns-addon to Kubernetes 
INFO[0122] [addons] Successfully saved ConfigMap for addon rke-kube-dns-addon to Kubernetes 
INFO[0122] [addons] Executing deploy job rke-kube-dns-addon 
INFO[0127] [addons] kube-dns deployed successfully      
INFO[0127] [dns] DNS provider kube-dns deployed successfully 
INFO[0127] [addons] Setting up Metrics Server           
INFO[0127] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes 
INFO[0127] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes 
INFO[0127] [addons] Executing deploy job rke-metrics-addon 
INFO[0132] [addons] Metrics Server deployed successfully 
INFO[0132] [ingress] Setting up nginx ingress controller 
INFO[0132] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0132] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes 
INFO[0132] [addons] Executing deploy job rke-ingress-controller 
INFO[0137] [ingress] ingress controller nginx deployed successfully 
INFO[0137] [addons] Setting up user addons              
INFO[0137] [addons] no user addons defined              
INFO[0137] Finished building Kubernetes cluster successfully

# finish kubectl install
sudo curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.13.5/bin/linux/amd64/kubectl
sudo chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
sudo mkdir ~/.kube

# finish helm
#https://github.com/helm/helm/releases
# there is no helm 2.12.5 - last is 2.12.3 - trying 2.13.1
wget http://storage.googleapis.com/kubernetes-helm/helm-v2.13.1-linux-amd64.tar.gz
sudo tar -zxvf helm-v2.13.1-linux-amd64.tar.gz
sudo cp kube_config_cluster.yml ~/.kube/config
sudo chmod 777 ~/.kube/config

# test
ubuntu@ip-172-31-38-182:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE       NAME                                    READY   STATUS    RESTARTS   AGE   IP               NODE             NOMINATED NODE   READINESS GATES
ingress-nginx   default-http-backend-78fccfc5d9-f6z25   1/1     Running   0          17m   10.42.5.2        18.220.70.253    <none>           <none>
ingress-nginx   nginx-ingress-controller-2zxs7          1/1     Running   0          17m   18.188.214.137   18.188.214.137   <none>           <none>
ingress-nginx   nginx-ingress-controller-6b7gs          1/1     Running   0          17m   3.17.76.33       3.17.76.33       <none>           <none>
ingress-nginx   nginx-ingress-controller-nv4qg          1/1     Running   0          17m   18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-48579                             2/2     Running   0          17m   18.220.62.6      18.220.62.6      <none>           <none>
kube-system     canal-6skkm                             2/2     Running   0          17m   18.188.214.137   18.188.214.137   <none>           <none>
kube-system     canal-9xmxv                             2/2     Running   0          17m   18.217.96.12     18.217.96.12     <none>           <none>
kube-system     canal-c582x                             2/2     Running   0          17m   18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-whbck                             2/2     Running   0          17m   3.14.102.175     3.14.102.175     <none>           <none>
kube-system     canal-xbbnh                             2/2     Running   0          17m   3.17.76.33       3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-6mcm7               3/3     Running   0          17m   10.42.3.3        3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-cd5dg               3/3     Running   0          17m   10.42.4.2        18.188.214.137   <none>           <none>
kube-system     kube-dns-autoscaler-77bc5fd84-p4zfd     1/1     Running   0          17m   10.42.3.2        3.17.76.33       <none>           <none>
kube-system     metrics-server-58bd5dd8d7-kftjn         1/1     Running   0          17m   10.42.3.4        3.17.76.33       <none>           <none>


# install tiller

ubuntu@ip-172-31-38-182:~$ kubectl -n kube-system create serviceaccount tiller
serviceaccount/tiller created
ubuntu@ip-172-31-38-182:~$ kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller created
ubuntu@ip-172-31-38-182:~$ helm init --service-account tiller
Creating /home/ubuntu/.helm 
Creating /home/ubuntu/.helm/repository 
Creating /home/ubuntu/.helm/repository/cache 
Creating /home/ubuntu/.helm/repository/local 
Creating /home/ubuntu/.helm/plugins 
Creating /home/ubuntu/.helm/starters 
Creating /home/ubuntu/.helm/cache/archive 
Creating /home/ubuntu/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /home/ubuntu/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
ubuntu@ip-172-31-38-182:~$ kubectl -n kube-system  rollout status deploy/tiller-deploy
deployment "tiller-deploy" successfully rolled out
ubuntu@ip-172-31-38-182:~$ sudo helm init --upgrade
$HELM_HOME has been configured at /home/ubuntu/.helm.

Tiller (the Helm server-side component) has been upgraded to the current version.
Happy Helming!


ubuntu@ip-172-31-38-182:~$ sudo helm version
Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
ubuntu@ip-172-31-38-182:~$ sudo helm serve &
[1] 706
ubuntu@ip-172-31-38-182:~$ Regenerating index. This may take a moment.
Now serving you on 127.0.0.1:8879

ubuntu@ip-172-31-38-182:~$ sudo helm list
ubuntu@ip-172-31-38-182:~$ sudo helm repo add local http://127.0.0.1:8879
"local" has been added to your repositories
ubuntu@ip-172-31-38-182:~$  sudo helm repo list
NAME  	URL                                             
stable	https://kubernetes-charts.storage.googleapis.com
local 	http://127.0.0.1:8879                           

ubuntu@ip-172-31-38-182:~$ kubectl get nodes -o wide
NAME             STATUS   ROLES               AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
18.188.214.137   Ready    worker              22m   v1.13.5   18.188.214.137   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
18.217.96.12     Ready    controlplane,etcd   22m   v1.13.5   18.217.96.12     <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
18.220.62.6      Ready    controlplane,etcd   22m   v1.13.5   18.220.62.6      <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
18.220.70.253    Ready    worker              22m   v1.13.5   18.220.70.253    <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
3.14.102.175     Ready    controlplane,etcd   22m   v1.13.5   3.14.102.175     <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5
3.17.76.33       Ready    worker              22m   v1.13.5   3.17.76.33       <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.9.5


# install make
sudo apt-get install make -y

# install nfs/efs
ubuntu@ip-172-31-38-182:~$ sudo apt-get install nfs-common -y
ubuntu@ip-172-31-38-182:~$ sudo mkdir /dockerdata-nfs
ubuntu@ip-172-31-38-182:~$ sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-5fd6ab26.efs.us-east-2.amazonaws.com:/ /dockerdata-nfs



# check
sudo nohup ./cd.sh -b master -e onap -p false -n nexus3.onap.org:10001 -f false -s 600 -c false -d false -w false -r false &
ubuntu@ip-172-31-38-182:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE       NAME                                                     READY   STATUS             RESTARTS   AGE     IP               NODE             NOMINATED NODE   READINESS GATES
ingress-nginx   default-http-backend-78fccfc5d9-f6z25                    1/1     Running            0          103m    10.42.5.2        18.220.70.253    <none>           <none>
ingress-nginx   nginx-ingress-controller-2zxs7                           1/1     Running            0          103m    18.188.214.137   18.188.214.137   <none>           <none>
ingress-nginx   nginx-ingress-controller-6b7gs                           1/1     Running            0          103m    3.17.76.33       3.17.76.33       <none>           <none>
ingress-nginx   nginx-ingress-controller-nv4qg                           1/1     Running            0          103m    18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-48579                                              2/2     Running            0          103m    18.220.62.6      18.220.62.6      <none>           <none>
kube-system     canal-6skkm                                              2/2     Running            0          103m    18.188.214.137   18.188.214.137   <none>           <none>
kube-system     canal-9xmxv                                              2/2     Running            0          103m    18.217.96.12     18.217.96.12     <none>           <none>
kube-system     canal-c582x                                              2/2     Running            0          103m    18.220.70.253    18.220.70.253    <none>           <none>
kube-system     canal-whbck                                              2/2     Running            0          103m    3.14.102.175     3.14.102.175     <none>           <none>
kube-system     canal-xbbnh                                              2/2     Running            0          103m    3.17.76.33       3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-6mcm7                                3/3     Running            0          103m    10.42.3.3        3.17.76.33       <none>           <none>
kube-system     kube-dns-58bd5b8dd7-cd5dg                                3/3     Running            0          103m    10.42.4.2        18.188.214.137   <none>           <none>
kube-system     kube-dns-autoscaler-77bc5fd84-p4zfd                      1/1     Running            0          103m    10.42.3.2        3.17.76.33       <none>           <none>
kube-system     metrics-server-58bd5dd8d7-kftjn                          1/1     Running            0          103m    10.42.3.4        3.17.76.33       <none>           <none>
kube-system     tiller-deploy-5f4fc5bcc6-gc4tc                           1/1     Running            0          84m     10.42.5.3        18.220.70.253    <none>           <none>
onap            onap-aai-aai-587cb79c6d-mzpbs                            0/1     Init:0/1           1          13m     10.42.5.17       18.220.70.253    <none>           <none>
onap            onap-aai-aai-babel-8c755bcfc-kmzdm                       2/2     Running            0          29m     10.42.5.8        18.220.70.253    <none>           <none>
onap            onap-aai-aai-champ-78b9d7f68b-98tm9                      0/2     Init:0/1           2          29m     10.42.3.6        3.17.76.33       <none>           <none>
onap            onap-aai-aai-data-router-64fcfbc5bb-wkkvz                1/2     CrashLoopBackOff   9          29m     10.42.5.7        18.220.70.253    <none>           <none>
onap            onap-aai-aai-elasticsearch-6dcf5d9966-j7z67              1/1     Running            0          29m     10.42.4.4        18.188.214.137   <none>           <none>
onap            onap-aai-aai-gizmo-5bddb87589-zn8pl                      2/2     Running            0          29m     10.42.4.3        18.188.214.137   <none>           <none>
onap            onap-aai-aai-graphadmin-774f9d698f-f8lwv                 0/2     Init:0/1           2          29m     10.42.5.4        18.220.70.253    <none>           <none>
onap            onap-aai-aai-graphadmin-create-db-schema-94q4l           0/1     Init:Error         0          18m     10.42.4.16       18.188.214.137   <none>           <none>
onap            onap-aai-aai-graphadmin-create-db-schema-s42pq           0/1     Init:0/1           0          7m54s   10.42.5.20       18.220.70.253    <none>           <none>
onap            onap-aai-aai-graphadmin-create-db-schema-tsvcw           0/1     Init:Error         0          29m     10.42.4.5        18.188.214.137   <none>           <none>
onap            onap-aai-aai-modelloader-845fc684bd-r7mdw                2/2     Running            0          29m     10.42.4.6        18.188.214.137   <none>           <none>
onap            onap-aai-aai-resources-67f8dfcbdb-kz6cp                  0/2     Init:0/1           2          29m     10.42.5.11       18.220.70.253    <none>           <none>
onap            onap-aai-aai-schema-service-6c56b45b7c-7zlfz             2/2     Running            0          29m     10.42.3.7        3.17.76.33       <none>           <none>
onap            onap-aai-aai-search-data-5d8d7759b8-flxwj                2/2     Running            0          29m     10.42.3.9        3.17.76.33       <none>           <none>
onap            onap-aai-aai-sparky-be-8444df749c-mzc2n                  0/2     Init:0/1           0          29m     10.42.4.11       18.188.214.137   <none>           <none>
onap            onap-aai-aai-spike-54ff77787f-d678x                      2/2     Running            0          29m     10.42.5.6        18.220.70.253    <none>           <none>
onap            onap-aai-aai-traversal-6ff868f477-lzv2f                  0/2     Init:0/1           2          29m     10.42.3.8        3.17.76.33       <none>           <none>
onap            onap-aai-aai-traversal-update-query-data-9g2b8           0/1     Init:0/1           2          29m     10.42.5.12       18.220.70.253    <none>           <none>
onap            onap-dmaap-dbc-pg-0                                      1/1     Running            0          29m     10.42.5.9        18.220.70.253    <none>           <none>
onap            onap-dmaap-dbc-pg-1                                      1/1     Running            0          26m     10.42.3.14       3.17.76.33       <none>           <none>
onap            onap-dmaap-dbc-pgpool-8666b57857-97zjc                   1/1     Running            0          29m     10.42.5.5        18.220.70.253    <none>           <none>
onap            onap-dmaap-dbc-pgpool-8666b57857-vr8gk                   1/1     Running            0          29m     10.42.4.8        18.188.214.137   <none>           <none>
onap            onap-dmaap-dmaap-bc-745995bf74-m6hhq                     0/1     Init:0/2           2          29m     10.42.4.12       18.188.214.137   <none>           <none>
onap            onap-dmaap-dmaap-bc-post-install-6ff4j                   1/1     Running            0          29m     10.42.4.9        18.188.214.137   <none>           <none>
onap            onap-dmaap-dmaap-dr-db-0                                 1/1     Running            0          29m     10.42.4.10       18.188.214.137   <none>           <none>
onap            onap-dmaap-dmaap-dr-db-1                                 1/1     Running            1          24m     10.42.5.15       18.220.70.253    <none>           <none>
onap            onap-dmaap-dmaap-dr-node-0                               2/2     Running            0          29m     10.42.3.11       3.17.76.33       <none>           <none>
onap            onap-dmaap-dmaap-dr-prov-fbf6c94f5-v9bmq                 2/2     Running            0          29m     10.42.5.10       18.220.70.253    <none>           <none>
onap            onap-dmaap-message-router-0                              1/1     Running            0          29m     10.42.4.14       18.188.214.137   <none>           <none>
onap            onap-dmaap-message-router-kafka-0                        1/1     Running            1          29m     10.42.5.13       18.220.70.253    <none>           <none>
onap            onap-dmaap-message-router-kafka-1                        1/1     Running            1          29m     10.42.3.13       3.17.76.33       <none>           <none>
onap            onap-dmaap-message-router-kafka-2                        1/1     Running            0          29m     10.42.4.15       18.188.214.137   <none>           <none>
onap            onap-dmaap-message-router-mirrormaker-8587c4c9cf-lfnd8   0/1     CrashLoopBackOff   9          29m     10.42.4.7        18.188.214.137   <none>           <none>
onap            onap-dmaap-message-router-zookeeper-0                    1/1     Running            0          29m     10.42.5.14       18.220.70.253    <none>           <none>
onap            onap-dmaap-message-router-zookeeper-1                    1/1     Running            0          29m     10.42.4.13       18.188.214.137   <none>           <none>
onap            onap-dmaap-message-router-zookeeper-2                    1/1     Running            0          29m     10.42.3.12       3.17.76.33       <none>           <none>
onap            onap-nfs-provisioner-nfs-provisioner-57c999dc57-mdcw5    1/1     Running            0          24m     10.42.3.15       3.17.76.33       <none>           <none>
onap            onap-robot-robot-677bdbb454-zj9jk                        1/1     Running            0          24m     10.42.5.16       18.220.70.253    <none>           <none>
onap            onap-so-so-8569947cbd-jn5x4                              0/1     Init:0/1           1          13m     10.42.4.19       18.188.214.137   <none>           <none>
onap            onap-so-so-bpmn-infra-78c8fd665d-b47qn                   0/1     Init:0/1           1          13m     10.42.3.16       3.17.76.33       <none>           <none>
onap            onap-so-so-catalog-db-adapter-565f9767ff-lvbgx           0/1     Init:0/1           1          13m     10.42.3.17       3.17.76.33       <none>           <none>
onap            onap-so-so-mariadb-config-job-d9sdb                      0/1     Init:0/2           0          3m37s   10.42.3.20       3.17.76.33       <none>           <none>
onap            onap-so-so-mariadb-config-job-rkqdl                      0/1     Init:Error         0          13m     10.42.5.19       18.220.70.253    <none>           <none>
onap            onap-so-so-monitoring-69b9fdd94c-dks4v                   1/1     Running            0          13m     10.42.4.18       18.188.214.137   <none>           <none>
onap            onap-so-so-openstack-adapter-5f9cf896d7-mgbdd            0/1     Init:0/1           1          13m     10.42.4.17       18.188.214.137   <none>           <none>
onap            onap-so-so-request-db-adapter-5c9bfd7b57-2krnp           0/1     Init:0/1           1          13m     10.42.3.18       3.17.76.33       <none>           <none>
onap            onap-so-so-sdc-controller-6fb5cf5775-bsxhm               0/1     Init:0/1           1          13m     10.42.4.20       18.188.214.137   <none>           <none>
onap            onap-so-so-sdnc-adapter-8555689c75-r6vkb                 1/1     Running            0          13m     10.42.5.18       18.220.70.253    <none>           <none>
onap            onap-so-so-vfc-adapter-68fccc8bb8-c56t2                  0/1     Init:0/1           1          13m     10.42.4.21       18.188.214.137   <none>           <none>
onap            onap-so-so-vnfm-adapter-65c4c5944b-72nlf                 1/1     Running            0          13m     10.42.3.19       3.17.76.33       <none>           <none>


# on worker nodes only
# nfs client

DI 20190507: ARM support using RKE 0.2.1 ARM friendly install

LOG-331 - Getting issue details... STATUS

Instance type(s) a1.4xlarge $0.408

AMI ID ami-0b9bd0b532ebcf4c9

Notes

Pre-RKE installation details in Cloud Native Deployment

Browser not supported