OOM RKE Kubernetes Deployment
- Michael O'Brien
Owned by Michael O'Brien
Tracking OOM-1598 - Getting issue details... STATUS
NON-HA version in https://git.onap.org/oom/tree/kubernetes/contrib/tools/rke/rke_setup.sh
https://gerrit.onap.org/r/#/c/79067/
Move to https://onap.readthedocs.io/en/beijing/submodules/oom.git/docs/oom_cloud_setup_guide.html or similar when this documentation is released
Automated Installation Video of RKE install
20190227 - VMware
20190330 - AWS
Quickstart
Get your public and private keys on the Ubuntu 16.04 VM.
Adjust authorized_keys with your public key if not already - aws has it, openstack may not
get rke script from jira, gerrit or by cloning OOM when the review is done.
# on your laptop/where your cert is # chmod 777 your cert before you scp it over obrienbiometrics:full michaelobrien$ scp ~/wse_onap/onap_rsa ubuntu@rke0.onap.info:~/ # on the host sudo cp onap_rsa ~/.ssh sudo chmod 400 ~/.ssh/onap_rsa sudo chown ubuntu:ubuntu ~/.ssh/onap_rsa # just verify sudo vi ~/.ssh/authorized_keys git clone --recurse-submodules https://gerrit.onap.org/r/oom sudo cp oom/kubernetes/contrib/tools/rke/rke_setup.sh . sudo nohup ./rke_setup.sh -b master -s 104.209.161.210 -e onap -k onap_rsa -l ubuntu & ubuntu@a-rke0-master:~$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE ingress-nginx default-http-backend-797c5bc547-55fpn 1/1 Running 0 4m ingress-nginx nginx-ingress-controller-znhgz 1/1 Running 0 4m kube-system canal-dqt2m 3/3 Running 0 5m kube-system kube-dns-7588d5b5f5-pzdfh 3/3 Running 0 5m kube-system kube-dns-autoscaler-5db9bbb766-b7vvg 1/1 Running 0 5m kube-system metrics-server-97bc649d5-fmqjd 1/1 Running 0 4m kube-system rke-ingress-controller-deploy-job-dxmbd 0/1 Completed 0 4m kube-system rke-kubedns-addon-deploy-job-wqccp 0/1 Completed 0 5m kube-system rke-metrics-addon-deploy-job-ssrgp 0/1 Completed 0 4m kube-system rke-network-plugin-deploy-job-jkffq 0/1 Completed 0 5m kube-system tiller-deploy-759cb9df9-rlt7v 1/1 Running 0 2m ubuntu@a-rke0-master:~$ helm list
Versions
Currently Docker 18.06, RKE 0.1.16, Kubernetes 1.11.6, Kubectl 1.11.6, Helm 2.12.3
TODO: verify later versions of helm and a way to get RKE to install Kubernetes 1.13
Prerequisites
Ubuntu 16.04 VM
Determine RKE and Docker versions
Don't just use the latest docker version - check the RKE release page to get the version pair - 0.1.15/17.03 and 0.1.16/18.06 - see https://github.com/docker/docker-ce/releases - currently https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce
ubuntu@a-rke:~$ sudo curl https://releases.rancher.com/install-docker/18.06.sh | sh ubuntu@a-rke:~$ sudo usermod -aG docker ubuntu ubuntu@a-rke:~$ sudo docker version Client: Version: 18.06.3-ce API version: 1.38 Go version: go1.10.3 Git commit: d7080c1 Built: Wed Feb 20 02:27:18 2019 # install RKE sudo wget https://github.com/rancher/rke/releases/download/v0.1.16/rke_linux-amd64 mv rke_linux-amd64 rke sudo mv ./rke /usr/local/bin/rke ubuntu@a-rke:~$ rke --version rke version v0.1.16
Private SSH key
scp your public key to the box - ideally to ~/.ssh and chmod 400 it - make sure you add your key to authorized_keys
Elastic Reserved IP
get a VIP or EIP and assign this to your VM
generate cluster.yml - optional
cluster.yml will generated by the script rke_setup.sh
azure config - no need to hand build the yml
Watch the path of your 2 keys
Also don't add an "addon" until you have one of the config job will fail
{noformat}
ubuntu@a-rke:~$ rke config --name cluster.yml
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/onap_rsa
[+] Number of Hosts [1]:
[+] SSH Address of host (1) [none]: rke.onap.cloud
[+] SSH Port of host (1) [22]:
[+] SSH Private Key Path of host (rke.onap.cloud) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (rke.onap.cloud) [ubuntu]:
[+] Is host (rke.onap.cloud) a Control Plane host (y/n)? [y]: y
[+] Is host (rke.onap.cloud) a Worker host (y/n)? [n]: y
[+] Is host (rke.onap.cloud) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (rke.onap.cloud) [none]:
[+] Internal IP of host (rke.onap.cloud) [none]:
[+] Docker socket path on host (rke.onap.cloud) [/var/run/docker.sock]:
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]:
[+] Authentication Strategy [x509]:
[+] Authorization Mode (rbac, none) [rbac]:
[+] Kubernetes Docker image [rancher/hyperkube:v1.11.6-rancher1]:
[+] Cluster domain [cluster.local]:
[+] Service Cluster IP Range [10.43.0.0/16]:
[+] Enable PodSecurityPolicy [n]:
[+] Cluster Network CIDR [10.42.0.0/16]:
[+] Cluster DNS Service IP [10.43.0.10]:
[+] Add addon manifest URLs or YAML files [no]: no
ubuntu@a-rke:~$ sudo cat cluster.yml
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: rke.onap.cloud
port: "22"
internal_address: ""
role:
- controlplane
- worker
- etcd
hostname_override: ""
user: ubuntu
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: ~/.ssh/onap_rsa
labels: {}
services:
etcd:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
external_urls: []
ca_cert: ""
cert: ""
key: ""
path: ""
snapshot: null
retention: ""
creation: ""
kube-api:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
service_cluster_ip_range: 10.43.0.0/16
service_node_port_range: ""
pod_security_policy: false
kube-controller:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
cluster_cidr: 10.42.0.0/16
service_cluster_ip_range: 10.43.0.0/16
scheduler:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
kubelet:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
cluster_domain: cluster.local
infra_container_image: ""
cluster_dns_server: 10.43.0.10
fail_swap_on: false
kubeproxy:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
network:
plugin: canal
options: {}
authentication:
strategy: x509
options: {}
sans: []
system_images:
etcd: rancher/coreos-etcd:v3.2.18
alpine: rancher/rke-tools:v0.1.15
nginx_proxy: rancher/rke-tools:v0.1.15
cert_downloader: rancher/rke-tools:v0.1.15
kubernetes_services_sidecar: rancher/rke-tools:v0.1.15
kubedns: rancher/k8s-dns-kube-dns-amd64:1.14.10
dnsmasq: rancher/k8s-dns-dnsmasq-nanny-amd64:1.14.10
kubedns_sidecar: rancher/k8s-dns-sidecar-amd64:1.14.10
kubedns_autoscaler: rancher/cluster-proportional-autoscaler-amd64:1.0.0
kubernetes: rancher/hyperkube:v1.11.6-rancher1
flannel: rancher/coreos-flannel:v0.10.0
flannel_cni: rancher/coreos-flannel-cni:v0.3.0
calico_node: rancher/calico-node:v3.1.3
calico_cni: rancher/calico-cni:v3.1.3
calico_controllers: ""
calico_ctl: rancher/calico-ctl:v2.0.0
canal_node: rancher/calico-node:v3.1.3
canal_cni: rancher/calico-cni:v3.1.3
canal_flannel: rancher/coreos-flannel:v0.10.0
wave_node: weaveworks/weave-kube:2.1.2
weave_cni: weaveworks/weave-npc:2.1.2
pod_infra_container: rancher/pause-amd64:3.1
ingress: rancher/nginx-ingress-controller:0.16.2-rancher1
ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.4
metrics_server: rancher/metrics-server-amd64:v0.2.1
ssh_key_path: ~/.ssh/onap_rsa
ssh_agent_auth: false
authorization:
mode: rbac
options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
provider: ""
options: {}
node_selector: {}
extra_args: {}
cluster_name: ""
cloud_provider:
name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
address: ""
port: ""
user: ""
ssh_key: ""
ssh_key_path: ""
monitoring:
provider: ""
options: {}
{noformat}
Kubernetes Single Node Developer Installation
sudo ./rke_install.sh -b master -s localhost -e onap -l ubuntu
Kubernetes HA Cluster Production Installation
Design Issues
DI 20190225-1: RKE/Docker version pair
As of 20190215 RKE 0.16 supports Docker 18.06-ce (and 18.09 non-ce) (up from 0.15 supporting 17.03)
https://github.com/docker/docker-ce/releases/tag/v18.06.3-ce
https://github.com/rancher/rke/releases/tag/v0.1.16
ubuntu@a-rke:~$ sudo rke up INFO[0000] Building Kubernetes cluster INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] FATA[0000] Unsupported Docker version found [18.06.3-ce], supported versions are [1.11.x 1.12.x 1.13.x 17.03.x]
DI 20190225-2: RKE upgrade from 0.15 to 0.16 - not working
Does rke remove, regenerate the yaml (or hand upgrade the versions) then rke up
ubuntu@a-rke:~$ sudo rke remove Are you sure you want to remove Kubernetes cluster [y/n]: y INFO[0002] Tearing down Kubernetes cluster INFO[0002] [dialer] Setup tunnel for host [rke.onap.cloud] INFO[0002] [worker] Tearing down Worker Plane.. INFO[0002] [remove/kubelet] Successfully removed container on host [rke.onap.cloud] INFO[0003] [remove/kube-proxy] Successfully removed container on host [rke.onap.cloud] INFO[0003] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] INFO[0003] [worker] Successfully tore down Worker Plane.. INFO[0003] [controlplane] Tearing down the Controller Plane.. INFO[0003] [remove/kube-apiserver] Successfully removed container on host [rke.onap.cloud] INFO[0003] [remove/kube-controller-manager] Successfully removed container on host [rke.onap.cloud] INFO[0004] [remove/kube-scheduler] Successfully removed container on host [rke.onap.cloud] INFO[0004] [controlplane] Host [rke.onap.cloud] is already a worker host, skipping delete kubelet and kubeproxy. INFO[0004] [controlplane] Successfully tore down Controller Plane.. INFO[0004] [etcd] Tearing down etcd plane.. INFO[0004] [remove/etcd] Successfully removed container on host [rke.onap.cloud] INFO[0004] [etcd] Successfully tore down etcd plane.. INFO[0004] [hosts] Cleaning up host [rke.onap.cloud] INFO[0004] [hosts] Cleaning up host [rke.onap.cloud] INFO[0004] [hosts] Running cleaner container on host [rke.onap.cloud] INFO[0005] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] INFO[0005] [hosts] Removing cleaner container on host [rke.onap.cloud] INFO[0005] [hosts] Removing dead container logs on host [rke.onap.cloud] INFO[0006] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] INFO[0006] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] INFO[0006] [hosts] Successfully cleaned up host [rke.onap.cloud] INFO[0006] [hosts] Cleaning up host [rke.onap.cloud] INFO[0006] [hosts] Cleaning up host [rke.onap.cloud] INFO[0006] [hosts] Running cleaner container on host [rke.onap.cloud] INFO[0007] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] INFO[0008] [hosts] Removing cleaner container on host [rke.onap.cloud] INFO[0008] [hosts] Removing dead container logs on host [rke.onap.cloud] INFO[0008] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] INFO[0009] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] INFO[0009] [hosts] Successfully cleaned up host [rke.onap.cloud] INFO[0009] [hosts] Cleaning up host [rke.onap.cloud] INFO[0009] [hosts] Cleaning up host [rke.onap.cloud] INFO[0009] [hosts] Running cleaner container on host [rke.onap.cloud] INFO[0010] [kube-cleaner] Successfully started [kube-cleaner] container on host [rke.onap.cloud] INFO[0010] [hosts] Removing cleaner container on host [rke.onap.cloud] INFO[0010] [hosts] Removing dead container logs on host [rke.onap.cloud] INFO[0011] [cleanup] Successfully started [rke-log-cleaner] container on host [rke.onap.cloud] INFO[0011] [remove/rke-log-cleaner] Successfully removed container on host [rke.onap.cloud] INFO[0011] [hosts] Successfully cleaned up host [rke.onap.cloud] INFO[0011] Removing local admin Kubeconfig: ./kube_config_cluster.yml INFO[0011] Local admin Kubeconfig removed successfully INFO[0011] Cluster removed successfully ubuntu@a-rke:~$ rke config --name cluster.ym ubuntu@a-rke:~$ sudo rke up INFO[0000] Building Kubernetes cluster INFO[0000] [dialer] Setup tunnel for host [rke.onap.cloud] INFO[0000] [network] Deploying port listener containers INFO[0001] [network] Successfully started [rke-etcd-port-listener] container on host [rke.onap.cloud] INFO[0001] [network] Successfully started [rke-cp-port-listener] container on host [rke.onap.cloud] INFO[0002] [network] Successfully started [rke-worker-port-listener] container on host [rke.onap.cloud] INFO[0002] [network] Port listener containers deployed successfully INFO[0002] [network] Running control plane -> etcd port checks INFO[0003] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] INFO[0003] [network] Running control plane -> worker port checks INFO[0004] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] INFO[0004] [network] Running workers -> control plane port checks INFO[0005] [network] Successfully started [rke-port-checker] container on host [rke.onap.cloud] INFO[0005] [network] Checking KubeAPI port Control Plane hosts INFO[0005] [network] Removing port listener containers INFO[0005] [remove/rke-etcd-port-listener] Successfully removed container on host [rke.onap.cloud] INFO[0006] [remove/rke-cp-port-listener] Successfully removed container on host [rke.onap.cloud] INFO[0006] [remove/rke-worker-port-listener] Successfully removed container on host [rke.onap.cloud] INFO[0006] [network] Port listener containers removed successfully INFO[0006] [certificates] Attempting to recover certificates from backup on [etcd,controlPlane] hosts INFO[0007] [certificates] No Certificate backup found on [etcd,controlPlane] hosts INFO[0007] [certificates] Generating CA kubernetes certificates INFO[0007] [certificates] Generating Kubernetes API server certficates INFO[0008] [certificates] Generating Kube Controller certificates INFO[0008] [certificates] Generating Kube Scheduler certificates INFO[0008] [certificates] Generating Kube Proxy certificates INFO[0009] [certificates] Generating Node certificate INFO[0009] [certificates] Generating admin certificates and kubeconfig INFO[0009] [certificates] Generating etcd-rke.onap.cloud certificate and key INFO[0009] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates INFO[0009] [certificates] Generating Kubernetes API server proxy client certificates INFO[0010] [certificates] Temporarily saving certs to [etcd,controlPlane] hosts INFO[0016] [certificates] Saved certs to [etcd,controlPlane] hosts INFO[0016] [reconcile] Reconciling cluster state INFO[0016] [reconcile] This is newly generated cluster INFO[0016] [certificates] Deploying kubernetes certificates to Cluster nodes INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes INFO[0022] Pre-pulling kubernetes images INFO[0022] Kubernetes images pulled successfully INFO[0022] [etcd] Building up etcd plane.. INFO[0023] [etcd] Successfully started [etcd] container on host [rke.onap.cloud] INFO[0023] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [rke.onap.cloud] INFO[0028] [certificates] Successfully started [rke-bundle-cert] container on host [rke.onap.cloud] INFO[0029] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [rke.onap.cloud] INFO[0029] [etcd] Successfully started [rke-log-linker] container on host [rke.onap.cloud] INFO[0030] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] INFO[0030] [etcd] Successfully started etcd plane.. INFO[0030] [controlplane] Building up Controller Plane.. INFO[0031] [controlplane] Successfully started [kube-apiserver] container on host [rke.onap.cloud] INFO[0031] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [rke.onap.cloud] INFO[0045] [healthcheck] service [kube-apiserver] on host [rke.onap.cloud] is healthy INFO[0046] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] INFO[0046] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] INFO[0047] [controlplane] Successfully started [kube-controller-manager] container on host [rke.onap.cloud] INFO[0047] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [rke.onap.cloud] INFO[0052] [healthcheck] service [kube-controller-manager] on host [rke.onap.cloud] is healthy INFO[0053] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] INFO[0053] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] INFO[0054] [controlplane] Successfully started [kube-scheduler] container on host [rke.onap.cloud] INFO[0054] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [rke.onap.cloud] INFO[0059] [healthcheck] service [kube-scheduler] on host [rke.onap.cloud] is healthy INFO[0060] [controlplane] Successfully started [rke-log-linker] container on host [rke.onap.cloud] INFO[0060] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] INFO[0060] [controlplane] Successfully started Controller Plane.. INFO[0060] [authz] Creating rke-job-deployer ServiceAccount INFO[0060] [authz] rke-job-deployer ServiceAccount created successfully INFO[0060] [authz] Creating system:node ClusterRoleBinding INFO[0060] [authz] system:node ClusterRoleBinding created successfully INFO[0060] [certificates] Save kubernetes certificates as secrets INFO[0060] [certificates] Successfully saved certificates as kubernetes secret [k8s-certs] INFO[0060] [state] Saving cluster state to Kubernetes INFO[0061] [state] Successfully Saved cluster state to Kubernetes ConfigMap: cluster-state INFO[0061] [state] Saving cluster state to cluster nodes INFO[0061] [state] Successfully started [cluster-state-deployer] container on host [rke.onap.cloud] INFO[0062] [remove/cluster-state-deployer] Successfully removed container on host [rke.onap.cloud] INFO[0062] [worker] Building up Worker Plane.. INFO[0062] [remove/service-sidekick] Successfully removed container on host [rke.onap.cloud] INFO[0063] [worker] Successfully started [kubelet] container on host [rke.onap.cloud] INFO[0063] [healthcheck] Start Healthcheck on service [kubelet] on host [rke.onap.cloud] INFO[0068] [healthcheck] service [kubelet] on host [rke.onap.cloud] is healthy INFO[0069] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] INFO[0070] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] INFO[0070] [worker] Successfully started [kube-proxy] container on host [rke.onap.cloud] INFO[0070] [healthcheck] Start Healthcheck on service [kube-proxy] on host [rke.onap.cloud] INFO[0076] [healthcheck] service [kube-proxy] on host [rke.onap.cloud] is healthy INFO[0076] [worker] Successfully started [rke-log-linker] container on host [rke.onap.cloud] INFO[0077] [remove/rke-log-linker] Successfully removed container on host [rke.onap.cloud] INFO[0077] [worker] Successfully started Worker Plane.. INFO[0077] [sync] Syncing nodes Labels and Taints INFO[0077] [sync] Successfully synced nodes Labels and Taints INFO[0077] [network] Setting up network plugin: canal INFO[0077] [addons] Saving addon ConfigMap to Kubernetes INFO[0077] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-network-plugin INFO[0077] [addons] Executing deploy job.. INFO[0082] [addons] Setting up KubeDNS INFO[0082] [addons] Saving addon ConfigMap to Kubernetes INFO[0082] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-kubedns-addon INFO[0082] [addons] Executing deploy job.. INFO[0087] [addons] KubeDNS deployed successfully.. INFO[0087] [addons] Setting up Metrics Server INFO[0087] [addons] Saving addon ConfigMap to Kubernetes INFO[0087] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-metrics-addon INFO[0087] [addons] Executing deploy job.. INFO[0092] [addons] KubeDNS deployed successfully.. INFO[0092] [ingress] Setting up nginx ingress controller INFO[0092] [addons] Saving addon ConfigMap to Kubernetes INFO[0092] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-ingress-controller INFO[0092] [addons] Executing deploy job.. INFO[0097] [ingress] ingress controller nginx is successfully deployed INFO[0097] [addons] Setting up user addons INFO[0097] [addons] Checking for included user addons WARN[0097] [addons] Unable to determine if is a file path or url, skipping INFO[0097] [addons] Deploying rke-user-includes-addons INFO[0097] [addons] Saving addon ConfigMap to Kubernetes INFO[0097] [addons] Successfully Saved addon to Kubernetes ConfigMap: rke-user-includes-addons INFO[0097] [addons] Executing deploy job.. WARN[0128] Failed to deploy addon execute job [rke-user-includes-addons]: Failed to get job complete status: <nil> INFO[0128] Finished building Kubernetes cluster successfully ubuntu@a-rke:~$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ec26c4bd24b5 846921f0fe0e "/server" 10 minutes ago Up 10 minutes k8s_default-http-backend_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0 f8d5db205e14 8a7739f672b4 "/sidecar --v=2 --lo…" 10 minutes ago Up 10 minutes k8s_sidecar_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0 490461545ae4 rancher/metrics-server-amd64 "/metrics-server --s…" 10 minutes ago Up 10 minutes k8s_metrics-server_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0 aaf03b62bd41 6816817d9dce "/dnsmasq-nanny -v=2…" 10 minutes ago Up 10 minutes k8s_dnsmasq_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0 58ec007db72f 55ffe31ac578 "/kube-dns --domain=…" 10 minutes ago Up 10 minutes k8s_kubedns_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0 0a95c06f6aa6 e183460c484d "/cluster-proportion…" 10 minutes ago Up 10 minutes k8s_autoscaler_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0 968a7c99b210 rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_default-http-backend-797c5bc547-45msr_ingress-nginx_0eddfe19-394e-11e9-b708-000d3a0e23f3_0 69969b331e49 rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_metrics-server-97bc649d5-q84tz_kube-system_0c566ec8-394e-11e9-b708-000d3a0e23f3_0 baa5f03c16ff rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_kube-dns-7588d5b5f5-6k286_kube-system_08c13783-394e-11e9-b708-000d3a0e23f3_0 82b2a9f640cb rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_kube-dns-autoscaler-5db9bbb766-6slz7_kube-system_08b5495c-394e-11e9-b708-000d3a0e23f3_0 953a4d4be0c1 df4469c42185 "/usr/bin/dumb-init …" 10 minutes ago Up 10 minutes k8s_nginx-ingress-controller_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0 cce552840749 rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_nginx-ingress-controller-dfhp8_ingress-nginx_0ed3bdbf-394e-11e9-b708-000d3a0e23f3_0 baa65f9c6f97 f0fad859c909 "/opt/bin/flanneld -…" 10 minutes ago Up 10 minutes k8s_kube-flannel_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0 1736ce68f41a 9f355e076ea7 "/install-cni.sh" 10 minutes ago Up 10 minutes k8s_install-cni_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0 615d3f702ee7 7eca10056c8e "start_runit" 10 minutes ago Up 10 minutes k8s_calico-node_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0 1c4a702f0f18 rancher/pause-amd64:3.1 "/pause" 10 minutes ago Up 10 minutes k8s_POD_canal-lc6g6_kube-system_05904de9-394e-11e9-b708-000d3a0e23f3_0 0da1cada08e1 rancher/hyperkube:v1.11.6-rancher1 "/opt/rke-tools/entr…" 10 minutes ago Up 10 minutes kube-proxy 57f44998f34a rancher/hyperkube:v1.11.6-rancher1 "/opt/rke-tools/entr…" 11 minutes ago Up 11 minutes kubelet 50f424c4daec rancher/hyperkube:v1.11.6-rancher1 "/opt/rke-tools/entr…" 11 minutes ago Up 11 minutes kube-scheduler 502d327912d9 rancher/hyperkube:v1.11.6-rancher1 "/opt/rke-tools/entr…" 11 minutes ago Up 11 minutes kube-controller-manager 9fc706bbf3a5 rancher/hyperkube:v1.11.6-rancher1 "/opt/rke-tools/entr…" 11 minutes ago Up 11 minutes kube-apiserver 2e7630c2047c rancher/coreos-etcd:v3.2.18 "/usr/local/bin/etcd…" 11 minutes ago Up 11 minutes etcd fef566337eb6 rancher/rke-tools:v0.1.15 "/opt/rke-tools/rke-…" 26 minutes ago Up 26 minutes etcd-rolling-snapshots amdocs@obriensystemsu0:~$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE ingress-nginx default-http-backend-797c5bc547-m8hbx 1/1 Running 0 1h ingress-nginx nginx-ingress-controller-2v7w7 1/1 Running 0 1h kube-system canal-thmfg 3/3 Running 0 1h kube-system kube-dns-7588d5b5f5-j66s8 3/3 Running 0 1h kube-system kube-dns-autoscaler-5db9bbb766-rg5n8 1/1 Running 0 1h kube-system metrics-server-97bc649d5-jd2rr 1/1 Running 0 1h kube-system rke-ingress-controller-deploy-job-znp9n 0/1 Completed 0 1h kube-system rke-kubedns-addon-deploy-job-dzxsj 0/1 Completed 0 1h kube-system rke-metrics-addon-deploy-job-gpm4j 0/1 Completed 0 1h kube-system rke-network-plugin-deploy-job-kqdds 0/1 Completed 0 1h kube-system tiller-deploy-69458576b-khgr5 1/1 Running 0 1h
DI 20190226-1: RKE up segmentation fault on 0.1.16 - use correct user
amdocs@obriensystemsu0:~$ sudo rke up Segmentation fault (core dumped) # issue was I was using ubuntu as the yml user not amdocs in this case for a particular VM
DI 20190227-1: Verify no 110 pod limit per VM
https://forums.rancher.com/t/solved-setting-max-pods/11866
kubelet:
image: ""
extra_args:
max-pods: 900
DI 20190228-1: deploy casablanca MR to RKE under K8S 1.11.6, Docker 18.06, Helm 2.12.3
sudo git clone https://gerrit.onap.org/r/logging-analytics sudo wget https://git.onap.org/oom/plain/kubernetes/onap/resources/environments/dev.yaml sudo cp dev.yaml dev0.yaml sudo vi dev0.yaml sudo cp dev0.yaml dev1.yaml sudo cp logging-analytics/deploy/cd.sh . sudo ./cd.sh -b casablanca -e onap -p false nexus3.onap.org:10001 -f true -s 300 -c true -d false -w false -r false no good for helm 2.12.3 deployment - just using 2.9.1 for now - Error: Chart incompatible with Tiller v2.12.3 in the casablanca branch only - flip https://git.onap.org/oom/tree/kubernetes/onap/Chart.yaml?h=casablanca#n24 tillerVersion: "~2.9.1"
DI 20190305-1: Azure 256G VM full ONAP Testing
obrienbiometrics:oom michaelobrien$ ssh ubuntu@onap-dmz.onap.cloud ./oom_deployment.sh -b master -s rke.onap.cloud -e onap -r a_rke0_master -t _arm_deploy_onap_cd.json -p _arm_deploy_onap_rke_z_parameters.json
DI 20190425: HA RKE install Testing
Manual first for RC0, later retrofit the script in https://git.onap.org/oom/tree/kubernetes/contrib/tools/rke/rke_setup.sh and move/adjust the heat template in https://git.onap.org/logging-analytics/tree/deploy/heat/logging_openstack_13_16g.yaml
Installing on 6 nodes on AWS (windriver is having an issue right now).
We are good on RKE 0.2.1, Ubuntu 18.04 / Kubernetes/kubectl 1.13.5 / Helm 2.13.1 / Docker 18.09.5
https://github.com/rancher/rke/releases RKE 0.2.2 has experimental k8s 1.14 support - running with 0.2.1 for now
Just need to test onap deployments
I'll do the NFS/EFS later before deployment
# on all VMs (control, etcd, worker)
# move the key to all vms
scp ~/wse_onap/onap_rsa ubuntu@rke0.onap.info:~/
sudo curl https://releases.rancher.com/install-docker/18.09.sh | sh
sudo usermod -aG docker ubuntu
# nfs server
# on control/etcd nodes only
# from script
sudo wget https://github.com/rancher/rke/releases/download/v0.2.1/rke_linux-amd64
mv rke_linux-amd64 rke
sudo chmod +x rke
sudo mv ./rke /usr/local/bin/rke
# one time setup of the yaml or use the generated one
ubuntu@ip-172-31-38-182:~$ sudo rke config
[+] Cluster Level SSH Private Key Path [~/.ssh/id_rsa]: ~/.ssh/onap_rsa
[+] Number of Hosts [1]: 6
[+] SSH Address of host (1) [none]: 3.14.102.175
[+] SSH Port of host (1) [22]:
[+] SSH Private Key Path of host (3.14.102.175) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (3.14.102.175) [ubuntu]: ubuntu
[+] Is host (3.14.102.175) a Control Plane host (y/n)? [y]: y
[+] Is host (3.14.102.175) a Worker host (y/n)? [n]: n
[+] Is host (3.14.102.175) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (3.14.102.175) [none]:
[+] Internal IP of host (3.14.102.175) [none]:
[+] Docker socket path on host (3.14.102.175) [/var/run/docker.sock]:
[+] SSH Address of host (2) [none]: 18.220.62.6
[+] SSH Port of host (2) [22]:
[+] SSH Private Key Path of host (18.220.62.6) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.220.62.6) [ubuntu]:
[+] Is host (18.220.62.6) a Control Plane host (y/n)? [y]: y
[+] Is host (18.220.62.6) a Worker host (y/n)? [n]: n
[+] Is host (18.220.62.6) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (18.220.62.6) [none]:
[+] Internal IP of host (18.220.62.6) [none]:
[+] Docker socket path on host (18.220.62.6) [/var/run/docker.sock]:
[+] SSH Address of host (3) [none]: 18.217.96.12
[+] SSH Port of host (3) [22]:
[+] SSH Private Key Path of host (18.217.96.12) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.217.96.12) [ubuntu]:
[+] Is host (18.217.96.12) a Control Plane host (y/n)? [y]: y
[+] Is host (18.217.96.12) a Worker host (y/n)? [n]: n
[+] Is host (18.217.96.12) an etcd host (y/n)? [n]: y
[+] Override Hostname of host (18.217.96.12) [none]:
[+] Internal IP of host (18.217.96.12) [none]:
[+] Docker socket path on host (18.217.96.12) [/var/run/docker.sock]:
[+] SSH Address of host (4) [none]: 18.188.214.137
[+] SSH Port of host (4) [22]:
[+] SSH Private Key Path of host (18.188.214.137) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.188.214.137) [ubuntu]:
[+] Is host (18.188.214.137) a Control Plane host (y/n)? [y]: n
[+] Is host (18.188.214.137) a Worker host (y/n)? [n]: y
[+] Is host (18.188.214.137) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (18.188.214.137) [none]:
[+] Internal IP of host (18.188.214.137) [none]:
[+] Docker socket path on host (18.188.214.137) [/var/run/docker.sock]:
[+] SSH Address of host (5) [none]: 18.220.70.253
[+] SSH Port of host (5) [22]:
[+] SSH Private Key Path of host (18.220.70.253) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (18.220.70.253) [ubuntu]:
[+] Is host (18.220.70.253) a Control Plane host (y/n)? [y]: n
[+] Is host (18.220.70.253) a Worker host (y/n)? [n]: y
[+] Is host (18.220.70.253) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (18.220.70.253) [none]:
[+] Internal IP of host (18.220.70.253) [none]:
[+] Docker socket path on host (18.220.70.253) [/var/run/docker.sock]:
[+] SSH Address of host (6) [none]: 3.17.76.33
[+] SSH Port of host (6) [22]:
[+] SSH Private Key Path of host (3.17.76.33) [none]: ~/.ssh/onap_rsa
[+] SSH User of host (3.17.76.33) [ubuntu]:
[+] Is host (3.17.76.33) a Control Plane host (y/n)? [y]: n
[+] Is host (3.17.76.33) a Worker host (y/n)? [n]: y
[+] Is host (3.17.76.33) an etcd host (y/n)? [n]: n
[+] Override Hostname of host (3.17.76.33) [none]:
[+] Internal IP of host (3.17.76.33) [none]:
[+] Docker socket path on host (3.17.76.33) [/var/run/docker.sock]:
[+] Network Plugin Type (flannel, calico, weave, canal) [canal]:
[+] Authentication Strategy [x509]:
[+] Authorization Mode (rbac, none) [rbac]:
[+] Kubernetes Docker image [rancher/hyperkube:v1.13.5-rancher1]:
[+] Cluster domain [cluster.local]:
[+] Service Cluster IP Range [10.43.0.0/16]:
[+] Enable PodSecurityPolicy [n]:
[+] Cluster Network CIDR [10.42.0.0/16]:
[+] Cluster DNS Service IP [10.43.0.10]:
[+] Add addon manifest URLs or YAML files [no]:
# new
[+] Cluster domain [cluster.local]:
ubuntu@ip-172-31-38-182:~$ sudo rke up
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [certificates] Generating CA kubernetes certificates
INFO[0000] [certificates] Generating Kubernetes API server aggregation layer requestheader client CA certificates
INFO[0000] [certificates] Generating Kubernetes API server certificates
INFO[0000] [certificates] Generating Service account token key
INFO[0000] [certificates] Generating etcd-3.14.102.175 certificate and key
INFO[0000] [certificates] Generating etcd-18.220.62.6 certificate and key
INFO[0001] [certificates] Generating etcd-18.217.96.12 certificate and key
INFO[0001] [certificates] Generating Kube Controller certificates
INFO[0001] [certificates] Generating Kube Scheduler certificates
INFO[0001] [certificates] Generating Kube Proxy certificates
INFO[0001] [certificates] Generating Node certificate
INFO[0001] [certificates] Generating admin certificates and kubeconfig
INFO[0001] [certificates] Generating Kubernetes API server proxy client certificates
INFO[0002] Successfully Deployed state file at [./cluster.rkestate]
INFO[0002] Building Kubernetes cluster
INFO[0002] [dialer] Setup tunnel for host [3.14.102.175]
INFO[0002] [dialer] Setup tunnel for host [18.188.214.137]
INFO[0002] [dialer] Setup tunnel for host [18.220.70.253]
INFO[0002] [dialer] Setup tunnel for host [18.220.62.6]
INFO[0002] [dialer] Setup tunnel for host [18.217.96.12]
INFO[0002] [dialer] Setup tunnel for host [3.17.76.33]
INFO[0002] [network] Deploying port listener containers
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.220.62.6]
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [3.14.102.175]
INFO[0002] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.217.96.12]
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.220.62.6]
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.217.96.12]
INFO[0006] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [3.14.102.175]
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [18.220.62.6]
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [18.217.96.12]
INFO[0007] [network] Successfully started [rke-etcd-port-listener] container on host [3.14.102.175]
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [18.217.96.12]
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [18.220.62.6]
INFO[0008] [network] Successfully started [rke-cp-port-listener] container on host [3.14.102.175]
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.188.214.137]
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [18.220.70.253]
INFO[0008] [network] Pulling image [rancher/rke-tools:v0.1.27] on host [3.17.76.33]
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [3.17.76.33]
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.220.70.253]
INFO[0012] [network] Successfully pulled image [rancher/rke-tools:v0.1.27] on host [18.188.214.137]
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [3.17.76.33]
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [18.220.70.253]
INFO[0013] [network] Successfully started [rke-worker-port-listener] container on host [18.188.214.137]
INFO[0013] [network] Port listener containers deployed successfully
INFO[0013] [network] Running etcd <-> etcd port checks
INFO[0013] [network] Successfully started [rke-port-checker] container on host [18.220.62.6]
INFO[0013] [network] Successfully started [rke-port-checker] container on host [3.14.102.175]
INFO[0013] [network] Successfully started [rke-port-checker] container on host [18.217.96.12]
INFO[0014] [network] Running control plane -> etcd port checks
INFO[0014] [network] Successfully started [rke-port-checker] container on host [18.220.62.6]
INFO[0014] [network] Successfully started [rke-port-checker] container on host [3.14.102.175]
INFO[0014] [network] Successfully started [rke-port-checker] container on host [18.217.96.12]
INFO[0014] [network] Running control plane -> worker port checks
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.220.62.6]
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.217.96.12]
INFO[0015] [network] Successfully started [rke-port-checker] container on host [3.14.102.175]
INFO[0015] [network] Running workers -> control plane port checks
INFO[0015] [network] Successfully started [rke-port-checker] container on host [3.17.76.33]
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.220.70.253]
INFO[0015] [network] Successfully started [rke-port-checker] container on host [18.188.214.137]
INFO[0016] [network] Checking KubeAPI port Control Plane hosts
INFO[0016] [network] Removing port listener containers
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [18.220.62.6]
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [3.14.102.175]
INFO[0016] [remove/rke-etcd-port-listener] Successfully removed container on host [18.217.96.12]
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [18.217.96.12]
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [3.14.102.175]
INFO[0016] [remove/rke-cp-port-listener] Successfully removed container on host [18.220.62.6]
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [18.220.70.253]
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [3.17.76.33]
INFO[0017] [remove/rke-worker-port-listener] Successfully removed container on host [18.188.214.137]
INFO[0017] [network] Port listener containers removed successfully
INFO[0017] [certificates] Deploying kubernetes certificates to Cluster nodes
INFO[0022] [reconcile] Rebuilding and updating local kube config
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0022] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]
INFO[0022] [certificates] Successfully deployed kubernetes certificates to Cluster nodes
INFO[0022] [reconcile] Reconciling cluster state
INFO[0022] [reconcile] This is newly generated cluster
INFO[0022] Pre-pulling kubernetes images
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [3.14.102.175]
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.188.214.137]
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.70.253]
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.217.96.12]
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [3.17.76.33]
INFO[0022] [pre-deploy] Pulling image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.62.6]
INFO[0038] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.62.6]
INFO[0038] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.220.70.253]
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [3.17.76.33]
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.217.96.12]
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [18.188.214.137]
INFO[0039] [pre-deploy] Successfully pulled image [rancher/hyperkube:v1.13.5-rancher1] on host [3.14.102.175]
INFO[0039] Kubernetes images pulled successfully
INFO[0039] [etcd] Building up etcd plane..
INFO[0039] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [3.14.102.175]
INFO[0041] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [3.14.102.175]
INFO[0051] [etcd] Successfully started [etcd] container on host [3.14.102.175]
INFO[0051] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [3.14.102.175]
INFO[0052] [etcd] Successfully started [etcd-rolling-snapshots] container on host [3.14.102.175]
INFO[0057] [certificates] Successfully started [rke-bundle-cert] container on host [3.14.102.175]
INFO[0058] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [3.14.102.175]
INFO[0058] [etcd] Successfully started [rke-log-linker] container on host [3.14.102.175]
INFO[0058] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175]
INFO[0058] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.220.62.6]
INFO[0063] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.220.62.6]
INFO[0069] [etcd] Successfully started [etcd] container on host [18.220.62.6]
INFO[0069] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [18.220.62.6]
INFO[0069] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.220.62.6]
INFO[0075] [certificates] Successfully started [rke-bundle-cert] container on host [18.220.62.6]
INFO[0075] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.220.62.6]
INFO[0076] [etcd] Successfully started [rke-log-linker] container on host [18.220.62.6]
INFO[0076] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6]
INFO[0076] [etcd] Pulling image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.217.96.12]
INFO[0078] [etcd] Successfully pulled image [rancher/coreos-etcd:v3.2.24-rancher1] on host [18.217.96.12]
INFO[0078] [etcd] Successfully started [etcd] container on host [18.217.96.12]
INFO[0078] [etcd] Saving snapshot [etcd-rolling-snapshots] on host [18.217.96.12]
INFO[0078] [etcd] Successfully started [etcd-rolling-snapshots] container on host [18.217.96.12]
INFO[0084] [certificates] Successfully started [rke-bundle-cert] container on host [18.217.96.12]
INFO[0084] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [18.217.96.12]
INFO[0085] [etcd] Successfully started [rke-log-linker] container on host [18.217.96.12]
INFO[0085] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12]
INFO[0085] [etcd] Successfully started etcd plane.. Checking etcd cluster health
INFO[0086] [controlplane] Building up Controller Plane..
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [18.220.62.6]
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.220.62.6]
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [3.14.102.175]
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [3.14.102.175]
INFO[0086] [controlplane] Successfully started [kube-apiserver] container on host [18.217.96.12]
INFO[0086] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [18.217.96.12]
INFO[0098] [healthcheck] service [kube-apiserver] on host [18.220.62.6] is healthy
INFO[0099] [healthcheck] service [kube-apiserver] on host [18.217.96.12] is healthy
INFO[0099] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6]
INFO[0099] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12]
INFO[0099] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6]
INFO[0099] [healthcheck] service [kube-apiserver] on host [3.14.102.175] is healthy
INFO[0099] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12]
INFO[0099] [controlplane] Successfully started [kube-controller-manager] container on host [18.220.62.6]
INFO[0099] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.220.62.6]
INFO[0100] [controlplane] Successfully started [kube-controller-manager] container on host [18.217.96.12]
INFO[0100] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [18.217.96.12]
INFO[0100] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175]
INFO[0100] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175]
INFO[0100] [healthcheck] service [kube-controller-manager] on host [18.220.62.6] is healthy
INFO[0100] [healthcheck] service [kube-controller-manager] on host [18.217.96.12] is healthy
INFO[0100] [controlplane] Successfully started [kube-controller-manager] container on host [3.14.102.175]
INFO[0100] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [3.14.102.175]
INFO[0101] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6]
INFO[0101] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12]
INFO[0101] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6]
INFO[0101] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12]
INFO[0101] [controlplane] Successfully started [kube-scheduler] container on host [18.220.62.6]
INFO[0101] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.220.62.6]
INFO[0101] [healthcheck] service [kube-controller-manager] on host [3.14.102.175] is healthy
INFO[0101] [controlplane] Successfully started [kube-scheduler] container on host [18.217.96.12]
INFO[0101] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [18.217.96.12]
INFO[0102] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175]
INFO[0102] [healthcheck] service [kube-scheduler] on host [18.220.62.6] is healthy
INFO[0102] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175]
INFO[0102] [healthcheck] service [kube-scheduler] on host [18.217.96.12] is healthy
INFO[0103] [controlplane] Successfully started [kube-scheduler] container on host [3.14.102.175]
INFO[0103] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [3.14.102.175]
INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.220.62.6]
INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6]
INFO[0103] [controlplane] Successfully started [rke-log-linker] container on host [18.217.96.12]
INFO[0103] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12]
INFO[0103] [healthcheck] service [kube-scheduler] on host [3.14.102.175] is healthy
INFO[0104] [controlplane] Successfully started [rke-log-linker] container on host [3.14.102.175]
INFO[0104] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175]
INFO[0104] [controlplane] Successfully started Controller Plane..
INFO[0104] [authz] Creating rke-job-deployer ServiceAccount
INFO[0104] [authz] rke-job-deployer ServiceAccount created successfully
INFO[0104] [authz] Creating system:node ClusterRoleBinding
INFO[0104] [authz] system:node ClusterRoleBinding created successfully
INFO[0104] Successfully Deployed state file at [./cluster.rkestate]
INFO[0104] [state] Saving full cluster state to Kubernetes
INFO[0104] [state] Successfully Saved full cluster state to Kubernetes ConfigMap: cluster-state
INFO[0104] [worker] Building up Worker Plane..
INFO[0104] [sidekick] Sidekick container already created on host [18.220.62.6]
INFO[0104] [sidekick] Sidekick container already created on host [3.14.102.175]
INFO[0104] [sidekick] Sidekick container already created on host [18.217.96.12]
INFO[0105] [worker] Successfully started [kubelet] container on host [3.14.102.175]
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [3.14.102.175]
INFO[0105] [worker] Successfully started [kubelet] container on host [18.220.62.6]
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [18.220.62.6]
INFO[0105] [worker] Successfully started [kubelet] container on host [18.217.96.12]
INFO[0105] [healthcheck] Start Healthcheck on service [kubelet] on host [18.217.96.12]
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [18.220.70.253]
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [3.17.76.33]
INFO[0105] [worker] Successfully started [nginx-proxy] container on host [18.188.214.137]
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253]
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33]
INFO[0106] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137]
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33]
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253]
INFO[0106] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137]
INFO[0106] [worker] Successfully started [kubelet] container on host [3.17.76.33]
INFO[0106] [healthcheck] Start Healthcheck on service [kubelet] on host [3.17.76.33]
INFO[0107] [worker] Successfully started [kubelet] container on host [18.220.70.253]
INFO[0107] [healthcheck] Start Healthcheck on service [kubelet] on host [18.220.70.253]
INFO[0107] [worker] Successfully started [kubelet] container on host [18.188.214.137]
INFO[0107] [healthcheck] Start Healthcheck on service [kubelet] on host [18.188.214.137]
INFO[0111] [healthcheck] service [kubelet] on host [18.220.62.6] is healthy
INFO[0111] [healthcheck] service [kubelet] on host [18.217.96.12] is healthy
INFO[0111] [healthcheck] service [kubelet] on host [3.14.102.175] is healthy
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [18.220.62.6]
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [3.14.102.175]
INFO[0111] [worker] Successfully started [rke-log-linker] container on host [18.217.96.12]
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6]
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12]
INFO[0112] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175]
INFO[0112] [worker] Successfully started [kube-proxy] container on host [18.220.62.6]
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.220.62.6]
INFO[0112] [worker] Successfully started [kube-proxy] container on host [18.217.96.12]
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.217.96.12]
INFO[0112] [worker] Successfully started [kube-proxy] container on host [3.14.102.175]
INFO[0112] [healthcheck] Start Healthcheck on service [kube-proxy] on host [3.14.102.175]
INFO[0113] [healthcheck] service [kube-proxy] on host [18.220.62.6] is healthy
INFO[0113] [healthcheck] service [kube-proxy] on host [18.217.96.12] is healthy
INFO[0113] [healthcheck] service [kube-proxy] on host [3.14.102.175] is healthy
INFO[0113] [healthcheck] service [kubelet] on host [18.220.70.253] is healthy
INFO[0113] [healthcheck] service [kubelet] on host [3.17.76.33] is healthy
INFO[0113] [healthcheck] service [kubelet] on host [18.188.214.137] is healthy
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.220.62.6]
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.217.96.12]
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [3.14.102.175]
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33]
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253]
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [18.220.62.6]
INFO[0113] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137]
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [18.217.96.12]
INFO[0113] [remove/rke-log-linker] Successfully removed container on host [3.14.102.175]
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33]
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253]
INFO[0114] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137]
INFO[0114] [worker] Successfully started [kube-proxy] container on host [3.17.76.33]
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [3.17.76.33]
INFO[0114] [worker] Successfully started [kube-proxy] container on host [18.220.70.253]
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.220.70.253]
INFO[0114] [worker] Successfully started [kube-proxy] container on host [18.188.214.137]
INFO[0114] [healthcheck] Start Healthcheck on service [kube-proxy] on host [18.188.214.137]
INFO[0114] [healthcheck] service [kube-proxy] on host [18.220.70.253] is healthy
INFO[0114] [healthcheck] service [kube-proxy] on host [3.17.76.33] is healthy
INFO[0115] [healthcheck] service [kube-proxy] on host [18.188.214.137] is healthy
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [18.220.70.253]
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [3.17.76.33]
INFO[0115] [worker] Successfully started [rke-log-linker] container on host [18.188.214.137]
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [18.220.70.253]
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [3.17.76.33]
INFO[0115] [remove/rke-log-linker] Successfully removed container on host [18.188.214.137]
INFO[0115] [worker] Successfully started Worker Plane..
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.188.214.137]
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [3.17.76.33]
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.220.70.253]
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.220.62.6]
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [18.217.96.12]
INFO[0116] [cleanup] Successfully started [rke-log-cleaner] container on host [3.14.102.175]
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [3.17.76.33]
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.188.214.137]
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.220.62.6]
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.220.70.253]
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [18.217.96.12]
INFO[0116] [remove/rke-log-cleaner] Successfully removed container on host [3.14.102.175]
INFO[0116] [sync] Syncing nodes Labels and Taints
INFO[0117] [sync] Successfully synced nodes Labels and Taints
INFO[0117] [network] Setting up network plugin: canal
INFO[0117] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes
INFO[0117] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes
INFO[0117] [addons] Executing deploy job rke-network-plugin
INFO[0122] [addons] Setting up kube-dns
INFO[0122] [addons] Saving ConfigMap for addon rke-kube-dns-addon to Kubernetes
INFO[0122] [addons] Successfully saved ConfigMap for addon rke-kube-dns-addon to Kubernetes
INFO[0122] [addons] Executing deploy job rke-kube-dns-addon
INFO[0127] [addons] kube-dns deployed successfully
INFO[0127] [dns] DNS provider kube-dns deployed successfully
INFO[0127] [addons] Setting up Metrics Server
INFO[0127] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes
INFO[0127] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes
INFO[0127] [addons] Executing deploy job rke-metrics-addon
INFO[0132] [addons] Metrics Server deployed successfully
INFO[0132] [ingress] Setting up nginx ingress controller
INFO[0132] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes
INFO[0132] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes
INFO[0132] [addons] Executing deploy job rke-ingress-controller
INFO[0137] [ingress] ingress controller nginx deployed successfully
INFO[0137] [addons] Setting up user addons
INFO[0137] [addons] no user addons defined
INFO[0137] Finished building Kubernetes cluster successfully
# finish kubectl install
sudo curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.13.5/bin/linux/amd64/kubectl
sudo chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
sudo mkdir ~/.kube
# finish helm
#https://github.com/helm/helm/releases
# there is no helm 2.12.5 - last is 2.12.3 - trying 2.13.1
wget http://storage.googleapis.com/kubernetes-helm/helm-v2.13.1-linux-amd64.tar.gz
sudo tar -zxvf helm-v2.13.1-linux-amd64.tar.gz
sudo cp kube_config_cluster.yml ~/.kube/config
sudo chmod 777 ~/.kube/config
# test
ubuntu@ip-172-31-38-182:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx default-http-backend-78fccfc5d9-f6z25 1/1 Running 0 17m 10.42.5.2 18.220.70.253 <none> <none>
ingress-nginx nginx-ingress-controller-2zxs7 1/1 Running 0 17m 18.188.214.137 18.188.214.137 <none> <none>
ingress-nginx nginx-ingress-controller-6b7gs 1/1 Running 0 17m 3.17.76.33 3.17.76.33 <none> <none>
ingress-nginx nginx-ingress-controller-nv4qg 1/1 Running 0 17m 18.220.70.253 18.220.70.253 <none> <none>
kube-system canal-48579 2/2 Running 0 17m 18.220.62.6 18.220.62.6 <none> <none>
kube-system canal-6skkm 2/2 Running 0 17m 18.188.214.137 18.188.214.137 <none> <none>
kube-system canal-9xmxv 2/2 Running 0 17m 18.217.96.12 18.217.96.12 <none> <none>
kube-system canal-c582x 2/2 Running 0 17m 18.220.70.253 18.220.70.253 <none> <none>
kube-system canal-whbck 2/2 Running 0 17m 3.14.102.175 3.14.102.175 <none> <none>
kube-system canal-xbbnh 2/2 Running 0 17m 3.17.76.33 3.17.76.33 <none> <none>
kube-system kube-dns-58bd5b8dd7-6mcm7 3/3 Running 0 17m 10.42.3.3 3.17.76.33 <none> <none>
kube-system kube-dns-58bd5b8dd7-cd5dg 3/3 Running 0 17m 10.42.4.2 18.188.214.137 <none> <none>
kube-system kube-dns-autoscaler-77bc5fd84-p4zfd 1/1 Running 0 17m 10.42.3.2 3.17.76.33 <none> <none>
kube-system metrics-server-58bd5dd8d7-kftjn 1/1 Running 0 17m 10.42.3.4 3.17.76.33 <none> <none>
# install tiller
ubuntu@ip-172-31-38-182:~$ kubectl -n kube-system create serviceaccount tiller
serviceaccount/tiller created
ubuntu@ip-172-31-38-182:~$ kubectl create clusterrolebinding tiller --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller created
ubuntu@ip-172-31-38-182:~$ helm init --service-account tiller
Creating /home/ubuntu/.helm
Creating /home/ubuntu/.helm/repository
Creating /home/ubuntu/.helm/repository/cache
Creating /home/ubuntu/.helm/repository/local
Creating /home/ubuntu/.helm/plugins
Creating /home/ubuntu/.helm/starters
Creating /home/ubuntu/.helm/cache/archive
Creating /home/ubuntu/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /home/ubuntu/.helm.
Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.
Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
ubuntu@ip-172-31-38-182:~$ kubectl -n kube-system rollout status deploy/tiller-deploy
deployment "tiller-deploy" successfully rolled out
ubuntu@ip-172-31-38-182:~$ sudo helm init --upgrade
$HELM_HOME has been configured at /home/ubuntu/.helm.
Tiller (the Helm server-side component) has been upgraded to the current version.
Happy Helming!
ubuntu@ip-172-31-38-182:~$ sudo helm version
Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
ubuntu@ip-172-31-38-182:~$ sudo helm serve &
[1] 706
ubuntu@ip-172-31-38-182:~$ Regenerating index. This may take a moment.
Now serving you on 127.0.0.1:8879
ubuntu@ip-172-31-38-182:~$ sudo helm list
ubuntu@ip-172-31-38-182:~$ sudo helm repo add local http://127.0.0.1:8879
"local" has been added to your repositories
ubuntu@ip-172-31-38-182:~$ sudo helm repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com
local http://127.0.0.1:8879
ubuntu@ip-172-31-38-182:~$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
18.188.214.137 Ready worker 22m v1.13.5 18.188.214.137 <none> Ubuntu 18.04.1 LTS 4.15.0-1021-aws docker://18.9.5
18.217.96.12 Ready controlplane,etcd 22m v1.13.5 18.217.96.12 <none> Ubuntu 18.04.1 LTS 4.15.0-1021-aws docker://18.9.5
18.220.62.6 Ready controlplane,etcd 22m v1.13.5 18.220.62.6 <none> Ubuntu 18.04.1 LTS 4.15.0-1021-aws docker://18.9.5
18.220.70.253 Ready worker 22m v1.13.5 18.220.70.253 <none> Ubuntu 18.04.1 LTS 4.15.0-1021-aws docker://18.9.5
3.14.102.175 Ready controlplane,etcd 22m v1.13.5 3.14.102.175 <none> Ubuntu 18.04.1 LTS 4.15.0-1021-aws docker://18.9.5
3.17.76.33 Ready worker 22m v1.13.5 3.17.76.33 <none> Ubuntu 18.04.1 LTS 4.15.0-1021-aws docker://18.9.5
# install make
sudo apt-get install make -y
# install nfs/efs
ubuntu@ip-172-31-38-182:~$ sudo apt-get install nfs-common -y
ubuntu@ip-172-31-38-182:~$ sudo mkdir /dockerdata-nfs
ubuntu@ip-172-31-38-182:~$ sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-5fd6ab26.efs.us-east-2.amazonaws.com:/ /dockerdata-nfs
# check
sudo nohup ./cd.sh -b master -e onap -p false -n nexus3.onap.org:10001 -f false -s 600 -c false -d false -w false -r false &
ubuntu@ip-172-31-38-182:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx default-http-backend-78fccfc5d9-f6z25 1/1 Running 0 103m 10.42.5.2 18.220.70.253 <none> <none>
ingress-nginx nginx-ingress-controller-2zxs7 1/1 Running 0 103m 18.188.214.137 18.188.214.137 <none> <none>
ingress-nginx nginx-ingress-controller-6b7gs 1/1 Running 0 103m 3.17.76.33 3.17.76.33 <none> <none>
ingress-nginx nginx-ingress-controller-nv4qg 1/1 Running 0 103m 18.220.70.253 18.220.70.253 <none> <none>
kube-system canal-48579 2/2 Running 0 103m 18.220.62.6 18.220.62.6 <none> <none>
kube-system canal-6skkm 2/2 Running 0 103m 18.188.214.137 18.188.214.137 <none> <none>
kube-system canal-9xmxv 2/2 Running 0 103m 18.217.96.12 18.217.96.12 <none> <none>
kube-system canal-c582x 2/2 Running 0 103m 18.220.70.253 18.220.70.253 <none> <none>
kube-system canal-whbck 2/2 Running 0 103m 3.14.102.175 3.14.102.175 <none> <none>
kube-system canal-xbbnh 2/2 Running 0 103m 3.17.76.33 3.17.76.33 <none> <none>
kube-system kube-dns-58bd5b8dd7-6mcm7 3/3 Running 0 103m 10.42.3.3 3.17.76.33 <none> <none>
kube-system kube-dns-58bd5b8dd7-cd5dg 3/3 Running 0 103m 10.42.4.2 18.188.214.137 <none> <none>
kube-system kube-dns-autoscaler-77bc5fd84-p4zfd 1/1 Running 0 103m 10.42.3.2 3.17.76.33 <none> <none>
kube-system metrics-server-58bd5dd8d7-kftjn 1/1 Running 0 103m 10.42.3.4 3.17.76.33 <none> <none>
kube-system tiller-deploy-5f4fc5bcc6-gc4tc 1/1 Running 0 84m 10.42.5.3 18.220.70.253 <none> <none>
onap onap-aai-aai-587cb79c6d-mzpbs 0/1 Init:0/1 1 13m 10.42.5.17 18.220.70.253 <none> <none>
onap onap-aai-aai-babel-8c755bcfc-kmzdm 2/2 Running 0 29m 10.42.5.8 18.220.70.253 <none> <none>
onap onap-aai-aai-champ-78b9d7f68b-98tm9 0/2 Init:0/1 2 29m 10.42.3.6 3.17.76.33 <none> <none>
onap onap-aai-aai-data-router-64fcfbc5bb-wkkvz 1/2 CrashLoopBackOff 9 29m 10.42.5.7 18.220.70.253 <none> <none>
onap onap-aai-aai-elasticsearch-6dcf5d9966-j7z67 1/1 Running 0 29m 10.42.4.4 18.188.214.137 <none> <none>
onap onap-aai-aai-gizmo-5bddb87589-zn8pl 2/2 Running 0 29m 10.42.4.3 18.188.214.137 <none> <none>
onap onap-aai-aai-graphadmin-774f9d698f-f8lwv 0/2 Init:0/1 2 29m 10.42.5.4 18.220.70.253 <none> <none>
onap onap-aai-aai-graphadmin-create-db-schema-94q4l 0/1 Init:Error 0 18m 10.42.4.16 18.188.214.137 <none> <none>
onap onap-aai-aai-graphadmin-create-db-schema-s42pq 0/1 Init:0/1 0 7m54s 10.42.5.20 18.220.70.253 <none> <none>
onap onap-aai-aai-graphadmin-create-db-schema-tsvcw 0/1 Init:Error 0 29m 10.42.4.5 18.188.214.137 <none> <none>
onap onap-aai-aai-modelloader-845fc684bd-r7mdw 2/2 Running 0 29m 10.42.4.6 18.188.214.137 <none> <none>
onap onap-aai-aai-resources-67f8dfcbdb-kz6cp 0/2 Init:0/1 2 29m 10.42.5.11 18.220.70.253 <none> <none>
onap onap-aai-aai-schema-service-6c56b45b7c-7zlfz 2/2 Running 0 29m 10.42.3.7 3.17.76.33 <none> <none>
onap onap-aai-aai-search-data-5d8d7759b8-flxwj 2/2 Running 0 29m 10.42.3.9 3.17.76.33 <none> <none>
onap onap-aai-aai-sparky-be-8444df749c-mzc2n 0/2 Init:0/1 0 29m 10.42.4.11 18.188.214.137 <none> <none>
onap onap-aai-aai-spike-54ff77787f-d678x 2/2 Running 0 29m 10.42.5.6 18.220.70.253 <none> <none>
onap onap-aai-aai-traversal-6ff868f477-lzv2f 0/2 Init:0/1 2 29m 10.42.3.8 3.17.76.33 <none> <none>
onap onap-aai-aai-traversal-update-query-data-9g2b8 0/1 Init:0/1 2 29m 10.42.5.12 18.220.70.253 <none> <none>
onap onap-dmaap-dbc-pg-0 1/1 Running 0 29m 10.42.5.9 18.220.70.253 <none> <none>
onap onap-dmaap-dbc-pg-1 1/1 Running 0 26m 10.42.3.14 3.17.76.33 <none> <none>
onap onap-dmaap-dbc-pgpool-8666b57857-97zjc 1/1 Running 0 29m 10.42.5.5 18.220.70.253 <none> <none>
onap onap-dmaap-dbc-pgpool-8666b57857-vr8gk 1/1 Running 0 29m 10.42.4.8 18.188.214.137 <none> <none>
onap onap-dmaap-dmaap-bc-745995bf74-m6hhq 0/1 Init:0/2 2 29m 10.42.4.12 18.188.214.137 <none> <none>
onap onap-dmaap-dmaap-bc-post-install-6ff4j 1/1 Running 0 29m 10.42.4.9 18.188.214.137 <none> <none>
onap onap-dmaap-dmaap-dr-db-0 1/1 Running 0 29m 10.42.4.10 18.188.214.137 <none> <none>
onap onap-dmaap-dmaap-dr-db-1 1/1 Running 1 24m 10.42.5.15 18.220.70.253 <none> <none>
onap onap-dmaap-dmaap-dr-node-0 2/2 Running 0 29m 10.42.3.11 3.17.76.33 <none> <none>
onap onap-dmaap-dmaap-dr-prov-fbf6c94f5-v9bmq 2/2 Running 0 29m 10.42.5.10 18.220.70.253 <none> <none>
onap onap-dmaap-message-router-0 1/1 Running 0 29m 10.42.4.14 18.188.214.137 <none> <none>
onap onap-dmaap-message-router-kafka-0 1/1 Running 1 29m 10.42.5.13 18.220.70.253 <none> <none>
onap onap-dmaap-message-router-kafka-1 1/1 Running 1 29m 10.42.3.13 3.17.76.33 <none> <none>
onap onap-dmaap-message-router-kafka-2 1/1 Running 0 29m 10.42.4.15 18.188.214.137 <none> <none>
onap onap-dmaap-message-router-mirrormaker-8587c4c9cf-lfnd8 0/1 CrashLoopBackOff 9 29m 10.42.4.7 18.188.214.137 <none> <none>
onap onap-dmaap-message-router-zookeeper-0 1/1 Running 0 29m 10.42.5.14 18.220.70.253 <none> <none>
onap onap-dmaap-message-router-zookeeper-1 1/1 Running 0 29m 10.42.4.13 18.188.214.137 <none> <none>
onap onap-dmaap-message-router-zookeeper-2 1/1 Running 0 29m 10.42.3.12 3.17.76.33 <none> <none>
onap onap-nfs-provisioner-nfs-provisioner-57c999dc57-mdcw5 1/1 Running 0 24m 10.42.3.15 3.17.76.33 <none> <none>
onap onap-robot-robot-677bdbb454-zj9jk 1/1 Running 0 24m 10.42.5.16 18.220.70.253 <none> <none>
onap onap-so-so-8569947cbd-jn5x4 0/1 Init:0/1 1 13m 10.42.4.19 18.188.214.137 <none> <none>
onap onap-so-so-bpmn-infra-78c8fd665d-b47qn 0/1 Init:0/1 1 13m 10.42.3.16 3.17.76.33 <none> <none>
onap onap-so-so-catalog-db-adapter-565f9767ff-lvbgx 0/1 Init:0/1 1 13m 10.42.3.17 3.17.76.33 <none> <none>
onap onap-so-so-mariadb-config-job-d9sdb 0/1 Init:0/2 0 3m37s 10.42.3.20 3.17.76.33 <none> <none>
onap onap-so-so-mariadb-config-job-rkqdl 0/1 Init:Error 0 13m 10.42.5.19 18.220.70.253 <none> <none>
onap onap-so-so-monitoring-69b9fdd94c-dks4v 1/1 Running 0 13m 10.42.4.18 18.188.214.137 <none> <none>
onap onap-so-so-openstack-adapter-5f9cf896d7-mgbdd 0/1 Init:0/1 1 13m 10.42.4.17 18.188.214.137 <none> <none>
onap onap-so-so-request-db-adapter-5c9bfd7b57-2krnp 0/1 Init:0/1 1 13m 10.42.3.18 3.17.76.33 <none> <none>
onap onap-so-so-sdc-controller-6fb5cf5775-bsxhm 0/1 Init:0/1 1 13m 10.42.4.20 18.188.214.137 <none> <none>
onap onap-so-so-sdnc-adapter-8555689c75-r6vkb 1/1 Running 0 13m 10.42.5.18 18.220.70.253 <none> <none>
onap onap-so-so-vfc-adapter-68fccc8bb8-c56t2 0/1 Init:0/1 1 13m 10.42.4.21 18.188.214.137 <none> <none>
onap onap-so-so-vnfm-adapter-65c4c5944b-72nlf 1/1 Running 0 13m 10.42.3.19 3.17.76.33 <none> <none>
# on worker nodes only
# nfs client
DI 20190507: ARM support using RKE 0.2.1 ARM friendly install
LOG-331 - Getting issue details... STATUS
a1.4xlarge $0.408
ami-0b9bd0b532ebcf4c9
Notes
Pre-RKE installation details in Cloud Native Deployment