This page details the current state of log streaming in ONAP and plans to increase compliance towards the goal of providing end to end log tracing (ONAP transactions, onap system, undercloud).
ONAP Log Streaming Deployment View
Analysis
Compliance
Streaming
Format
Library
Existing ONAP Logging State - Infrastructure and Schema
Deployment Dependency Tree
- LOG-924Getting issue details... STATUS
tested sdnc via sudo ./cd.sh -b master -e onap -p false -a false -t robot -x 1 -z false -f false -s 600 -c false -d true -w false -r false
where dmaap, consul and sdc are raised first via dev0.yaml and sdnc is in dev1.yaml
Note: these dependencies are at the lowest deployment level and represent a partial view of the REST/API dependency tree - they do not reflect any compile time or runtime/injection code dependencies (pom.xml) Use fo any containers stuck at the 0/1 Init:0/1 stage - these are likely waiting on dependent containers check the --container-name kv pair in StatefulSet/Deployment.yaml:spec:intiContainers:args or the corresponding defined chart/container names in values.yaml:config: 106 sets in 87 files overall order aaf<-aai aaf<-oof music<-oof dmaap<-aai dmaap<-pomba dmaap<-sdnc consul<-sdnc sdc<-sdnc consul<-dcaegen2 msb<-dcaegen2 order dmaap, aaf, msb, consul, dcaegen2, aai, oof, sdc, sdnc, pomba valid sets of pods aaf aai, dmaap, aaf dcaegen2, consul, msb oof, aaf, music pomba, dmaap sdnc, consul, sdc ..the rest can be singular aaf aaf-cm aaf-locate aaf-fs aaf-locate aaf-gui aaf-cm aaf-hello aaf-locate aaf-locate aaf-service aaf-oauth aaf-locate aaf-service aaf-cs aaf-sms aaf-sms-quorumclient (via job) aaf-sms-vault aaf-sms-vault-backend aai aai aai-resources aai-traversal aai-graphadmin aai-champ aai-cassandra aai-graphadmin aai-cassandra aai-resources aai-cassandra aai-sparky-be aai-elasticsearch aai-search-data aai aai-spike message-router-kafka aai-traversal aai aai-cassandra aaf-locate (conditional) appc appc mariadb-galera appc-ansible-server appc clamp clamp mariadb clamp-dash-kibana clamp-dash-es clamp-dash-logstash clamp-dash-es common controller-blueprints mariadb-galera mongo *-nfs-provisioner mysql *-nfs-provisioner dgbuilder network-name-gen mariadb-galera dcaegen2 dcae-bootstrap dcae-cloudify-manager consul-server msb-discovery kube2msb dep-health-init hbase dmaap dmaap-bus-controller postgres dmaap-dr-node dmaap-dr-prov dmaap-dr-prov mariadb message-router kafka zookeeper message-router-kafka zookeeper log log-kibana log-elasticsearch log-logstash log-elasticsearch msb kube2msb msb-discovery msb-discovery msb-consul msb-eag msb-discovery msb-iag msb-discovery music music-cassandra music-tomcat zookeeper oof oof-has-api oof-has-controller aaf-service oof-has-controller music-tomcat aaf-sms oof-has-data music-tomcat oof-has-reservation music-tomcat oof-has-service music-tomcat policy policy mariadb brmsgw pap drools mariadb nexus pdb pap pomba pomba-contextaggregator message-router pomba-kibana pomba-elasticsearch pomba-data-router pomba-search-data pomba-search-data pomba-elasticsearch portal portal-widget portal-db portal-sdk portal-db sdc sdc-dcae-be common.name sdc-be sdc-dcae-dt sdc-dcae-be sdc-dcae-fe sdc-dcae-be sdc-dcae-tosca-lab sdc-dcae-be sdc-fe sdc-kb sdc-wfd-fe sdc-wfd-be sdnc sdnc mysql sdnc-ansible-server sdnc dmaap-listener mysql sdnc message-router sdnc-portal mysql / sdnc-db sdnc sdnc-prom sdnc consul ueb-listener mysql sdnc sdc-be message-router so so mariadb so-bpmn-infra mariadb so-catalog-db-adapter mariadb so-openstack-adapter mariadb so-request-db-adapter mariadb so-sdc-adapter mariadb so-sdc-controller mariadb so-vfc-adapter mariadb vfc vfc-catalog vfc-db vfc-ems-driver mariadb // commented vfc-generic-vnfm-driver mariadb // commented vfc-hauwei-vnfm-driver mariadb // commented vfc-juju-vnfm-driver mariadb // commented vfc-multivim-proxy mariadb // commented vfc-nokia-vnfm-driver mariadb // commented vfc-nokia-v2vnfm-driver mariadb // commented vfc-nslcm vfc-db vfc-vnfmgr vfc-db vfc-resmgr mariadb // commented vfc-workflow mariadb // commented vfc-workflow-engine mariadb // commented vfc-vnflcm vfc-db vfc-vnfres vfc-db vfc-zte-sdnc-driver mariadb // commented vfc-zte-vnfm-driver mariadb // commented vid vid mariadb-galera vnfsdk postgres
Deployment Profiles
For use case specific deployments
# deployment profiles to cd.sh # each of the 28 components in alpha sequence --set pod.enabled=true|false status aaaccccd delmmnop pprssssu vvvv 11000100 10010101 01111011 0100 - vFW - no monitoring 11000100 10110101 11111011 0100 - vFW - with monitoring 11110111 10110111 11111111 0100 - vFW - cl
# | Team | container | Required y/n/fw/fwCl | Ram | Cpu | nodeport | Type | logback | Dependencies | |
---|---|---|---|---|---|---|---|---|---|---|
aaf | ||||||||||
aai | y | |||||||||
appc | fwCL | |||||||||
clamp | fwCL | |||||||||
cli | ||||||||||
consul | ||||||||||
fwCL | ||||||||||
dmaap | y | |||||||||
esr | ||||||||||
log | n | |||||||||
sniro- emulator | n | |||||||||
oof | n | |||||||||
msb | n | |||||||||
multicloud | n | |||||||||
nbi | ||||||||||
policy | y | |||||||||
pomba | ||||||||||
portal | y | |||||||||
robot | y | |||||||||
sdc | y | |||||||||
sdnc | y | |||||||||
so | y | |||||||||
uui | n | |||||||||
vfc | n | |||||||||
vid | y | |||||||||
vnfsdk | n |
Pod | Container/service | Tech | URLs | Filebeat Streaming | logback.xml | library | dependencies | Log level adjust | |
---|---|---|---|---|---|---|---|---|---|
aaf | aaf-cm aaf-fs aaf-gui aaf-hello aaf-locate aaf-oauth aaf-service aaf-sms-vault-0 | ||||||||
aai | aai-babel aai-champ aai-data-router aai-gizmo aai-graphadmin aai-modelloader aai-resources aai-search-data aai-sparky-be aai-spike aai-traversal | ||||||||
appc | appc-0 | ||||||||
clamp | - | ||||||||
cli | - | ||||||||
consul | - | ||||||||
contrib-netbox | - | ||||||||
dcaegen2 | - | ||||||||
dmaap | - | ||||||||
esr | - | ||||||||
log | - | ||||||||
logdemonode | - | ||||||||
msb | - | ||||||||
multicloud | - | ||||||||
nbi | - | ||||||||
oof | - | ||||||||
policy | - | ||||||||
pomba | - | ||||||||
portal | - | ||||||||
robot | - | ||||||||
sdc | - | ||||||||
sdnc | - | ||||||||
sniro | - | ||||||||
so | - | ||||||||
uui | - | ||||||||
vfc | - | ||||||||
vid | - | ||||||||
vnfsdk | - | ||||||||
vvp | - |
Deployment Profile
Type | VMs | Total RAM vCores HD | VM Flavor | K8S/Rancher Idle RAM | Deployed | Deployed ONAP RAM | Pods | Max vCores | Idle vCores | HD/VM | HD NFS only | IOPS | Date | Notes deployment post 75min |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Full Cluster (14 + 1) - recommended | 15 | 224G 112 vC 100G/VM | 16G, 8 vCores C5.2xLarge | 187Gb | 102Gb | 248 total 241 onap 217 up 0 error 24 config | 18 | 6+G master | 8.1G | 20181106 | ||||
Single VM (possible - not recommended) | 1 | 432G 64 vC 120G | 256G+ 32+ vCores | Rancher: 13G Kubernetes: 8G Top: 10G | 151Gb | 141Gb | 240 total | 55 | 22 | 108 of 128G | n/a | Max: 550/sec Idle: 220/sec | 20181105 | Tested on 432G/64vCore azure VM - R 1.6.22 K8S 1.11 |
Developer 1-n pods | 1 | 16G | 16/32G 4-16 vCores | 14Gb | 10Gb | 120+G | n/a | AAI+robot only |
Example 15 node (1 master + 14 nodes) OOM Deployment
empty
With ONAP deployed
Throughput and Volumetrics
Cloudwatch CPU Average
Specific to logging - we have a problem on any VM that contains AAI - the logstash container is being saturated there - see the 30+ percent VM - - LOG-376Getting issue details... STATUS
NFS Throughput for /dockerdata-nfs
Cloudwatch Network In Max
Details: 20181106:1800 EDT master
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | wc -l 248 ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | wc -l 241 ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | grep -E '1/1|2/2' | wc -l 217 ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | grep -E '0/|1/2' | wc -l 24 ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces -o wide | grep onap | grep -E '0/|1/2' onap onap-aaf-aaf-sms-preload-lvqx9 0/1 Completed 0 4h 10.42.75.71 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-aaf-aaf-sshsm-distcenter-ql5f8 0/1 Completed 0 4h 10.42.75.223 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-aaf-aaf-sshsm-testca-7rzcd 0/1 Completed 0 4h 10.42.18.37 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-aai-aai-graphadmin-create-db-schema-26pfs 0/1 Completed 0 4h 10.42.14.14 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-aai-aai-traversal-update-query-data-qlk7w 0/1 Completed 0 4h 10.42.88.122 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-gmmvj 0/1 Completed 0 4h 10.42.111.99 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-n6fw4 0/1 Error 0 4h 10.42.21.12 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-nc8ww 0/1 Error 0 4h 10.42.109.156 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-xcxds 0/1 Error 0 4h 10.42.152.223 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-dr-node-6496d8f55b-jfvrm 0/1 Init:0/1 28 4h 10.42.95.32 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-dr-prov-86f79c47f9-tldsp 0/1 CrashLoopBackOff 59 4h 10.42.76.248 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-oof-music-cassandra-job-config-7mb5f 0/1 Completed 0 4h 10.42.38.249 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-oof-oof-has-healthcheck-rpst7 0/1 Completed 0 4h 10.42.241.223 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-oof-oof-has-onboard-5bd2l 0/1 Completed 0 4h 10.42.205.75 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-portal-portal-db-config-qshzn 0/2 Completed 0 4h 10.42.112.46 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-portal-portal-db-config-rk4m2 0/2 Init:Error 0 4h 10.42.57.79 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-sdc-sdc-be-config-backend-2vw2q 0/1 Completed 0 4h 10.42.87.181 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-sdc-sdc-be-config-backend-k57lh 0/1 Init:Error 0 4h 10.42.148.79 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-sdc-sdc-cs-config-cassandra-vgnz2 0/1 Completed 0 4h 10.42.111.187 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-sdc-sdc-es-config-elasticsearch-lkb9m 0/1 Completed 0 4h 10.42.20.202 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-sdc-sdc-onboarding-be-cassandra-init-7zv5j 0/1 Completed 0 4h 10.42.218.1 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-sdc-sdc-wfd-be-workflow-init-q8t7z 0/1 Completed 0 4h 10.42.255.91 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-vid-vid-galera-config-4f274 0/1 Completed 0 4h 10.42.80.200 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-init-postgres-lf659 0/1 Completed 0 4h 10.42.238.204 ip-172-31-38-194.us-east-2.compute.internal <none> ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE kube-system heapster-7b48b696fc-kxjlx 1/1 Running 0 5h 10.42.237.56 ip-172-31-37-24.us-east-2.compute.internal <none> kube-system kube-dns-6655f78c68-wzmvh 3/3 Running 0 5h 10.42.18.171 ip-172-31-37-24.us-east-2.compute.internal <none> kube-system kubernetes-dashboard-6f54f7c4b-58k4h 1/1 Running 0 5h 10.42.205.56 ip-172-31-37-24.us-east-2.compute.internal <none> kube-system monitoring-grafana-7877679464-9ssz5 1/1 Running 0 5h 10.42.252.38 ip-172-31-37-24.us-east-2.compute.internal <none> kube-system monitoring-influxdb-64664c6cf5-h4bxt 1/1 Running 0 5h 10.42.226.121 ip-172-31-37-24.us-east-2.compute.internal <none> kube-system tiller-deploy-6f4745cbcf-q6kxx 1/1 Running 0 5h 10.42.12.224 ip-172-31-37-24.us-east-2.compute.internal <none> onap dep-config-binding-service-68d9cd6f4c-9n2nf 2/2 Running 0 4h 10.42.157.143 ip-172-31-41-30.us-east-2.compute.internal <none> onap dep-dcae-datafile-collector-77956c65df-8mfwc 2/2 Running 0 4h 10.42.110.106 ip-172-31-41-30.us-east-2.compute.internal <none> onap dep-dcae-hv-ves-collector-55844fb468-9zdss 2/2 Running 0 4h 10.42.231.39 ip-172-31-45-152.us-east-2.compute.internal <none> onap dep-dcae-prh-74977b5fb4-n47b6 2/2 Running 0 4h 10.42.115.19 ip-172-31-39-138.us-east-2.compute.internal <none> onap dep-dcae-snmptrap-collector-69f99dc589-f7ffl 1/1 Running 0 4h 10.42.173.221 ip-172-31-41-229.us-east-2.compute.internal <none> onap dep-dcae-tca-analytics-84fbfbc894-xzq8c 2/2 Running 0 4h 10.42.28.146 ip-172-31-38-194.us-east-2.compute.internal <none> onap dep-dcae-ves-collector-98f8ccb5c-tghs4 2/2 Running 0 4h 10.42.207.99 ip-172-31-34-111.us-east-2.compute.internal <none> onap dep-deployment-handler-76cf6d665-tvcwp 2/2 Running 0 4h 10.42.11.59 ip-172-31-37-59.us-east-2.compute.internal <none> onap dep-holmes-engine-mgmt-9889496c5-2jbnb 1/1 Running 27 4h 10.42.186.10 ip-172-31-36-163.us-east-2.compute.internal <none> onap dep-holmes-rule-mgmt-9b49769b5-n6hqj 1/1 Running 27 4h 10.42.185.154 ip-172-31-39-138.us-east-2.compute.internal <none> onap dep-inventory-5c9f5bcc6d-lzqvh 1/1 Running 0 4h 10.42.68.75 ip-172-31-37-59.us-east-2.compute.internal <none> onap dep-policy-handler-544dcdfd75-vpsm7 2/2 Running 0 4h 10.42.233.2 ip-172-31-38-194.us-east-2.compute.internal <none> onap dep-pstg-write-f67b57947-mkbpq 1/1 Running 0 4h 10.42.185.41 ip-172-31-39-138.us-east-2.compute.internal <none> onap dep-service-change-handler-66b78745c-995ks 1/1 Running 3 4h 10.42.92.139 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-aaf-aaf-cm-7fcc8548cc-nrmsf 1/1 Running 0 4h 10.42.226.234 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-aaf-aaf-cs-5755ccf8f4-tz6fn 1/1 Running 0 4h 10.42.181.148 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-aaf-aaf-fs-7f8f9dcd74-27gfd 1/1 Running 0 4h 10.42.118.145 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-aaf-aaf-gui-bfd9c7fd6-qk6zw 1/1 Running 0 4h 10.42.201.9 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-aaf-aaf-hello-6447444665-5t24n 1/1 Running 0 4h 10.42.150.107 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-aaf-aaf-locate-845c5867db-2q8qv 1/1 Running 0 4h 10.42.190.184 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-aaf-aaf-oauth-68bc85dbd4-7qv7w 1/1 Running 0 4h 10.42.233.42 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-aaf-aaf-service-69fb68d68-qhgqc 1/1 Running 0 4h 10.42.108.135 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-7497c9bfcc-dtc8s 1/1 Running 0 4h 10.42.85.30 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-preload-lvqx9 0/1 Completed 0 4h 10.42.75.71 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-quorumclient-0 1/1 Running 0 4h 10.42.30.16 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-quorumclient-1 1/1 Running 0 4h 10.42.164.247 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-quorumclient-2 1/1 Running 0 4h 10.42.232.51 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-vault-0 2/2 Running 1 4h 10.42.242.253 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-aaf-aaf-sshsm-distcenter-ql5f8 0/1 Completed 0 4h 10.42.75.223 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-aaf-aaf-sshsm-testca-7rzcd 0/1 Completed 0 4h 10.42.18.37 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-aai-aai-695546c99-l4vd4 1/1 Running 0 4h 10.42.61.157 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-aai-aai-babel-c489c944c-l6fxm 2/2 Running 0 4h 10.42.151.176 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-aai-aai-cassandra-0 1/1 Running 0 4h 10.42.60.200 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-aai-aai-cassandra-1 1/1 Running 1 4h 10.42.102.208 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-aai-aai-cassandra-2 1/1 Running 0 4h 10.42.168.115 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-aai-aai-champ-54f7986b6b-6qqk8 2/2 Running 0 4h 10.42.233.99 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-aai-aai-data-router-5b777d798c-hkt9p 2/2 Running 0 4h 10.42.254.152 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-aai-aai-elasticsearch-c9bf9dbf6-flljw 1/1 Running 0 4h 10.42.136.54 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-aai-aai-gizmo-5f8bf54f6f-4sj45 2/2 Running 0 4h 10.42.128.212 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-aai-aai-graphadmin-789c67bb8b-kd9kw 2/2 Running 0 4h 10.42.196.56 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-aai-aai-graphadmin-create-db-schema-26pfs 0/1 Completed 0 4h 10.42.14.14 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-aai-aai-modelloader-695fd4bf48-5gdmt 2/2 Running 0 4h 10.42.188.20 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-aai-aai-resources-765d6f6678-xg4jv 2/2 Running 0 4h 10.42.224.167 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-aai-aai-search-data-559b8dbc7f-k549g 2/2 Running 0 4h 10.42.110.212 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-aai-aai-sparky-be-75658695f5-wjkzw 2/2 Running 0 4h 10.42.7.139 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-aai-aai-spike-f854cf86d-fdx7j 2/2 Running 0 4h 10.42.175.92 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-aai-aai-traversal-5fcc7cc847-mrqtb 2/2 Running 0 4h 10.42.174.143 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-aai-aai-traversal-update-query-data-qlk7w 0/1 Completed 0 4h 10.42.88.122 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-appc-appc-0 2/2 Running 0 4h 10.42.106.203 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-appc-appc-ansible-server-78784978b6-jrz2b 1/1 Running 0 4h 10.42.232.15 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-appc-appc-cdt-d654974f6-bxhkw 1/1 Running 0 4h 10.42.170.154 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-appc-appc-db-0 1/1 Running 0 4h 10.42.215.183 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-appc-appc-db-1 1/1 Running 1 4h 10.42.113.105 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-appc-appc-db-2 1/1 Running 1 4h 10.42.245.153 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-appc-appc-dgbuilder-6bdc7bd56b-ssb2p 1/1 Running 0 4h 10.42.188.237 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-clamp-clamp-8cd95c6b4-kbmxg 2/2 Running 0 4h 10.42.149.44 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-clamp-clamp-dash-es-5566796d7c-b2mbd 1/1 Running 0 4h 10.42.51.26 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-clamp-clamp-dash-kibana-646fccf75d-vf68k 1/1 Running 0 4h 10.42.173.69 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-clamp-clamp-dash-logstash-7bfc956564-xhlj5 1/1 Running 0 4h 10.42.38.41 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-clamp-clampdb-6797447d-bkrbr 1/1 Running 0 4h 10.42.37.63 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-cli-cli-5498964cf9-ndk74 1/1 Running 0 4h 10.42.251.86 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-consul-consul-849447d678-8677q 1/1 Running 0 4h 10.42.56.217 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-consul-consul-server-0 1/1 Running 0 4h 10.42.69.75 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-consul-consul-server-1 1/1 Running 0 4h 10.42.156.89 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-consul-consul-server-2 1/1 Running 0 4h 10.42.69.41 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-6d4d768d9d-gksvt 1/1 Running 0 4h 10.42.179.105 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-gmmvj 0/1 Completed 0 4h 10.42.111.99 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-n6fw4 0/1 Error 0 4h 10.42.21.12 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-nc8ww 0/1 Error 0 4h 10.42.109.156 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-xcxds 0/1 Error 0 4h 10.42.152.223 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-contrib-netbox-nginx-54c979ff7f-nk5kk 1/1 Running 0 4h 10.42.244.179 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-contrib-netbox-postgres-784b5d5fd9-2wdf7 1/1 Running 0 4h 10.42.175.55 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-bootstrap-7bb4b6bc57-gvz47 1/1 Running 0 4h 10.42.74.208 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-cloudify-manager-9cb7f4c66-7h77h 1/1 Running 0 4h 10.42.250.189 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-db-0 1/1 Running 0 4h 10.42.230.118 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-db-1 1/1 Running 0 4h 10.42.29.230 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-healthcheck-7b664b566c-w5w4h 1/1 Running 0 4h 10.42.9.151 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-pgpool-688d65b66d-jpv8x 1/1 Running 0 4h 10.42.177.56 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-pgpool-688d65b66d-srm5q 1/1 Running 0 4h 10.42.164.252 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-0 1/1 Running 0 4h 10.42.32.17 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-1 1/1 Running 0 4h 10.42.52.99 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-2 1/1 Running 0 4h 10.42.45.144 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-3 1/1 Running 0 4h 10.42.79.151 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-4 1/1 Running 0 4h 10.42.110.141 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-5 1/1 Running 0 4h 10.42.253.134 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-dmaap-dbc-pg-0 1/1 Running 0 4h 10.42.166.51 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-dmaap-dbc-pg-1 1/1 Running 0 4h 10.42.230.243 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-dmaap-dbc-pgpool-c5f8498-lr854 1/1 Running 0 4h 10.42.35.8 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-dmaap-dbc-pgpool-c5f8498-m84f5 1/1 Running 0 4h 10.42.240.60 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-bus-controller-59c96d6b8f-nlc4t 1/1 Running 0 4h 10.42.240.232 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-dr-db-557c66dc9d-4ltgp 1/1 Running 1 4h 10.42.210.116 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-dr-node-6496d8f55b-jfvrm 0/1 Init:0/1 28 4h 10.42.95.32 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-dr-prov-86f79c47f9-tldsp 0/1 CrashLoopBackOff 59 4h 10.42.76.248 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-dmaap-message-router-557cb765-sc4pb 1/1 Running 0 4h 10.42.157.140 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-dmaap-message-router-kafka-7b4d45b44c-7g5pz 1/1 Running 0 4h 10.42.218.166 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-dmaap-message-router-zookeeper-5bc7588665-pz9x5 1/1 Running 0 4h 10.42.129.130 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-esr-esr-gui-6c5ccd59d6-xzh46 1/1 Running 0 4h 10.42.89.47 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-esr-esr-server-5f967d4767-nl5xc 2/2 Running 0 4h 10.42.146.185 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-log-log-elasticsearch-7c5b74569f-g64hw 1/1 Running 0 4h 10.42.54.225 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-log-log-kibana-84b875cc6b-gdzp8 1/1 Running 0 4h 10.42.56.186 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-log-log-logstash-7hmrz 1/1 Running 0 4h 10.42.200.38 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-log-log-logstash-8cb9c 1/1 Running 0 4h 10.42.162.152 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-log-log-logstash-8pqrp 1/1 Running 0 4h 10.42.97.239 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-log-log-logstash-bvq5r 1/1 Running 0 4h 10.42.241.246 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-log-log-logstash-cckqk 1/1 Running 0 4h 10.42.222.107 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-log-log-logstash-gq826 1/1 Running 0 4h 10.42.129.134 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-log-log-logstash-kzvbv 1/1 Running 0 4h 10.42.195.222 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-log-log-logstash-mdbtv 1/1 Running 0 4h 10.42.54.127 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-log-log-logstash-n78q5 1/1 Running 0 4h 10.42.22.101 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-log-log-logstash-ns76c 1/1 Running 0 4h 10.42.124.239 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-log-log-logstash-pvsnw 1/1 Running 0 4h 10.42.164.130 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-log-log-logstash-q86z9 1/1 Running 0 4h 10.42.73.74 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-log-log-logstash-skr52 1/1 Running 0 4h 10.42.0.150 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-log-log-logstash-zjkph 1/1 Running 0 4h 10.42.47.5 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-msb-kube2msb-5c79ddd89f-4m524 1/1 Running 0 4h 10.42.86.56 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-msb-msb-consul-6949bd46f4-gsm59 1/1 Running 0 4h 10.42.19.136 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-msb-msb-discovery-86c7b945f9-ffxtx 2/2 Running 0 4h 10.42.27.240 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-msb-msb-eag-5f86f89c4f-4pr2t 2/2 Running 0 4h 10.42.36.35 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-msb-msb-iag-56cdd4c87b-4xtz7 2/2 Running 0 4h 10.42.152.29 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-867df449f8-gwkdw 2/2 Running 0 4h 10.42.87.4 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-azure-5db4ddd44d-cpphs 2/2 Running 0 4h 10.42.84.246 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-ocata-c557948df-5sgfs 2/2 Running 0 4h 10.42.39.16 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-pike-8ff99dfb6-zmrw8 2/2 Running 0 4h 10.42.176.68 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-vio-866cd4bf9b-g2zlx 2/2 Running 0 4h 10.42.64.228 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-windriver-7775ddc4b9-4278w 2/2 Running 0 4h 10.42.87.19 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-nbi-nbi-7595d4dbc-9m95t 1/1 Running 1 4h 10.42.94.19 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-nbi-nbi-mariadb-65f556b8b7-fgjzx 1/1 Running 0 4h 10.42.195.127 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-nbi-nbi-mongo-0 1/1 Running 0 4h 10.42.184.20 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-oof-cmso-db-0 1/1 Running 0 4h 10.42.193.70 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-oof-music-cassandra-0 1/1 Running 1 4h 10.42.189.19 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-oof-music-cassandra-1 1/1 Running 0 4h 10.42.163.171 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-oof-music-cassandra-2 1/1 Running 0 4h 10.42.195.0 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-oof-music-cassandra-job-config-7mb5f 0/1 Completed 0 4h 10.42.38.249 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-oof-music-tomcat-7974f8748c-4pgg8 1/1 Running 0 4h 10.42.118.22 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-oof-music-tomcat-7974f8748c-dxdds 1/1 Running 0 4h 10.42.159.58 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-oof-music-tomcat-7974f8748c-jh7jg 1/1 Running 0 4h 10.42.64.7 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-oof-oof-6dbf897c86-8d6bx 1/1 Running 0 4h 10.42.20.170 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-oof-oof-cmso-service-7d54b6b478-djvdm 1/1 Running 0 4h 10.42.192.55 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-oof-oof-has-api-774f9458cb-mtfg8 1/1 Running 0 4h 10.42.208.73 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-oof-oof-has-controller-7998c7f7b6-vg6nz 1/1 Running 0 4h 10.42.91.250 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-oof-oof-has-data-5dffb4c7df-qmjzz 1/1 Running 0 4h 10.42.213.36 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-oof-oof-has-healthcheck-rpst7 0/1 Completed 0 4h 10.42.241.223 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-oof-oof-has-onboard-5bd2l 0/1 Completed 0 4h 10.42.205.75 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-oof-oof-has-reservation-59759db679-dz4cs 1/1 Running 0 4h 10.42.48.244 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-oof-oof-has-solver-5bfc76f46-hfc88 1/1 Running 0 4h 10.42.225.129 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-oof-zookeeper-0 1/1 Running 0 4h 10.42.171.61 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-oof-zookeeper-1 1/1 Running 0 4h 10.42.251.101 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-oof-zookeeper-2 1/1 Running 0 4h 10.42.65.144 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-policy-brmsgw-86dd8fc4f7-xtzk9 1/1 Running 0 4h 10.42.56.63 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-policy-drools-0 1/1 Running 0 4h 10.42.75.29 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-policy-nexus-b7f484bdb-gmn76 1/1 Running 0 4h 10.42.45.249 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-policy-pap-6555659676-77nl7 2/2 Running 0 4h 10.42.87.216 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-policy-pdp-0 2/2 Running 0 4h 10.42.63.230 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-policy-policy-apex-pdp-0 1/1 Running 0 4h 10.42.73.87 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-policy-policy-distribution-79544b78c7-hk5mk 1/1 Running 0 4h 10.42.104.145 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-policy-policydb-79cdf7cb7c-jb82c 1/1 Running 0 4h 10.42.98.252 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-pomba-pomba-aaictxbuilder-684d5977cd-sj8dm 2/2 Running 0 4h 10.42.170.23 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-pomba-pomba-contextaggregator-779745878d-4s8c9 1/1 Running 0 4h 10.42.154.6 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-pomba-pomba-data-router-59bb4fd899-dc5zc 1/1 Running 0 4h 10.42.171.88 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-pomba-pomba-elasticsearch-7b8bc5f864-5mjqk 1/1 Running 0 4h 10.42.111.181 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-pomba-pomba-kibana-6dd8dbb68b-xltcm 1/1 Running 0 4h 10.42.181.201 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-pomba-pomba-networkdiscovery-d47f68c88-j2zw4 2/2 Running 0 4h 10.42.34.101 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-pomba-pomba-networkdiscoveryctxbuilder-6978dc6c79-6z7zm 2/2 Running 0 4h 10.42.116.177 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-pomba-pomba-sdcctxbuilder-fcbc7f96b-nj2c9 1/1 Running 0 4h 10.42.87.59 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-pomba-pomba-search-data-5b4d8f7dc6-k7qlx 2/2 Running 0 4h 10.42.37.226 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-pomba-pomba-servicedecomposition-99b445978-6crkz 2/2 Running 0 4h 10.42.46.61 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-pomba-pomba-validation-service-5c8454484-t4kls 1/1 Running 0 4h 10.42.235.206 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-portal-portal-app-67bc49d88-tpxbq 2/2 Running 0 4h 10.42.25.53 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-portal-portal-cassandra-75d4746d76-jn78x 1/1 Running 0 4h 10.42.128.240 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-portal-portal-db-7b768ffcc-92q8b 1/1 Running 1 4h 10.42.187.56 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-portal-portal-db-config-qshzn 0/2 Completed 0 4h 10.42.112.46 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-portal-portal-db-config-rk4m2 0/2 Init:Error 0 4h 10.42.57.79 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-portal-portal-sdk-59fbd4bd9d-6j9x9 2/2 Running 0 4h 10.42.217.29 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-portal-portal-widget-79587fdf56-5dh2c 1/1 Running 0 4h 10.42.12.249 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-portal-portal-zookeeper-7c9444bfb5-9bvwq 1/1 Running 0 4h 10.42.113.205 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-robot-robot-86c5f986b7-kd257 1/1 Running 0 4h 10.42.246.138 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-sdc-sdc-be-7fdc576bfb-6q5bx 2/2 Running 0 4h 10.42.142.158 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-sdc-sdc-be-config-backend-2vw2q 0/1 Completed 0 4h 10.42.87.181 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-sdc-sdc-be-config-backend-k57lh 0/1 Init:Error 0 4h 10.42.148.79 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-sdc-sdc-cs-5b9bb5756d-28zhd 1/1 Running 1 4h 10.42.232.24 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-sdc-sdc-cs-config-cassandra-vgnz2 0/1 Completed 0 4h 10.42.111.187 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-sdc-sdc-es-8bdf86659-bvgj6 1/1 Running 0 4h 10.42.21.223 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-sdc-sdc-es-config-elasticsearch-lkb9m 0/1 Completed 0 4h 10.42.20.202 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-sdc-sdc-fe-65c95cf8bf-2944f 2/2 Running 0 4h 10.42.193.193 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-sdc-sdc-kb-79bfffc4cf-6p5qn 1/1 Running 0 4h 10.42.59.138 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-sdc-sdc-onboarding-be-84466885d5-pwqpn 2/2 Running 0 4h 10.42.163.142 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-sdc-sdc-onboarding-be-cassandra-init-7zv5j 0/1 Completed 0 4h 10.42.218.1 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-sdc-sdc-wfd-be-54f49cb9f-8wmkz 1/1 Running 0 4h 10.42.69.253 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-sdc-sdc-wfd-be-workflow-init-q8t7z 0/1 Completed 0 4h 10.42.255.91 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-sdc-sdc-wfd-fe-585bd84f8d-grwwh 2/2 Running 0 4h 10.42.110.244 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-sdnc-controller-blueprints-7c8f9dbfb7-tq4f5 1/1 Running 0 4h 10.42.200.245 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-sdnc-controller-blueprints-db-0 1/1 Running 0 4h 10.42.207.149 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-sdnc-nengdb-0 1/1 Running 0 4h 10.42.211.255 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-sdnc-network-name-gen-f4ff8b99f-8bxbh 1/1 Running 0 4h 10.42.107.56 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-0 2/2 Running 0 4h 10.42.220.124 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-ansible-server-dffcf4c8b-jdpmh 1/1 Running 0 4h 10.42.84.202 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-db-0 2/2 Running 0 4h 10.42.45.121 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-dgbuilder-5496ff5f86-5l27s 1/1 Running 0 4h 10.42.17.254 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-dmaap-listener-54f5c69c7c-bxp2n 1/1 Running 0 4h 10.42.42.177 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-portal-8ff4bb696-j9xz5 1/1 Running 0 4h 10.42.250.3 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-ueb-listener-5d5749f879-2pw5j 1/1 Running 0 4h 10.42.93.48 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-sniro-emulator-sniro-emulator-5bdc8cc4cb-qjbk8 1/1 Running 0 4h 10.42.207.251 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-so-so-7f5f98949b-w85kc 1/1 Running 0 4h 10.42.110.129 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-so-so-bpmn-infra-74fcf888b4-9tnfc 1/1 Running 0 4h 10.42.60.90 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-so-so-catalog-db-adapter-5b8cb47dc-j44bt 1/1 Running 0 4h 10.42.222.142 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-so-so-mariadb-5b5bd695c4-jdx8v 1/1 Running 0 4h 10.42.249.209 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-so-so-monitoring-855456948b-fr24q 1/1 Running 0 4h 10.42.14.54 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-so-so-openstack-adapter-c44cd98dc-pk2gb 1/1 Running 1 4h 10.42.25.240 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-so-so-request-db-adapter-745dc6677-z7tcx 1/1 Running 0 4h 10.42.186.49 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-so-so-sdc-controller-67f8cbbc7c-kdfmr 1/1 Running 0 4h 10.42.23.215 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-so-so-sdnc-adapter-5cbc5f8dd8-6k22h 1/1 Running 0 4h 10.42.14.50 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-so-so-vfc-adapter-6c5996d748-5h2jj 1/1 Running 0 4h 10.42.205.174 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-uui-uui-d5ff69677-xkmq2 1/1 Running 0 4h 10.42.157.9 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-uui-uui-server-7cc8bcb956-c28g5 1/1 Running 0 4h 10.42.245.99 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-vfc-vfc-catalog-66dd84cc97-htgsr 2/2 Running 0 4h 10.42.248.74 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-vfc-vfc-db-65d8fd5567-25z45 1/1 Running 0 4h 10.42.181.50 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-vfc-vfc-ems-driver-6f5fd887db-hrh7q 1/1 Running 0 4h 10.42.28.44 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-vfc-vfc-generic-vnfm-driver-7964b7f4cc-nd2d2 2/2 Running 1 4h 10.42.172.200 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-vfc-vfc-huawei-vnfm-driver-6c5676758d-vs8fh 2/2 Running 0 4h 10.42.58.24 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-vfc-vfc-juju-vnfm-driver-67b9c4bb4c-fhv45 2/2 Running 0 4h 10.42.10.58 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-vfc-vfc-multivim-proxy-5f6f455bfd-tg7m2 1/1 Running 0 4h 10.42.51.118 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-vfc-vfc-nokia-v2vnfm-driver-6f7646f69d-22p88 1/1 Running 0 4h 10.42.158.31 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-vfc-vfc-nokia-vnfm-driver-7c9d59597f-gkmct 2/2 Running 0 4h 10.42.184.15 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-vfc-vfc-nslcm-c8478dcc8-cnlvd 2/2 Running 0 4h 10.42.144.183 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-vfc-vfc-resmgr-6d7d7bc78b-f5v88 2/2 Running 0 4h 10.42.255.238 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-vfc-vfc-vnflcm-fb656d967-p6wh2 2/2 Running 0 4h 10.42.56.73 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-vfc-vfc-vnfmgr-66d5cd85f7-ld4v4 2/2 Running 0 4h 10.42.121.241 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-vfc-vfc-vnfres-7757d4b486-bkqqz 2/2 Running 0 4h 10.42.100.120 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-vfc-vfc-workflow-68f85b8fcf-mxgwl 1/1 Running 0 4h 10.42.72.65 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-vfc-vfc-workflow-engine-7d76bb4b56-tj6ks 1/1 Running 0 4h 10.42.48.13 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-vfc-vfc-zte-sdnc-driver-58d4fc6d9f-hk4d9 1/1 Running 0 4h 10.42.245.72 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-vfc-vfc-zte-vnfm-driver-fc669cc8d-64k6j 2/2 Running 1 4h 10.42.68.226 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-vid-vid-5cb464799b-fw69j 2/2 Running 0 4h 10.42.21.193 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-vid-vid-galera-config-4f274 0/1 Completed 0 4h 10.42.80.200 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-vid-vid-mariadb-galera-0 1/1 Running 0 4h 10.42.40.225 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-6b45bfffb4-phcwx 1/1 Running 0 4h 10.42.32.180 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-init-postgres-lf659 0/1 Completed 0 4h 10.42.238.204 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-pgpool-6874f7b695-69njk 1/1 Running 0 4h 10.42.239.14 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-pgpool-6874f7b695-pjmwp 1/1 Running 0 4h 10.42.171.177 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-postgres-0 1/1 Running 0 4h 10.42.106.183 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-postgres-1 1/1 Running 0 4h 10.42.47.66 ip-172-31-41-229.us-east-2.compute.internal <none> ubuntu@ip-172-31-40-250:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-172-31-33-223.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.222.148.116 18.222.148.116 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-34-111.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 3.16.37.170 3.16.37.170 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-34-207.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.225.32.201 18.225.32.201 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-36-163.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 13.58.189.251 13.58.189.251 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-37-24.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.224.180.26 18.224.180.26 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-37-59.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.191.248.14 18.191.248.14 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-38-194.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.217.45.91 18.217.45.91 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-38-95.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 52.15.39.21 52.15.39.21 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-39-138.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.224.199.40 18.224.199.40 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-41-110.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.223.151.180 18.223.151.180 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-41-229.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.218.252.13 18.218.252.13 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-41-30.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 3.16.113.3 3.16.113.3 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-42-33.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 13.59.2.86 13.59.2.86 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-45-152.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.219.56.50 18.219.56.50 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ubuntu@ip-172-31-40-250:~$ kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% ip-172-31-33-223.us-east-2.compute.internal 852m 10% 13923Mi 90% ip-172-31-34-111.us-east-2.compute.internal 1160m 14% 11643Mi 75% ip-172-31-34-207.us-east-2.compute.internal 1101m 13% 7981Mi 51% ip-172-31-36-163.us-east-2.compute.internal 656m 8% 13377Mi 87% ip-172-31-37-24.us-east-2.compute.internal 401m 5% 8543Mi 55% ip-172-31-37-59.us-east-2.compute.internal 711m 8% 10873Mi 70% ip-172-31-38-194.us-east-2.compute.internal 1136m 14% 8195Mi 53% ip-172-31-38-95.us-east-2.compute.internal 1195m 14% 9127Mi 59% ip-172-31-39-138.us-east-2.compute.internal 296m 3% 10870Mi 70% ip-172-31-41-110.us-east-2.compute.internal 2586m 32% 10950Mi 71% ip-172-31-41-229.us-east-2.compute.internal 159m 1% 9138Mi 59% ip-172-31-41-30.us-east-2.compute.internal 180m 2% 9862Mi 64% ip-172-31-42-33.us-east-2.compute.internal 1573m 19% 6352Mi 41% ip-172-31-45-152.us-east-2.compute.internal 1579m 19% 10633Mi 69% ubuntu@ip-172-31-40-250:~/oom/kubernetes/robot$ sudo ./ete-k8s.sh onap health ./ete-k8s.sh: 22: [: onap: unexpected operator ./ete-k8s.sh: 22: [: health: unexpected operator + export NAMESPACE=onap Testsuites ============================================================================== Testsuites.Health-Check :: Testing ecomp components are available via calls. ============================================================================== Basic A&AI Health Check | PASS | ------------------------------------------------------------------------------ Basic AAF Health Check | PASS | ------------------------------------------------------------------------------ Basic AAF SMS Health Check | PASS | ------------------------------------------------------------------------------ Basic APPC Health Check | PASS | ------------------------------------------------------------------------------ Basic CLI Health Check | PASS | ------------------------------------------------------------------------------ Basic CLAMP Health Check | PASS | ------------------------------------------------------------------------------ Basic DCAE Health Check | PASS | ------------------------------------------------------------------------------ Basic DMAAP Data Router Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f536960ea50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f536972e450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5369617090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv | FAIL | ConnectionError: HTTPConnectionPool(host='dmaap-dr-node.onap', port=8080): Max retries exceeded with url: /internal/fetchProv (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5369d21850>: Failed to establish a new connection: [Errno 111] Connection refused',)) ------------------------------------------------------------------------------ Basic DMAAP Message Router Health Check | PASS | ------------------------------------------------------------------------------ Basic External API NBI Health Check | PASS | ------------------------------------------------------------------------------ Basic Log Elasticsearch Health Check | PASS | ------------------------------------------------------------------------------ Basic Log Kibana Health Check | PASS | ------------------------------------------------------------------------------ Basic Log Logstash Health Check | PASS | ------------------------------------------------------------------------------ Basic Microservice Bus Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-ocata API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-pike API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-titanium_cloud API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-vio API Health Check | PASS | ------------------------------------------------------------------------------ Basic OOF-Homing Health Check | PASS | ------------------------------------------------------------------------------ Basic OOF-SNIRO Health Check | PASS | ------------------------------------------------------------------------------ Basic OOF-CMSO Health Check | PASS | ------------------------------------------------------------------------------ Basic Policy Health Check | PASS | ------------------------------------------------------------------------------ Basic Pomba AAI-context-builder Health Check | PASS | ------------------------------------------------------------------------------ Basic Pomba SDC-context-builder Health Check | PASS | ------------------------------------------------------------------------------ Basic Pomba Network-discovery-context-builder Health Check | PASS | ------------------------------------------------------------------------------ Basic Portal Health Check | PASS | ------------------------------------------------------------------------------ Basic SDC Health Check (DMaaP:UP)| PASS | ------------------------------------------------------------------------------ Basic SDNC Health Check | PASS | ------------------------------------------------------------------------------ Basic SO Health Check | PASS | ------------------------------------------------------------------------------ Basic UseCaseUI API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC catalog API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC emsdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC gvnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC huaweivnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC jujuvnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC multivimproxy API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC nokiavnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC nokiav2driver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC nslcm API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC resmgr API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC vnflcm API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC vnfmgr API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC vnfres API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC workflow API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC ztesdncdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC ztevnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VID Health Check | PASS | ------------------------------------------------------------------------------ Basic VNFSDK Health Check | PASS | ------------------------------------------------------------------------------ Basic Holmes Rule Management API Health Check | FAIL | 502 != 200 ------------------------------------------------------------------------------ Basic Holmes Engine Management API Health Check | FAIL | 502 != 200 ------------------------------------------------------------------------------ Testsuites.Health-Check :: Testing ecomp components are available ... | FAIL | 51 critical tests, 48 passed, 3 failed 51 tests total, 48 passed, 3 failed
ONAP Module Dependencies - Compile and Runtime
Existing Library Research
Portal/SDK
SO
AAI
Library Selected for Reference Implementation
Java
Python
Specification
Developed in Casablanca - implemented in Dublin - ONAP Application Logging Specification v1.2 (Casablanca)