/
Log Streaming Compliance and API

Log Streaming Compliance and API

Jan 22, 2019

This page details the current state of log streaming in ONAP and plans to increase compliance towards the goal of providing end to end log tracing (ONAP transactions, onap system, undercloud).

 

ONAP Log Streaming Deployment View

 

Analysis

using

http://openlab13.onap.info:30253/app/kibana

Compliance

Streaming

https://12factor.net/logs

Format

Library

Fields

timestampadd t versionadd t ACTIONadd t API_KEYadd t ARTIFACT_DATAadd t AUTH_STATUSadd t AlertSeverityadd AuditBeginTimestampadd t AuditMessageadd BeginTimestampadd t CONSUMER_IDadd t CURR_ARTIFACT_UUIDadd t CURR_STATEadd t CURR_VERSIONadd t ClassNameadd t CustomField1add t CustomField2add t DESCadd t DNOTIF_TOPICadd t DSTATUS_TOPICadd t D_ENVadd t ECOMP_USERadd t ElapsedTimeadd EndTimestampadd t ErrorCategoryadd t ErrorCodeadd t INVARIANT_UUIDadd t InstanceUUIDadd t MODIFIERadd t PREV_ARTIFACT_UUIDadd t PREV_STATEadd t PREV_VERSIONadd t PartnerNameadd t REALMadd t RESOURCE_NAMEadd t RESOURCE_TYPEadd t RemoteHostadd t RequestIdadd t ResponseCodeadd t ResponseDescriptionadd t SERVICE_INSTANCE_IDadd t STATUSadd t ServerFQDNadd t ServerIPAddressadd t ServiceInstanceIdadd t ServiceNameadd t StatusCodeadd t TargetEntityadd t TargetServiceNameadd t Threadadd t Timeradd Timestampadd t URLadd t USERadd t USER_AFTERadd t USER_BEFOREadd t _idadd t _indexadd # _scoreadd t _typeadd t auditOnadd t beat.hostnameadd t beat.nameadd t beat.versionadd t componentLogFileadd t componentNameadd t hostadd t input_typeadd t logleveladd t messageadd # offsetadd t serviceInstanceIDadd t sourceadd t tagsadd t timeradd t typeadd

 

 

Existing ONAP Logging State - Infrastructure and Schema

Deployment Dependency Tree

Note: these dependencies - introduced using the readiness container by Mandeep Khinda are at the lowest deployment level and represent a partial view of the REST/API dependency tree - they do not reflect any compile time or runtime/injection code dependencies (pom.xml)

Gliffy Diagram

https://lf-onap.atlassian.net/browse/LOG-924

tested sdnc via  sudo ./cd.sh -b master -e onap -p false -a false -t robot -x 1 -z false -f false -s 600 -c false -d true -w false -r false

where dmaap, consul and sdc are raised first via dev0.yaml and sdnc is in dev1.yaml

Deployment Dependency Tree - Container level

Gliffy Diagram
Use for any containers stuck at the 0/1 Init:0/1 stage - these are likely waiting on dependent containers check the --container-name kv pair in StatefulSet/Deployment.yaml:spec:intiContainers:args or the corresponding defined chart/container names in values.yaml:config: 106 sets in 87 files overall order aaf<-aai aaf<-oof music<-oof dmaap<-aai dmaap<-pomba dmaap<-sdnc consul<-sdnc sdc<-sdnc consul<-dcaegen2 msb<-dcaegen2 order dmaap, aaf, msb, consul, dcaegen2, aai, oof, sdc, sdnc, pomba valid sets of pods aaf aai, dmaap, aaf dcaegen2, consul, msb oof, aaf, music pomba, dmaap sdnc, consul, sdc ..the rest can be singular aaf aaf-cm aaf-locate aaf-fs aaf-locate aaf-gui aaf-cm aaf-hello aaf-locate aaf-locate aaf-service aaf-oauth aaf-locate aaf-service aaf-cs aaf-sms aaf-sms-quorumclient (via job) aaf-sms-vault aaf-sms-vault-backend aai aai aai-resources aai-traversal aai-graphadmin aai-champ aai-cassandra aai-graphadmin aai-cassandra aai-resources aai-cassandra aai-sparky-be aai-elasticsearch aai-search-data aai aai-spike message-router-kafka aai-traversal aai aai-cassandra aaf-locate (conditional) appc appc mariadb-galera appc-ansible-server appc clamp clamp mariadb clamp-dash-kibana clamp-dash-es clamp-dash-logstash clamp-dash-es common controller-blueprints mariadb-galera mongo *-nfs-provisioner mysql *-nfs-provisioner dgbuilder network-name-gen mariadb-galera dcaegen2 dcae-bootstrap dcae-cloudify-manager consul-server msb-discovery kube2msb dep-health-init hbase dmaap dmaap-bus-controller postgres dmaap-dr-node dmaap-dr-prov dmaap-dr-prov mariadb message-router kafka zookeeper message-router-kafka zookeeper log log-kibana log-elasticsearch log-logstash log-elasticsearch msb kube2msb msb-discovery msb-discovery msb-consul msb-eag msb-discovery msb-iag msb-discovery music music-cassandra music-tomcat zookeeper oof oof-has-api oof-has-controller aaf-service oof-has-controller music-tomcat aaf-sms oof-has-data music-tomcat oof-has-reservation music-tomcat oof-has-service music-tomcat policy policy mariadb brmsgw pap drools mariadb nexus pdb pap pomba pomba-contextaggregator message-router pomba-kibana pomba-elasticsearch pomba-data-router pomba-search-data pomba-search-data pomba-elasticsearch portal portal-widget portal-db portal-sdk portal-db sdc sdc-dcae-be common.name sdc-be sdc-dcae-dt sdc-dcae-be sdc-dcae-fe sdc-dcae-be sdc-dcae-tosca-lab sdc-dcae-be sdc-fe sdc-kb sdc-wfd-fe sdc-wfd-be sdnc sdnc mysql sdnc-ansible-server sdnc dmaap-listener mysql sdnc message-router sdnc-portal mysql / sdnc-db sdnc sdnc-prom sdnc consul ueb-listener mysql sdnc sdc-be message-router so so mariadb so-bpmn-infra mariadb so-catalog-db-adapter mariadb so-openstack-adapter mariadb so-request-db-adapter mariadb so-sdc-adapter mariadb so-sdc-controller mariadb so-vfc-adapter mariadb vfc vfc-catalog vfc-db vfc-ems-driver mariadb // commented vfc-generic-vnfm-driver mariadb // commented vfc-hauwei-vnfm-driver mariadb // commented vfc-juju-vnfm-driver mariadb // commented vfc-multivim-proxy mariadb // commented vfc-nokia-vnfm-driver mariadb // commented vfc-nokia-v2vnfm-driver mariadb // commented vfc-nslcm vfc-db vfc-vnfmgr vfc-db vfc-resmgr mariadb // commented vfc-workflow mariadb // commented vfc-workflow-engine mariadb // commented vfc-vnflcm vfc-db vfc-vnfres vfc-db vfc-zte-sdnc-driver mariadb // commented vfc-zte-vnfm-driver mariadb // commented vid vid mariadb-galera vnfsdk postgres

 

Deployment Profiles

For use case specific deployments

# deployment profiles to cd.sh # each of the 28 components in alpha sequence --set pod.enabled=true|false status aaaccccd delmmnop pprssssu vvvv 11000100 10010101 01111011 0100 - vFW - no monitoring 11000100 10110101 11111011 0100 - vFW - with monitoring 11110111 10110111 11111111 0100 - vFW - cl

 

 

#

Team

pod

container

Required

y/n/fw/fwCl

Ram

Cpu

nodeport

Type

logback

Dependencies

#

Team

pod

container

Required

y/n/fw/fwCl

Ram

Cpu

nodeport

Type

logback

Dependencies

 

aaf

 

 

 

 

 

 

 

 

 

 

aai

 

 

y

 

 

 

 

 

 

 

appc

 

 

fwCL

 

 

 

 

 

 

 

clamp

 

 

fwCL

 

 

 

 

 

 

 

cli

 

 

 

 

 

 

 

 

 

 

consul

 

 

 

 

 

 

 

 

 

 

 

 

 

fwCL

 

 

 

 

 

 

 

dmaap

 

 

y

 

 

 

 

 

 

 

esr

 

 

 

 

 

 

 

 

 

 

log

 

 

n

 

 

 

 

 

 

 

sniro-

emulator

 

 

n

 

 

 

 

 

 

 

oof

 

 

n

 

 

 

 

 

 

 

msb

 

 

n

 

 

 

 

 

 

 

multicloud

 

 

n

 

 

 

 

 

 

 

nbi

 

 

 

 

 

 

 

 

 

 

policy

 

 

y

 

 

 

 

 

 

 

pomba

 

 

 

 

 

 

 

 

 

 

portal

 

 

y

 

 

 

 

 

 

 

robot

 

 

y

 

 

 

 

 

 

 

sdc

 

 

y

 

 

 

 

 

 

 

sdnc

 

 

y

 

 

 

 

 

 

 

so

 

 

y

 

 

 

 

 

 

 

uui

 

 

n

 

 

 

 

 

 

 

vfc

 

 

n

 

 

 

 

 

 

 

vid

 

 

y

 

 

 

 

 

 

 

vnfsdk

 

 

n

 

 

 

 

 

 

Pod

Container/service

Tech

URLs

Filebeat

Streaming

logback.xml

library

 

dependencies

Log level adjust

Pod

Container/service

Tech

URLs

Filebeat

Streaming

logback.xml

library

 

dependencies

Log level adjust

aaf

aaf-cm aaf-fs aaf-gui aaf-hello aaf-locate aaf-oauth aaf-service aaf-sms-vault-0

 

 

 

 

 

 

 

 

aai

aai-babel aai-champ aai-data-router aai-gizmo aai-graphadmin aai-modelloader aai-resources aai-search-data aai-sparky-be aai-spike aai-traversal

 

 

 

 

 

 

 

 

appc

appc-0

 

 

 

 

 

 

 

 

clamp

-

 

 

 

 

 

 

 

 

cli

-

 

 

 

 

 

 

 

 

consul

-

 

 

 

 

 

 

 

 

contrib-netbox

-

 

 

 

 

 

 

 

 

dcaegen2

-

 

 

 

 

 

 

 

 

dmaap

-

 

 

 

 

 

 

 

 

esr

-

 

 

 

 

 

 

 

 

log

-

 

 

 

 

 

 

 

 

logdemonode

-

 

 

 

 

 

 

 

 

msb

-

 

 

 

 

 

 

 

 

multicloud

-

 

 

 

 

 

 

 

 

nbi

-

 

 

 

 

 

 

 

 

oof

-

 

 

 

 

 

 

 

 

policy

-

 

 

 

 

 

 

 

 

pomba

-

 

 

 

 

 

 

 

 

portal

-

 

 

 

 

 

 

 

 

robot

-

 

 

 

 

 

 

 

 

sdc

-

 

 

 

 

 

 

 

 

sdnc

-

 

 

 

 

 

 

 

 

sniro

-

 

 

 

 

 

 

 

 

so

-

 

 

 

 

 

 

 

 

uui

-

 

 

 

 

 

 

 

 

vfc

-

 

 

 

 

 

 

 

 

vid

-

 

 

 

 

 

 

 

 

vnfsdk

-

 

 

 

 

 

 

 

 

vvp

-

 

 

 

 

 

 

 

 

Deployment Profile

Type

VMs

Total
RAM
vCores
HD

VM Flavor

K8S/Rancher Idle RAM

Deployed
Total RAM

Deployed
ONAP RAM

Pods

Max vCores

Idle vCores

HD/VM

HD
NFS
only

IOPS

Date

Notes
deployment post 75min

Type

VMs

Total
RAM
vCores
HD

VM Flavor

K8S/Rancher Idle RAM

Deployed
Total RAM

Deployed
ONAP RAM

Pods

Max vCores

Idle vCores

HD/VM

HD
NFS
only

IOPS

Date

Notes
deployment post 75min

Full Cluster (14 + 1) - recommended

15

224G
112 vC
100G/VM

16G, 8 vCores

C5.2xLarge

 

187Gb

102Gb

248 total
241 onap
217 up
0 error
24 config

 

18

6+G master
14 to 50 +G slave(s)

8.1G

 

20181106

 

Single VM (possible - not recommended)

1

432G
64 vC
120G

256G+ 32+ vCores

Rancher: 13G
Kubernetes: 8G
Top: 10G

151Gb

141Gb

240 total
233 onap
200 up
6 error
38 config

55

22

108 of 128G

n/a

Max: 550/sec
Idle: 220/sec

20181105

Tested on 432G/64vCore azure VM - R 1.6.22 K8S 1.11

Developer 1-n pods

1

16G
4 vC
100G

16/32G 4-16 vCores

 

14Gb

10Gb

 

 

 

120+G

n/a

 

 

AAI+robot only

Example 15 node (1 master + 14 nodes) OOM Deployment

empty

With ONAP deployed

Throughput and Volumetrics

Cloudwatch CPU Average

Specific to logging - we have a problem on any VM that contains AAI - the logstash container is being saturated there - see the 30+ percent VM - https://lf-onap.atlassian.net/browse/LOG-376

NFS Throughput for /dockerdata-nfs
Cloudwatch Network In Max

 

 

Details: 20181106:1800 EDT master

ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | wc -l 248 ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | wc -l 241 ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | grep -E '1/1|2/2' | wc -l 217 ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | grep -E '0/|1/2' | wc -l 24 ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces -o wide | grep onap | grep -E '0/|1/2' onap onap-aaf-aaf-sms-preload-lvqx9 0/1 Completed 0 4h 10.42.75.71 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-aaf-aaf-sshsm-distcenter-ql5f8 0/1 Completed 0 4h 10.42.75.223 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-aaf-aaf-sshsm-testca-7rzcd 0/1 Completed 0 4h 10.42.18.37 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-aai-aai-graphadmin-create-db-schema-26pfs 0/1 Completed 0 4h 10.42.14.14 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-aai-aai-traversal-update-query-data-qlk7w 0/1 Completed 0 4h 10.42.88.122 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-gmmvj 0/1 Completed 0 4h 10.42.111.99 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-n6fw4 0/1 Error 0 4h 10.42.21.12 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-nc8ww 0/1 Error 0 4h 10.42.109.156 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-xcxds 0/1 Error 0 4h 10.42.152.223 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-dr-node-6496d8f55b-jfvrm 0/1 Init:0/1 28 4h 10.42.95.32 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-dr-prov-86f79c47f9-tldsp 0/1 CrashLoopBackOff 59 4h 10.42.76.248 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-oof-music-cassandra-job-config-7mb5f 0/1 Completed 0 4h 10.42.38.249 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-oof-oof-has-healthcheck-rpst7 0/1 Completed 0 4h 10.42.241.223 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-oof-oof-has-onboard-5bd2l 0/1 Completed 0 4h 10.42.205.75 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-portal-portal-db-config-qshzn 0/2 Completed 0 4h 10.42.112.46 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-portal-portal-db-config-rk4m2 0/2 Init:Error 0 4h 10.42.57.79 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-sdc-sdc-be-config-backend-2vw2q 0/1 Completed 0 4h 10.42.87.181 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-sdc-sdc-be-config-backend-k57lh 0/1 Init:Error 0 4h 10.42.148.79 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-sdc-sdc-cs-config-cassandra-vgnz2 0/1 Completed 0 4h 10.42.111.187 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-sdc-sdc-es-config-elasticsearch-lkb9m 0/1 Completed 0 4h 10.42.20.202 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-sdc-sdc-onboarding-be-cassandra-init-7zv5j 0/1 Completed 0 4h 10.42.218.1 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-sdc-sdc-wfd-be-workflow-init-q8t7z 0/1 Completed 0 4h 10.42.255.91 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-vid-vid-galera-config-4f274 0/1 Completed 0 4h 10.42.80.200 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-init-postgres-lf659 0/1 Completed 0 4h 10.42.238.204 ip-172-31-38-194.us-east-2.compute.internal <none> ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE kube-system heapster-7b48b696fc-kxjlx 1/1 Running 0 5h 10.42.237.56 ip-172-31-37-24.us-east-2.compute.internal <none> kube-system kube-dns-6655f78c68-wzmvh 3/3 Running 0 5h 10.42.18.171 ip-172-31-37-24.us-east-2.compute.internal <none> kube-system kubernetes-dashboard-6f54f7c4b-58k4h 1/1 Running 0 5h 10.42.205.56 ip-172-31-37-24.us-east-2.compute.internal <none> kube-system monitoring-grafana-7877679464-9ssz5 1/1 Running 0 5h 10.42.252.38 ip-172-31-37-24.us-east-2.compute.internal <none> kube-system monitoring-influxdb-64664c6cf5-h4bxt 1/1 Running 0 5h 10.42.226.121 ip-172-31-37-24.us-east-2.compute.internal <none> kube-system tiller-deploy-6f4745cbcf-q6kxx 1/1 Running 0 5h 10.42.12.224 ip-172-31-37-24.us-east-2.compute.internal <none> onap dep-config-binding-service-68d9cd6f4c-9n2nf 2/2 Running 0 4h 10.42.157.143 ip-172-31-41-30.us-east-2.compute.internal <none> onap dep-dcae-datafile-collector-77956c65df-8mfwc 2/2 Running 0 4h 10.42.110.106 ip-172-31-41-30.us-east-2.compute.internal <none> onap dep-dcae-hv-ves-collector-55844fb468-9zdss 2/2 Running 0 4h 10.42.231.39 ip-172-31-45-152.us-east-2.compute.internal <none> onap dep-dcae-prh-74977b5fb4-n47b6 2/2 Running 0 4h 10.42.115.19 ip-172-31-39-138.us-east-2.compute.internal <none> onap dep-dcae-snmptrap-collector-69f99dc589-f7ffl 1/1 Running 0 4h 10.42.173.221 ip-172-31-41-229.us-east-2.compute.internal <none> onap dep-dcae-tca-analytics-84fbfbc894-xzq8c 2/2 Running 0 4h 10.42.28.146 ip-172-31-38-194.us-east-2.compute.internal <none> onap dep-dcae-ves-collector-98f8ccb5c-tghs4 2/2 Running 0 4h 10.42.207.99 ip-172-31-34-111.us-east-2.compute.internal <none> onap dep-deployment-handler-76cf6d665-tvcwp 2/2 Running 0 4h 10.42.11.59 ip-172-31-37-59.us-east-2.compute.internal <none> onap dep-holmes-engine-mgmt-9889496c5-2jbnb 1/1 Running 27 4h 10.42.186.10 ip-172-31-36-163.us-east-2.compute.internal <none> onap dep-holmes-rule-mgmt-9b49769b5-n6hqj 1/1 Running 27 4h 10.42.185.154 ip-172-31-39-138.us-east-2.compute.internal <none> onap dep-inventory-5c9f5bcc6d-lzqvh 1/1 Running 0 4h 10.42.68.75 ip-172-31-37-59.us-east-2.compute.internal <none> onap dep-policy-handler-544dcdfd75-vpsm7 2/2 Running 0 4h 10.42.233.2 ip-172-31-38-194.us-east-2.compute.internal <none> onap dep-pstg-write-f67b57947-mkbpq 1/1 Running 0 4h 10.42.185.41 ip-172-31-39-138.us-east-2.compute.internal <none> onap dep-service-change-handler-66b78745c-995ks 1/1 Running 3 4h 10.42.92.139 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-aaf-aaf-cm-7fcc8548cc-nrmsf 1/1 Running 0 4h 10.42.226.234 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-aaf-aaf-cs-5755ccf8f4-tz6fn 1/1 Running 0 4h 10.42.181.148 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-aaf-aaf-fs-7f8f9dcd74-27gfd 1/1 Running 0 4h 10.42.118.145 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-aaf-aaf-gui-bfd9c7fd6-qk6zw 1/1 Running 0 4h 10.42.201.9 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-aaf-aaf-hello-6447444665-5t24n 1/1 Running 0 4h 10.42.150.107 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-aaf-aaf-locate-845c5867db-2q8qv 1/1 Running 0 4h 10.42.190.184 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-aaf-aaf-oauth-68bc85dbd4-7qv7w 1/1 Running 0 4h 10.42.233.42 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-aaf-aaf-service-69fb68d68-qhgqc 1/1 Running 0 4h 10.42.108.135 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-7497c9bfcc-dtc8s 1/1 Running 0 4h 10.42.85.30 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-preload-lvqx9 0/1 Completed 0 4h 10.42.75.71 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-quorumclient-0 1/1 Running 0 4h 10.42.30.16 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-quorumclient-1 1/1 Running 0 4h 10.42.164.247 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-quorumclient-2 1/1 Running 0 4h 10.42.232.51 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-aaf-aaf-sms-vault-0 2/2 Running 1 4h 10.42.242.253 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-aaf-aaf-sshsm-distcenter-ql5f8 0/1 Completed 0 4h 10.42.75.223 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-aaf-aaf-sshsm-testca-7rzcd 0/1 Completed 0 4h 10.42.18.37 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-aai-aai-695546c99-l4vd4 1/1 Running 0 4h 10.42.61.157 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-aai-aai-babel-c489c944c-l6fxm 2/2 Running 0 4h 10.42.151.176 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-aai-aai-cassandra-0 1/1 Running 0 4h 10.42.60.200 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-aai-aai-cassandra-1 1/1 Running 1 4h 10.42.102.208 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-aai-aai-cassandra-2 1/1 Running 0 4h 10.42.168.115 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-aai-aai-champ-54f7986b6b-6qqk8 2/2 Running 0 4h 10.42.233.99 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-aai-aai-data-router-5b777d798c-hkt9p 2/2 Running 0 4h 10.42.254.152 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-aai-aai-elasticsearch-c9bf9dbf6-flljw 1/1 Running 0 4h 10.42.136.54 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-aai-aai-gizmo-5f8bf54f6f-4sj45 2/2 Running 0 4h 10.42.128.212 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-aai-aai-graphadmin-789c67bb8b-kd9kw 2/2 Running 0 4h 10.42.196.56 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-aai-aai-graphadmin-create-db-schema-26pfs 0/1 Completed 0 4h 10.42.14.14 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-aai-aai-modelloader-695fd4bf48-5gdmt 2/2 Running 0 4h 10.42.188.20 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-aai-aai-resources-765d6f6678-xg4jv 2/2 Running 0 4h 10.42.224.167 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-aai-aai-search-data-559b8dbc7f-k549g 2/2 Running 0 4h 10.42.110.212 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-aai-aai-sparky-be-75658695f5-wjkzw 2/2 Running 0 4h 10.42.7.139 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-aai-aai-spike-f854cf86d-fdx7j 2/2 Running 0 4h 10.42.175.92 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-aai-aai-traversal-5fcc7cc847-mrqtb 2/2 Running 0 4h 10.42.174.143 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-aai-aai-traversal-update-query-data-qlk7w 0/1 Completed 0 4h 10.42.88.122 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-appc-appc-0 2/2 Running 0 4h 10.42.106.203 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-appc-appc-ansible-server-78784978b6-jrz2b 1/1 Running 0 4h 10.42.232.15 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-appc-appc-cdt-d654974f6-bxhkw 1/1 Running 0 4h 10.42.170.154 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-appc-appc-db-0 1/1 Running 0 4h 10.42.215.183 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-appc-appc-db-1 1/1 Running 1 4h 10.42.113.105 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-appc-appc-db-2 1/1 Running 1 4h 10.42.245.153 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-appc-appc-dgbuilder-6bdc7bd56b-ssb2p 1/1 Running 0 4h 10.42.188.237 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-clamp-clamp-8cd95c6b4-kbmxg 2/2 Running 0 4h 10.42.149.44 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-clamp-clamp-dash-es-5566796d7c-b2mbd 1/1 Running 0 4h 10.42.51.26 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-clamp-clamp-dash-kibana-646fccf75d-vf68k 1/1 Running 0 4h 10.42.173.69 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-clamp-clamp-dash-logstash-7bfc956564-xhlj5 1/1 Running 0 4h 10.42.38.41 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-clamp-clampdb-6797447d-bkrbr 1/1 Running 0 4h 10.42.37.63 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-cli-cli-5498964cf9-ndk74 1/1 Running 0 4h 10.42.251.86 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-consul-consul-849447d678-8677q 1/1 Running 0 4h 10.42.56.217 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-consul-consul-server-0 1/1 Running 0 4h 10.42.69.75 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-consul-consul-server-1 1/1 Running 0 4h 10.42.156.89 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-consul-consul-server-2 1/1 Running 0 4h 10.42.69.41 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-6d4d768d9d-gksvt 1/1 Running 0 4h 10.42.179.105 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-gmmvj 0/1 Completed 0 4h 10.42.111.99 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-n6fw4 0/1 Error 0 4h 10.42.21.12 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-nc8ww 0/1 Error 0 4h 10.42.109.156 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-contrib-netbox-app-provisioning-xcxds 0/1 Error 0 4h 10.42.152.223 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-contrib-netbox-nginx-54c979ff7f-nk5kk 1/1 Running 0 4h 10.42.244.179 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-contrib-netbox-postgres-784b5d5fd9-2wdf7 1/1 Running 0 4h 10.42.175.55 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-bootstrap-7bb4b6bc57-gvz47 1/1 Running 0 4h 10.42.74.208 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-cloudify-manager-9cb7f4c66-7h77h 1/1 Running 0 4h 10.42.250.189 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-db-0 1/1 Running 0 4h 10.42.230.118 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-db-1 1/1 Running 0 4h 10.42.29.230 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-healthcheck-7b664b566c-w5w4h 1/1 Running 0 4h 10.42.9.151 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-pgpool-688d65b66d-jpv8x 1/1 Running 0 4h 10.42.177.56 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-pgpool-688d65b66d-srm5q 1/1 Running 0 4h 10.42.164.252 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-0 1/1 Running 0 4h 10.42.32.17 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-1 1/1 Running 0 4h 10.42.52.99 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-2 1/1 Running 0 4h 10.42.45.144 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-3 1/1 Running 0 4h 10.42.79.151 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-4 1/1 Running 0 4h 10.42.110.141 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-dcaegen2-dcae-redis-5 1/1 Running 0 4h 10.42.253.134 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-dmaap-dbc-pg-0 1/1 Running 0 4h 10.42.166.51 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-dmaap-dbc-pg-1 1/1 Running 0 4h 10.42.230.243 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-dmaap-dbc-pgpool-c5f8498-lr854 1/1 Running 0 4h 10.42.35.8 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-dmaap-dbc-pgpool-c5f8498-m84f5 1/1 Running 0 4h 10.42.240.60 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-bus-controller-59c96d6b8f-nlc4t 1/1 Running 0 4h 10.42.240.232 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-dr-db-557c66dc9d-4ltgp 1/1 Running 1 4h 10.42.210.116 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-dr-node-6496d8f55b-jfvrm 0/1 Init:0/1 28 4h 10.42.95.32 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-dmaap-dmaap-dr-prov-86f79c47f9-tldsp 0/1 CrashLoopBackOff 59 4h 10.42.76.248 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-dmaap-message-router-557cb765-sc4pb 1/1 Running 0 4h 10.42.157.140 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-dmaap-message-router-kafka-7b4d45b44c-7g5pz 1/1 Running 0 4h 10.42.218.166 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-dmaap-message-router-zookeeper-5bc7588665-pz9x5 1/1 Running 0 4h 10.42.129.130 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-esr-esr-gui-6c5ccd59d6-xzh46 1/1 Running 0 4h 10.42.89.47 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-esr-esr-server-5f967d4767-nl5xc 2/2 Running 0 4h 10.42.146.185 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-log-log-elasticsearch-7c5b74569f-g64hw 1/1 Running 0 4h 10.42.54.225 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-log-log-kibana-84b875cc6b-gdzp8 1/1 Running 0 4h 10.42.56.186 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-log-log-logstash-7hmrz 1/1 Running 0 4h 10.42.200.38 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-log-log-logstash-8cb9c 1/1 Running 0 4h 10.42.162.152 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-log-log-logstash-8pqrp 1/1 Running 0 4h 10.42.97.239 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-log-log-logstash-bvq5r 1/1 Running 0 4h 10.42.241.246 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-log-log-logstash-cckqk 1/1 Running 0 4h 10.42.222.107 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-log-log-logstash-gq826 1/1 Running 0 4h 10.42.129.134 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-log-log-logstash-kzvbv 1/1 Running 0 4h 10.42.195.222 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-log-log-logstash-mdbtv 1/1 Running 0 4h 10.42.54.127 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-log-log-logstash-n78q5 1/1 Running 0 4h 10.42.22.101 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-log-log-logstash-ns76c 1/1 Running 0 4h 10.42.124.239 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-log-log-logstash-pvsnw 1/1 Running 0 4h 10.42.164.130 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-log-log-logstash-q86z9 1/1 Running 0 4h 10.42.73.74 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-log-log-logstash-skr52 1/1 Running 0 4h 10.42.0.150 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-log-log-logstash-zjkph 1/1 Running 0 4h 10.42.47.5 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-msb-kube2msb-5c79ddd89f-4m524 1/1 Running 0 4h 10.42.86.56 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-msb-msb-consul-6949bd46f4-gsm59 1/1 Running 0 4h 10.42.19.136 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-msb-msb-discovery-86c7b945f9-ffxtx 2/2 Running 0 4h 10.42.27.240 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-msb-msb-eag-5f86f89c4f-4pr2t 2/2 Running 0 4h 10.42.36.35 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-msb-msb-iag-56cdd4c87b-4xtz7 2/2 Running 0 4h 10.42.152.29 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-867df449f8-gwkdw 2/2 Running 0 4h 10.42.87.4 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-azure-5db4ddd44d-cpphs 2/2 Running 0 4h 10.42.84.246 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-ocata-c557948df-5sgfs 2/2 Running 0 4h 10.42.39.16 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-pike-8ff99dfb6-zmrw8 2/2 Running 0 4h 10.42.176.68 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-vio-866cd4bf9b-g2zlx 2/2 Running 0 4h 10.42.64.228 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-multicloud-multicloud-windriver-7775ddc4b9-4278w 2/2 Running 0 4h 10.42.87.19 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-nbi-nbi-7595d4dbc-9m95t 1/1 Running 1 4h 10.42.94.19 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-nbi-nbi-mariadb-65f556b8b7-fgjzx 1/1 Running 0 4h 10.42.195.127 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-nbi-nbi-mongo-0 1/1 Running 0 4h 10.42.184.20 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-oof-cmso-db-0 1/1 Running 0 4h 10.42.193.70 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-oof-music-cassandra-0 1/1 Running 1 4h 10.42.189.19 ip-172-31-42-33.us-east-2.compute.internal <none> onap onap-oof-music-cassandra-1 1/1 Running 0 4h 10.42.163.171 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-oof-music-cassandra-2 1/1 Running 0 4h 10.42.195.0 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-oof-music-cassandra-job-config-7mb5f 0/1 Completed 0 4h 10.42.38.249 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-oof-music-tomcat-7974f8748c-4pgg8 1/1 Running 0 4h 10.42.118.22 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-oof-music-tomcat-7974f8748c-dxdds 1/1 Running 0 4h 10.42.159.58 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-oof-music-tomcat-7974f8748c-jh7jg 1/1 Running 0 4h 10.42.64.7 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-oof-oof-6dbf897c86-8d6bx 1/1 Running 0 4h 10.42.20.170 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-oof-oof-cmso-service-7d54b6b478-djvdm 1/1 Running 0 4h 10.42.192.55 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-oof-oof-has-api-774f9458cb-mtfg8 1/1 Running 0 4h 10.42.208.73 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-oof-oof-has-controller-7998c7f7b6-vg6nz 1/1 Running 0 4h 10.42.91.250 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-oof-oof-has-data-5dffb4c7df-qmjzz 1/1 Running 0 4h 10.42.213.36 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-oof-oof-has-healthcheck-rpst7 0/1 Completed 0 4h 10.42.241.223 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-oof-oof-has-onboard-5bd2l 0/1 Completed 0 4h 10.42.205.75 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-oof-oof-has-reservation-59759db679-dz4cs 1/1 Running 0 4h 10.42.48.244 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-oof-oof-has-solver-5bfc76f46-hfc88 1/1 Running 0 4h 10.42.225.129 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-oof-zookeeper-0 1/1 Running 0 4h 10.42.171.61 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-oof-zookeeper-1 1/1 Running 0 4h 10.42.251.101 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-oof-zookeeper-2 1/1 Running 0 4h 10.42.65.144 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-policy-brmsgw-86dd8fc4f7-xtzk9 1/1 Running 0 4h 10.42.56.63 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-policy-drools-0 1/1 Running 0 4h 10.42.75.29 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-policy-nexus-b7f484bdb-gmn76 1/1 Running 0 4h 10.42.45.249 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-policy-pap-6555659676-77nl7 2/2 Running 0 4h 10.42.87.216 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-policy-pdp-0 2/2 Running 0 4h 10.42.63.230 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-policy-policy-apex-pdp-0 1/1 Running 0 4h 10.42.73.87 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-policy-policy-distribution-79544b78c7-hk5mk 1/1 Running 0 4h 10.42.104.145 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-policy-policydb-79cdf7cb7c-jb82c 1/1 Running 0 4h 10.42.98.252 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-pomba-pomba-aaictxbuilder-684d5977cd-sj8dm 2/2 Running 0 4h 10.42.170.23 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-pomba-pomba-contextaggregator-779745878d-4s8c9 1/1 Running 0 4h 10.42.154.6 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-pomba-pomba-data-router-59bb4fd899-dc5zc 1/1 Running 0 4h 10.42.171.88 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-pomba-pomba-elasticsearch-7b8bc5f864-5mjqk 1/1 Running 0 4h 10.42.111.181 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-pomba-pomba-kibana-6dd8dbb68b-xltcm 1/1 Running 0 4h 10.42.181.201 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-pomba-pomba-networkdiscovery-d47f68c88-j2zw4 2/2 Running 0 4h 10.42.34.101 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-pomba-pomba-networkdiscoveryctxbuilder-6978dc6c79-6z7zm 2/2 Running 0 4h 10.42.116.177 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-pomba-pomba-sdcctxbuilder-fcbc7f96b-nj2c9 1/1 Running 0 4h 10.42.87.59 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-pomba-pomba-search-data-5b4d8f7dc6-k7qlx 2/2 Running 0 4h 10.42.37.226 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-pomba-pomba-servicedecomposition-99b445978-6crkz 2/2 Running 0 4h 10.42.46.61 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-pomba-pomba-validation-service-5c8454484-t4kls 1/1 Running 0 4h 10.42.235.206 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-portal-portal-app-67bc49d88-tpxbq 2/2 Running 0 4h 10.42.25.53 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-portal-portal-cassandra-75d4746d76-jn78x 1/1 Running 0 4h 10.42.128.240 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-portal-portal-db-7b768ffcc-92q8b 1/1 Running 1 4h 10.42.187.56 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-portal-portal-db-config-qshzn 0/2 Completed 0 4h 10.42.112.46 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-portal-portal-db-config-rk4m2 0/2 Init:Error 0 4h 10.42.57.79 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-portal-portal-sdk-59fbd4bd9d-6j9x9 2/2 Running 0 4h 10.42.217.29 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-portal-portal-widget-79587fdf56-5dh2c 1/1 Running 0 4h 10.42.12.249 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-portal-portal-zookeeper-7c9444bfb5-9bvwq 1/1 Running 0 4h 10.42.113.205 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-robot-robot-86c5f986b7-kd257 1/1 Running 0 4h 10.42.246.138 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-sdc-sdc-be-7fdc576bfb-6q5bx 2/2 Running 0 4h 10.42.142.158 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-sdc-sdc-be-config-backend-2vw2q 0/1 Completed 0 4h 10.42.87.181 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-sdc-sdc-be-config-backend-k57lh 0/1 Init:Error 0 4h 10.42.148.79 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-sdc-sdc-cs-5b9bb5756d-28zhd 1/1 Running 1 4h 10.42.232.24 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-sdc-sdc-cs-config-cassandra-vgnz2 0/1 Completed 0 4h 10.42.111.187 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-sdc-sdc-es-8bdf86659-bvgj6 1/1 Running 0 4h 10.42.21.223 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-sdc-sdc-es-config-elasticsearch-lkb9m 0/1 Completed 0 4h 10.42.20.202 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-sdc-sdc-fe-65c95cf8bf-2944f 2/2 Running 0 4h 10.42.193.193 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-sdc-sdc-kb-79bfffc4cf-6p5qn 1/1 Running 0 4h 10.42.59.138 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-sdc-sdc-onboarding-be-84466885d5-pwqpn 2/2 Running 0 4h 10.42.163.142 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-sdc-sdc-onboarding-be-cassandra-init-7zv5j 0/1 Completed 0 4h 10.42.218.1 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-sdc-sdc-wfd-be-54f49cb9f-8wmkz 1/1 Running 0 4h 10.42.69.253 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-sdc-sdc-wfd-be-workflow-init-q8t7z 0/1 Completed 0 4h 10.42.255.91 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-sdc-sdc-wfd-fe-585bd84f8d-grwwh 2/2 Running 0 4h 10.42.110.244 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-sdnc-controller-blueprints-7c8f9dbfb7-tq4f5 1/1 Running 0 4h 10.42.200.245 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-sdnc-controller-blueprints-db-0 1/1 Running 0 4h 10.42.207.149 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-sdnc-nengdb-0 1/1 Running 0 4h 10.42.211.255 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-sdnc-network-name-gen-f4ff8b99f-8bxbh 1/1 Running 0 4h 10.42.107.56 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-0 2/2 Running 0 4h 10.42.220.124 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-ansible-server-dffcf4c8b-jdpmh 1/1 Running 0 4h 10.42.84.202 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-db-0 2/2 Running 0 4h 10.42.45.121 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-dgbuilder-5496ff5f86-5l27s 1/1 Running 0 4h 10.42.17.254 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-dmaap-listener-54f5c69c7c-bxp2n 1/1 Running 0 4h 10.42.42.177 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-portal-8ff4bb696-j9xz5 1/1 Running 0 4h 10.42.250.3 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-sdnc-sdnc-ueb-listener-5d5749f879-2pw5j 1/1 Running 0 4h 10.42.93.48 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-sniro-emulator-sniro-emulator-5bdc8cc4cb-qjbk8 1/1 Running 0 4h 10.42.207.251 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-so-so-7f5f98949b-w85kc 1/1 Running 0 4h 10.42.110.129 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-so-so-bpmn-infra-74fcf888b4-9tnfc 1/1 Running 0 4h 10.42.60.90 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-so-so-catalog-db-adapter-5b8cb47dc-j44bt 1/1 Running 0 4h 10.42.222.142 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-so-so-mariadb-5b5bd695c4-jdx8v 1/1 Running 0 4h 10.42.249.209 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-so-so-monitoring-855456948b-fr24q 1/1 Running 0 4h 10.42.14.54 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-so-so-openstack-adapter-c44cd98dc-pk2gb 1/1 Running 1 4h 10.42.25.240 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-so-so-request-db-adapter-745dc6677-z7tcx 1/1 Running 0 4h 10.42.186.49 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-so-so-sdc-controller-67f8cbbc7c-kdfmr 1/1 Running 0 4h 10.42.23.215 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-so-so-sdnc-adapter-5cbc5f8dd8-6k22h 1/1 Running 0 4h 10.42.14.50 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-so-so-vfc-adapter-6c5996d748-5h2jj 1/1 Running 0 4h 10.42.205.174 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-uui-uui-d5ff69677-xkmq2 1/1 Running 0 4h 10.42.157.9 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-uui-uui-server-7cc8bcb956-c28g5 1/1 Running 0 4h 10.42.245.99 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-vfc-vfc-catalog-66dd84cc97-htgsr 2/2 Running 0 4h 10.42.248.74 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-vfc-vfc-db-65d8fd5567-25z45 1/1 Running 0 4h 10.42.181.50 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-vfc-vfc-ems-driver-6f5fd887db-hrh7q 1/1 Running 0 4h 10.42.28.44 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-vfc-vfc-generic-vnfm-driver-7964b7f4cc-nd2d2 2/2 Running 1 4h 10.42.172.200 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-vfc-vfc-huawei-vnfm-driver-6c5676758d-vs8fh 2/2 Running 0 4h 10.42.58.24 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-vfc-vfc-juju-vnfm-driver-67b9c4bb4c-fhv45 2/2 Running 0 4h 10.42.10.58 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-vfc-vfc-multivim-proxy-5f6f455bfd-tg7m2 1/1 Running 0 4h 10.42.51.118 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-vfc-vfc-nokia-v2vnfm-driver-6f7646f69d-22p88 1/1 Running 0 4h 10.42.158.31 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-vfc-vfc-nokia-vnfm-driver-7c9d59597f-gkmct 2/2 Running 0 4h 10.42.184.15 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-vfc-vfc-nslcm-c8478dcc8-cnlvd 2/2 Running 0 4h 10.42.144.183 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-vfc-vfc-resmgr-6d7d7bc78b-f5v88 2/2 Running 0 4h 10.42.255.238 ip-172-31-37-24.us-east-2.compute.internal <none> onap onap-vfc-vfc-vnflcm-fb656d967-p6wh2 2/2 Running 0 4h 10.42.56.73 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-vfc-vfc-vnfmgr-66d5cd85f7-ld4v4 2/2 Running 0 4h 10.42.121.241 ip-172-31-34-207.us-east-2.compute.internal <none> onap onap-vfc-vfc-vnfres-7757d4b486-bkqqz 2/2 Running 0 4h 10.42.100.120 ip-172-31-34-111.us-east-2.compute.internal <none> onap onap-vfc-vfc-workflow-68f85b8fcf-mxgwl 1/1 Running 0 4h 10.42.72.65 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-vfc-vfc-workflow-engine-7d76bb4b56-tj6ks 1/1 Running 0 4h 10.42.48.13 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-vfc-vfc-zte-sdnc-driver-58d4fc6d9f-hk4d9 1/1 Running 0 4h 10.42.245.72 ip-172-31-41-30.us-east-2.compute.internal <none> onap onap-vfc-vfc-zte-vnfm-driver-fc669cc8d-64k6j 2/2 Running 1 4h 10.42.68.226 ip-172-31-41-229.us-east-2.compute.internal <none> onap onap-vid-vid-5cb464799b-fw69j 2/2 Running 0 4h 10.42.21.193 ip-172-31-41-110.us-east-2.compute.internal <none> onap onap-vid-vid-galera-config-4f274 0/1 Completed 0 4h 10.42.80.200 ip-172-31-33-223.us-east-2.compute.internal <none> onap onap-vid-vid-mariadb-galera-0 1/1 Running 0 4h 10.42.40.225 ip-172-31-45-152.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-6b45bfffb4-phcwx 1/1 Running 0 4h 10.42.32.180 ip-172-31-38-95.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-init-postgres-lf659 0/1 Completed 0 4h 10.42.238.204 ip-172-31-38-194.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-pgpool-6874f7b695-69njk 1/1 Running 0 4h 10.42.239.14 ip-172-31-36-163.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-pgpool-6874f7b695-pjmwp 1/1 Running 0 4h 10.42.171.177 ip-172-31-39-138.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-postgres-0 1/1 Running 0 4h 10.42.106.183 ip-172-31-37-59.us-east-2.compute.internal <none> onap onap-vnfsdk-vnfsdk-postgres-1 1/1 Running 0 4h 10.42.47.66 ip-172-31-41-229.us-east-2.compute.internal <none> ubuntu@ip-172-31-40-250:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-172-31-33-223.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.222.148.116 18.222.148.116 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-34-111.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 3.16.37.170 3.16.37.170 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-34-207.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.225.32.201 18.225.32.201 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-36-163.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 13.58.189.251 13.58.189.251 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-37-24.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.224.180.26 18.224.180.26 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-37-59.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.191.248.14 18.191.248.14 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-38-194.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.217.45.91 18.217.45.91 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-38-95.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 52.15.39.21 52.15.39.21 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-39-138.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.224.199.40 18.224.199.40 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-41-110.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.223.151.180 18.223.151.180 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-41-229.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 18.218.252.13 18.218.252.13 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-41-30.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 3.16.113.3 3.16.113.3 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-42-33.us-east-2.compute.internal Ready <none> 5h v1.11.2-rancher1 13.59.2.86 13.59.2.86 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ip-172-31-45-152.us-east-2.compute.internal Ready <none> 4h v1.11.2-rancher1 18.219.56.50 18.219.56.50 Ubuntu 16.04.1 LTS 4.4.0-1049-aws docker://17.3.2 ubuntu@ip-172-31-40-250:~$ kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% ip-172-31-33-223.us-east-2.compute.internal 852m 10% 13923Mi 90% ip-172-31-34-111.us-east-2.compute.internal 1160m 14% 11643Mi 75% ip-172-31-34-207.us-east-2.compute.internal 1101m 13% 7981Mi 51% ip-172-31-36-163.us-east-2.compute.internal 656m 8% 13377Mi 87% ip-172-31-37-24.us-east-2.compute.internal 401m 5% 8543Mi 55% ip-172-31-37-59.us-east-2.compute.internal 711m 8% 10873Mi 70% ip-172-31-38-194.us-east-2.compute.internal 1136m 14% 8195Mi 53% ip-172-31-38-95.us-east-2.compute.internal 1195m 14% 9127Mi 59% ip-172-31-39-138.us-east-2.compute.internal 296m 3% 10870Mi 70% ip-172-31-41-110.us-east-2.compute.internal 2586m 32% 10950Mi 71% ip-172-31-41-229.us-east-2.compute.internal 159m 1% 9138Mi 59% ip-172-31-41-30.us-east-2.compute.internal 180m 2% 9862Mi 64% ip-172-31-42-33.us-east-2.compute.internal 1573m 19% 6352Mi 41% ip-172-31-45-152.us-east-2.compute.internal 1579m 19% 10633Mi 69% ubuntu@ip-172-31-40-250:~/oom/kubernetes/robot$ sudo ./ete-k8s.sh onap health ./ete-k8s.sh: 22: [: onap: unexpected operator ./ete-k8s.sh: 22: [: health: unexpected operator + export NAMESPACE=onap Testsuites ============================================================================== Testsuites.Health-Check :: Testing ecomp components are available via calls. ============================================================================== Basic A&AI Health Check | PASS | ------------------------------------------------------------------------------ Basic AAF Health Check | PASS | ------------------------------------------------------------------------------ Basic AAF SMS Health Check | PASS | ------------------------------------------------------------------------------ Basic APPC Health Check | PASS | ------------------------------------------------------------------------------ Basic CLI Health Check | PASS | ------------------------------------------------------------------------------ Basic CLAMP Health Check | PASS | ------------------------------------------------------------------------------ Basic DCAE Health Check | PASS | ------------------------------------------------------------------------------ Basic DMAAP Data Router Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f536960ea50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv [ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f536972e450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv [ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5369617090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv | FAIL | ConnectionError: HTTPConnectionPool(host='dmaap-dr-node.onap', port=8080): Max retries exceeded with url: /internal/fetchProv (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5369d21850>: Failed to establish a new connection: [Errno 111] Connection refused',)) ------------------------------------------------------------------------------ Basic DMAAP Message Router Health Check | PASS | ------------------------------------------------------------------------------ Basic External API NBI Health Check | PASS | ------------------------------------------------------------------------------ Basic Log Elasticsearch Health Check | PASS | ------------------------------------------------------------------------------ Basic Log Kibana Health Check | PASS | ------------------------------------------------------------------------------ Basic Log Logstash Health Check | PASS | ------------------------------------------------------------------------------ Basic Microservice Bus Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-ocata API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-pike API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-titanium_cloud API Health Check | PASS | ------------------------------------------------------------------------------ Basic Multicloud-vio API Health Check | PASS | ------------------------------------------------------------------------------ Basic OOF-Homing Health Check | PASS | ------------------------------------------------------------------------------ Basic OOF-SNIRO Health Check | PASS | ------------------------------------------------------------------------------ Basic OOF-CMSO Health Check | PASS | ------------------------------------------------------------------------------ Basic Policy Health Check | PASS | ------------------------------------------------------------------------------ Basic Pomba AAI-context-builder Health Check | PASS | ------------------------------------------------------------------------------ Basic Pomba SDC-context-builder Health Check | PASS | ------------------------------------------------------------------------------ Basic Pomba Network-discovery-context-builder Health Check | PASS | ------------------------------------------------------------------------------ Basic Portal Health Check | PASS | ------------------------------------------------------------------------------ Basic SDC Health Check (DMaaP:UP)| PASS | ------------------------------------------------------------------------------ Basic SDNC Health Check | PASS | ------------------------------------------------------------------------------ Basic SO Health Check | PASS | ------------------------------------------------------------------------------ Basic UseCaseUI API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC catalog API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC emsdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC gvnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC huaweivnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC jujuvnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC multivimproxy API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC nokiavnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC nokiav2driver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC nslcm API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC resmgr API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC vnflcm API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC vnfmgr API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC vnfres API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC workflow API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC ztesdncdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VFC ztevnfmdriver API Health Check | PASS | ------------------------------------------------------------------------------ Basic VID Health Check | PASS | ------------------------------------------------------------------------------ Basic VNFSDK Health Check | PASS | ------------------------------------------------------------------------------ Basic Holmes Rule Management API Health Check | FAIL | 502 != 200 ------------------------------------------------------------------------------ Basic Holmes Engine Management API Health Check | FAIL | 502 != 200 ------------------------------------------------------------------------------ Testsuites.Health-Check :: Testing ecomp components are available ... | FAIL | 51 critical tests, 48 passed, 3 failed 51 tests total, 48 passed, 3 failed

ONAP Module Dependencies - Compile and Runtime

 

 

Existing Library Research

Portal/SDK

SO

AAI

Library Selected for Reference Implementation

Java

Python

Specification

Developed in Casablanca - implemented in Dublin - ONAP Application Logging Specification v1.2 (Casablanca)

Use Cases

Design Issues

Developer Guide

Testing Guide

Change Log

Comments