Versions Compared

Old Version 15

changes.mady.by.user Michael O'Brien

Saved on

compared with

New Version Current

changes.mady.by.user Michael O'Brien

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Going with the casablanca spec for Dublin - for the library to implement this spec - retrofit portal/sdk (or SO's library) - need to have an AOP enabled library like our reference - 3rd choice is to AOP enable AAI's

Active Logging Specifications

deprecated - 20181025: update: originally we decided to continue the casablanca version v111 of ONAP Application Logging Specification v1.2 (Casablanca) in Dublin but after constructive feedback/pushback from one small part of the AT&T based Acumos community and feedback/pushback from part of the AT&T based ECOMP community based on initial pulls into ECOMP and some feedback from 2 high level projects in ONAP - I recommend refocusing on only one requirement - a common ONAP first spec.  Creating a 3-way spec that satisfies Acumos, ECOMP and ONAP can no longer be our focus due to the limited time and very limited resources of the logging spec/ELK deployment work a couple of us need to do.  I have received tremendous support and spec/JIRA  work from Lorraine A. Welch, Spondon Dey, Shishir Thakore and Luke Parker (all the marker/mdc work and slf4j library - and originating the project spec/elk stack).  Creating a combined specification that meets the needs of private AT&T ECOMP that consumes Open Source ONAP, public Open Source Acumos that consumes parts of ONAP and even ONAP itself with it's several Logging implementations across Portal, SO, AAI is not really working or will take too long to accept a new library that everyone can accept.  Since I am the current PTL, I have decided that an "ONAP first" direction is required.  Any implementation/spec changes will need to be prototyped in public first and accepted directly into affected projects as we change the spec - not as an exercise after we have finalized the spec.   I recommend that the way forward as suggested by the core of my coworkers in Acumos is to take an existing library and slowly iterate that library to match an "ONAP" specific spec that includes markers and MDC's and only if possible keep the older formats by the ONAP consuming projects.   I will start from the bottom up for Dublin - using the portal/sdk library and adjust it in sync with any spec changes in parallel (no spec changes without first prototyping them in existing use cases by users of the portal/sdk library).  For the E release we can increase adoption to the rest of the components not currently using portal/sdk.  What is critical this time around is working with the team(s) affected by any adjustment to their library first.  Secondary consideration will be given to non-opensource consumers of this specification inside corporations that privately wrap ONAP.  This will mean becoming part of the public portal and policy weekly meetings as a de-facto sub-project. The first order of work will be infrastructure (shipping logs to ELK) and then an alignment to the existing portal/sdk logback.xml spec - before we start changing things. so we can start mining the ELK logs before we adjust them. 


Table of Contents

Statement of Work

    Detail any changes to the existing casablanca spec (implementing in Dublin) - Active Logging Specifications - for the upcoming E release.

    Each specification is created in the release prior to where it is implemented 

    Work to align the logs, ship them, consolidate, provide for analysis, backup and restore them will be done from a developer perspective for the Dublin release.

    see Logging Dublin Scope

Features

Key words are: out of the box, reuse components, iterative changes - not major spec changes.

...

UC1: Provide for transaction tracing via elasticsearch or dashboard for distributed transactions across ONAP components - enabled by the logs shipped to the ELK stack - in real time.
(the ELK stack is there - we just need to adjust log content - amount and format)

UC2: Provide for UC1 tracing via standardized logs - ideally via marker/mdc (label/key:value pair) markup


Logging Alignment before enhancement

Plan is to take the portal/sdk library - align the rest of onap to this library and then add marker/mdc (labels and key/value pair) support as required in phase 2

Work will include 1) verify pipeline 2) verify logback format

Open Standards

Look at a standard that will allow us to use standard parsing libraries all the way to eventual AI/ML by adopting a standard like open tracing.

Infrastructure Improvements

including filebeat sidecar anti-pattern/replacement, elk stack upgrade, deployment template, library language scope (python, swift...)

Monitoring

Investigate out of the box prometheus monitoring

S3P

backup/restore, security, log format monitoring, elk dashboards

3rd party consumption alignment

Lower in priority will be to align onap with requirements of Acumos and ECOMP for example - the focus will be on alignment of ONAP components only - to the primary goal of providing a transaction tracing system via the ELK stack.

Analysis

Existing ONAP Logging State - Infrastructure and Schema

...

Tech

...

Code Block
themeMidnight
aaf-cm
aaf-fs
aaf-gui
aaf-hello
aaf-locate
aaf-oauth
aaf-service
aaf-sms-vault-0

...

Code Block
themeMidnight
aai-babel
aai-champ
aai-data-router
aai-gizmo
aai-graphadmin
aai-modelloader
aai-resources
aai-search-data
aai-sparky-be
aai-spike
aai-traversal

...

Code Block
themeMidnight
appc-0

Deployment Profile

...

Deployed
Total RAM

...

16G, 8 vCores

C5.2xLarge

...

6+G master
14 to 50 +G slave(s)

...

240 total
233 onap
200 up
6 error
38 config

...

16G
4 vC
100G

...

Example 15 node (1 master + 14 nodes) OOM Deployment

empty

Image Removed

With ONAP deployed

Image Removed

Throughput and Volumetrics

Cloudwatch CPU Average

Specific to logging - we have a problem on any VM that contains AAI - the logstash container is being saturated there - see the 30+ percent VM - 

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyLOG-376

Image Removed

NFS Throughput for /dockerdata-nfs

Image Removed

Cloudwatch Network In Max

Image Removed

Details: 20181106:1800 EDT master

Code Block
themeMidnight
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | wc -l
248
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | wc -l
241
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | grep -E '1/1|2/2' | wc -l
217
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces | grep onap | grep -E '0/|1/2' | wc -l
24
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces -o wide | grep onap | grep -E '0/|1/2' 
onap          onap-aaf-aaf-sms-preload-lvqx9                                 0/1       Completed          0          4h        10.42.75.71     ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-aaf-aaf-sshsm-distcenter-ql5f8                            0/1       Completed          0          4h        10.42.75.223    ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-aaf-aaf-sshsm-testca-7rzcd                                0/1       Completed          0          4h        10.42.18.37     ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-aai-aai-graphadmin-create-db-schema-26pfs                 0/1       Completed          0          4h        10.42.14.14     ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-aai-aai-traversal-update-query-data-qlk7w                 0/1       Completed          0          4h        10.42.88.122    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-contrib-netbox-app-provisioning-gmmvj                     0/1       Completed          0          4h        10.42.111.99    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-contrib-netbox-app-provisioning-n6fw4                     0/1       Error              0          4h        10.42.21.12     ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-contrib-netbox-app-provisioning-nc8ww                     0/1       Error              0          4h        10.42.109.156   ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-contrib-netbox-app-provisioning-xcxds                     0/1       Error              0          4h        10.42.152.223   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-dmaap-dmaap-dr-node-6496d8f55b-jfvrm                      0/1       Init:0/1           28         4h        10.42.95.32     ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-dmaap-dmaap-dr-prov-86f79c47f9-tldsp                      0/1       CrashLoopBackOff   59         4h        10.42.76.248    ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-oof-music-cassandra-job-config-7mb5f                      0/1       Completed          0          4h        10.42.38.249    ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-oof-oof-has-healthcheck-rpst7                             0/1       Completed          0          4h        10.42.241.223   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-oof-oof-has-onboard-5bd2l                                 0/1       Completed          0          4h        10.42.205.75    ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-portal-portal-db-config-qshzn                             0/2       Completed          0          4h        10.42.112.46    ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-portal-portal-db-config-rk4m2                             0/2       Init:Error         0          4h        10.42.57.79     ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-be-config-backend-2vw2q                           0/1       Completed          0          4h        10.42.87.181    ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-be-config-backend-k57lh                           0/1       Init:Error         0          4h        10.42.148.79    ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-cs-config-cassandra-vgnz2                         0/1       Completed          0          4h        10.42.111.187   ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-es-config-elasticsearch-lkb9m                     0/1       Completed          0          4h        10.42.20.202    ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-onboarding-be-cassandra-init-7zv5j                0/1       Completed          0          4h        10.42.218.1     ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-wfd-be-workflow-init-q8t7z                        0/1       Completed          0          4h        10.42.255.91    ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-vid-vid-galera-config-4f274                               0/1       Completed          0          4h        10.42.80.200    ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-vnfsdk-vnfsdk-init-postgres-lf659                         0/1       Completed          0          4h        10.42.238.204   ip-172-31-38-194.us-east-2.compute.internal   <none>
ubuntu@ip-172-31-40-250:~$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                                           READY     STATUS             RESTARTS   AGE       IP              NODE                                          NOMINATED NODE
kube-system   heapster-7b48b696fc-kxjlx                                      1/1       Running            0          5h        10.42.237.56    ip-172-31-37-24.us-east-2.compute.internal    <none>
kube-system   kube-dns-6655f78c68-wzmvh                                      3/3       Running            0          5h        10.42.18.171    ip-172-31-37-24.us-east-2.compute.internal    <none>
kube-system   kubernetes-dashboard-6f54f7c4b-58k4h                           1/1       Running            0          5h        10.42.205.56    ip-172-31-37-24.us-east-2.compute.internal    <none>
kube-system   monitoring-grafana-7877679464-9ssz5                            1/1       Running            0          5h        10.42.252.38    ip-172-31-37-24.us-east-2.compute.internal    <none>
kube-system   monitoring-influxdb-64664c6cf5-h4bxt                           1/1       Running            0          5h        10.42.226.121   ip-172-31-37-24.us-east-2.compute.internal    <none>
kube-system   tiller-deploy-6f4745cbcf-q6kxx                                 1/1       Running            0          5h        10.42.12.224    ip-172-31-37-24.us-east-2.compute.internal    <none>
onap          dep-config-binding-service-68d9cd6f4c-9n2nf                    2/2       Running            0          4h        10.42.157.143   ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          dep-dcae-datafile-collector-77956c65df-8mfwc                   2/2       Running            0          4h        10.42.110.106   ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          dep-dcae-hv-ves-collector-55844fb468-9zdss                     2/2       Running            0          4h        10.42.231.39    ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          dep-dcae-prh-74977b5fb4-n47b6                                  2/2       Running            0          4h        10.42.115.19    ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          dep-dcae-snmptrap-collector-69f99dc589-f7ffl                   1/1       Running            0          4h        10.42.173.221   ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          dep-dcae-tca-analytics-84fbfbc894-xzq8c                        2/2       Running            0          4h        10.42.28.146    ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          dep-dcae-ves-collector-98f8ccb5c-tghs4                         2/2       Running            0          4h        10.42.207.99    ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          dep-deployment-handler-76cf6d665-tvcwp                         2/2       Running            0          4h        10.42.11.59     ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          dep-holmes-engine-mgmt-9889496c5-2jbnb                         1/1       Running            27         4h        10.42.186.10    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          dep-holmes-rule-mgmt-9b49769b5-n6hqj                           1/1       Running            27         4h        10.42.185.154   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          dep-inventory-5c9f5bcc6d-lzqvh                                 1/1       Running            0          4h        10.42.68.75     ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          dep-policy-handler-544dcdfd75-vpsm7                            2/2       Running            0          4h        10.42.233.2     ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          dep-pstg-write-f67b57947-mkbpq                                 1/1       Running            0          4h        10.42.185.41    ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          dep-service-change-handler-66b78745c-995ks                     1/1       Running            3          4h        10.42.92.139    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-aaf-aaf-cm-7fcc8548cc-nrmsf                               1/1       Running            0          4h        10.42.226.234   ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-aaf-aaf-cs-5755ccf8f4-tz6fn                               1/1       Running            0          4h        10.42.181.148   ip-172-31-42-33.us-east-2.compute.internal    <none>
onap          onap-aaf-aaf-fs-7f8f9dcd74-27gfd                               1/1       Running            0          4h        10.42.118.145   ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-aaf-aaf-gui-bfd9c7fd6-qk6zw                               1/1       Running            0          4h        10.42.201.9     ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-aaf-aaf-hello-6447444665-5t24n                            1/1       Running            0          4h        10.42.150.107   ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-aaf-aaf-locate-845c5867db-2q8qv                           1/1       Running            0          4h        10.42.190.184   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-aaf-aaf-oauth-68bc85dbd4-7qv7w                            1/1       Running            0          4h        10.42.233.42    ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-aaf-aaf-service-69fb68d68-qhgqc                           1/1       Running            0          4h        10.42.108.135   ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-aaf-aaf-sms-7497c9bfcc-dtc8s                              1/1       Running            0          4h        10.42.85.30     ip-172-31-42-33.us-east-2.compute.internal    <none>
onap          onap-aaf-aaf-sms-preload-lvqx9                                 0/1       Completed          0          4h        10.42.75.71     ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-aaf-aaf-sms-quorumclient-0                                1/1       Running            0          4h        10.42.30.16     ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-aaf-aaf-sms-quorumclient-1                                1/1       Running            0          4h        10.42.164.247   ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-aaf-aaf-sms-quorumclient-2                                1/1       Running            0          4h        10.42.232.51    ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-aaf-aaf-sms-vault-0                                       2/2       Running            1          4h        10.42.242.253   ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-aaf-aaf-sshsm-distcenter-ql5f8                            0/1       Completed          0          4h        10.42.75.223    ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-aaf-aaf-sshsm-testca-7rzcd                                0/1       Completed          0          4h        10.42.18.37     ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-aai-aai-695546c99-l4vd4                                   1/1       Running            0          4h        10.42.61.157    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-aai-aai-babel-c489c944c-l6fxm                             2/2       Running            0          4h        10.42.151.176   ip-172-31-42-33.us-east-2.compute.internal    <none>
onap          onap-aai-aai-cassandra-0                                       1/1       Running            0          4h        10.42.60.200    ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-aai-aai-cassandra-1                                       1/1       Running            1          4h        10.42.102.208   ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-aai-aai-cassandra-2                                       1/1       Running            0          4h        10.42.168.115   ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-aai-aai-champ-54f7986b6b-6qqk8                            2/2       Running            0          4h        10.42.233.99    ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-aai-aai-data-router-5b777d798c-hkt9p                      2/2       Running            0          4h        10.42.254.152   ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-aai-aai-elasticsearch-c9bf9dbf6-flljw                     1/1       Running            0          4h        10.42.136.54    ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-aai-aai-gizmo-5f8bf54f6f-4sj45                            2/2       Running            0          4h        10.42.128.212   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-aai-aai-graphadmin-789c67bb8b-kd9kw                       2/2       Running            0          4h        10.42.196.56    ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-aai-aai-graphadmin-create-db-schema-26pfs                 0/1       Completed          0          4h        10.42.14.14     ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-aai-aai-modelloader-695fd4bf48-5gdmt                      2/2       Running            0          4h        10.42.188.20    ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-aai-aai-resources-765d6f6678-xg4jv                        2/2       Running            0          4h        10.42.224.167   ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-aai-aai-search-data-559b8dbc7f-k549g                      2/2       Running            0          4h        10.42.110.212   ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-aai-aai-sparky-be-75658695f5-wjkzw                        2/2       Running            0          4h        10.42.7.139     ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-aai-aai-spike-f854cf86d-fdx7j                             2/2       Running            0          4h        10.42.175.92    ip-172-31-42-33.us-east-2.compute.internal    <none>
onap          onap-aai-aai-traversal-5fcc7cc847-mrqtb                        2/2       Running            0          4h        10.42.174.143   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-aai-aai-traversal-update-query-data-qlk7w                 0/1       Completed          0          4h        10.42.88.122    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-appc-appc-0                                               2/2       Running            0          4h        10.42.106.203   ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-appc-appc-ansible-server-78784978b6-jrz2b                 1/1       Running            0          4h        10.42.232.15    ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-appc-appc-cdt-d654974f6-bxhkw                             1/1       Running            0          4h        10.42.170.154   ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-appc-appc-db-0                                            1/1       Running            0          4h        10.42.215.183   ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-appc-appc-db-1                                            1/1       Running            1          4h        10.42.113.105   ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-appc-appc-db-2                                            1/1       Running            1          4h        10.42.245.153   ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-appc-appc-dgbuilder-6bdc7bd56b-ssb2p                      1/1       Running            0          4h        10.42.188.237   ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-clamp-clamp-8cd95c6b4-kbmxg                               2/2       Running            0          4h        10.42.149.44    ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-clamp-clamp-dash-es-5566796d7c-b2mbd                      1/1       Running            0          4h        10.42.51.26     ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-clamp-clamp-dash-kibana-646fccf75d-vf68k                  1/1       Running            0          4h        10.42.173.69    ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-clamp-clamp-dash-logstash-7bfc956564-xhlj5                1/1       Running            0          4h        10.42.38.41     ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-clamp-clampdb-6797447d-bkrbr                              1/1       Running            0          4h        10.42.37.63     ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-cli-cli-5498964cf9-ndk74                                  1/1       Running            0          4h        10.42.251.86    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-consul-consul-849447d678-8677q                            1/1       Running            0          4h        10.42.56.217    ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-consul-consul-server-0                                    1/1       Running            0          4h        10.42.69.75     ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-consul-consul-server-1                                    1/1       Running            0          4h        10.42.156.89    ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-consul-consul-server-2                                    1/1       Running            0          4h        10.42.69.41     ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-contrib-netbox-app-6d4d768d9d-gksvt                       1/1       Running            0          4h        10.42.179.105   ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-contrib-netbox-app-provisioning-gmmvj                     0/1       Completed          0          4h        10.42.111.99    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-contrib-netbox-app-provisioning-n6fw4                     0/1       Error              0          4h        10.42.21.12     ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-contrib-netbox-app-provisioning-nc8ww                     0/1       Error              0          4h        10.42.109.156   ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-contrib-netbox-app-provisioning-xcxds                     0/1       Error              0          4h        10.42.152.223   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-contrib-netbox-nginx-54c979ff7f-nk5kk                     1/1       Running            0          4h        10.42.244.179   ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-contrib-netbox-postgres-784b5d5fd9-2wdf7                  1/1       Running            0          4h        10.42.175.55    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-dcaegen2-dcae-bootstrap-7bb4b6bc57-gvz47                  1/1       Running            0          4h        10.42.74.208    ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-dcaegen2-dcae-cloudify-manager-9cb7f4c66-7h77h            1/1       Running            0          4h        10.42.250.189   ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-dcaegen2-dcae-db-0                                        1/1       Running            0          4h        10.42.230.118   ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-dcaegen2-dcae-db-1                                        1/1       Running            0          4h        10.42.29.230    ip-172-31-37-24.us-east-2.compute.internal    <none>
onap          onap-dcaegen2-dcae-healthcheck-7b664b566c-w5w4h                1/1       Running            0          4h        10.42.9.151     ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-dcaegen2-dcae-pgpool-688d65b66d-jpv8x                     1/1       Running            0          4h        10.42.177.56    ip-172-31-42-33.us-east-2.compute.internal    <none>
onap          onap-dcaegen2-dcae-pgpool-688d65b66d-srm5q                     1/1       Running            0          4h        10.42.164.252   ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-dcaegen2-dcae-redis-0                                     1/1       Running            0          4h        10.42.32.17     ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-dcaegen2-dcae-redis-1                                     1/1       Running            0          4h        10.42.52.99     ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-dcaegen2-dcae-redis-2                                     1/1       Running            0          4h        10.42.45.144    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-dcaegen2-dcae-redis-3                                     1/1       Running            0          4h        10.42.79.151    ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-dcaegen2-dcae-redis-4                                     1/1       Running            0          4h        10.42.110.141   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-dcaegen2-dcae-redis-5                                     1/1       Running            0          4h        10.42.253.134   ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-dmaap-dbc-pg-0                                            1/1       Running            0          4h        10.42.166.51    ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-dmaap-dbc-pg-1                                            1/1       Running            0          4h        10.42.230.243   ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-dmaap-dbc-pgpool-c5f8498-lr854                            1/1       Running            0          4h        10.42.35.8      ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-dmaap-dbc-pgpool-c5f8498-m84f5                            1/1       Running            0          4h        10.42.240.60    ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-dmaap-dmaap-bus-controller-59c96d6b8f-nlc4t               1/1       Running            0          4h        10.42.240.232   ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-dmaap-dmaap-dr-db-557c66dc9d-4ltgp                        1/1       Running            1          4h        10.42.210.116   ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-dmaap-dmaap-dr-node-6496d8f55b-jfvrm                      0/1       Init:0/1           28         4h        10.42.95.32     ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-dmaap-dmaap-dr-prov-86f79c47f9-tldsp                      0/1       CrashLoopBackOff   59         4h        10.42.76.248    ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-dmaap-message-router-557cb765-sc4pb                       1/1       Running            0          4h        10.42.157.140   ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-dmaap-message-router-kafka-7b4d45b44c-7g5pz               1/1       Running            0          4h        10.42.218.166   ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-dmaap-message-router-zookeeper-5bc7588665-pz9x5           1/1       Running            0          4h        10.42.129.130   ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-esr-esr-gui-6c5ccd59d6-xzh46                              1/1       Running            0          4h        10.42.89.47     ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-esr-esr-server-5f967d4767-nl5xc                           2/2       Running            0          4h        10.42.146.185   ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-log-log-elasticsearch-7c5b74569f-g64hw                    1/1       Running            0          4h        10.42.54.225    ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-log-log-kibana-84b875cc6b-gdzp8                           1/1       Running            0          4h        10.42.56.186    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-log-log-logstash-7hmrz                                    1/1       Running            0          4h        10.42.200.38    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-log-log-logstash-8cb9c                                    1/1       Running            0          4h        10.42.162.152   ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-log-log-logstash-8pqrp                                    1/1       Running            0          4h        10.42.97.239    ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-log-log-logstash-bvq5r                                    1/1       Running            0          4h        10.42.241.246   ip-172-31-42-33.us-east-2.compute.internal    <none>
onap          onap-log-log-logstash-cckqk                                    1/1       Running            0          4h        10.42.222.107   ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-log-log-logstash-gq826                                    1/1       Running            0          4h        10.42.129.134   ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-log-log-logstash-kzvbv                                    1/1       Running            0          4h        10.42.195.222   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-log-log-logstash-mdbtv                                    1/1       Running            0          4h        10.42.54.127    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-log-log-logstash-n78q5                                    1/1       Running            0          4h        10.42.22.101    ip-172-31-37-24.us-east-2.compute.internal    <none>
onap          onap-log-log-logstash-ns76c                                    1/1       Running            0          4h        10.42.124.239   ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-log-log-logstash-pvsnw                                    1/1       Running            0          4h        10.42.164.130   ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-log-log-logstash-q86z9                                    1/1       Running            0          4h        10.42.73.74     ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-log-log-logstash-skr52                                    1/1       Running            0          4h        10.42.0.150     ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-log-log-logstash-zjkph                                    1/1       Running            0          4h        10.42.47.5      ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-msb-kube2msb-5c79ddd89f-4m524                             1/1       Running            0          4h        10.42.86.56     ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-msb-msb-consul-6949bd46f4-gsm59                           1/1       Running            0          4h        10.42.19.136    ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-msb-msb-discovery-86c7b945f9-ffxtx                        2/2       Running            0          4h        10.42.27.240    ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-msb-msb-eag-5f86f89c4f-4pr2t                              2/2       Running            0          4h        10.42.36.35     ip-172-31-42-33.us-east-2.compute.internal    <none>
onap          onap-msb-msb-iag-56cdd4c87b-4xtz7                              2/2       Running            0          4h        10.42.152.29    ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-multicloud-multicloud-867df449f8-gwkdw                    2/2       Running            0          4h        10.42.87.4      ip-172-31-37-24.us-east-2.compute.internal    <none>
onap          onap-multicloud-multicloud-azure-5db4ddd44d-cpphs              2/2       Running            0          4h        10.42.84.246    ip-172-31-37-24.us-east-2.compute.internal    <none>
onap          onap-multicloud-multicloud-ocata-c557948df-5sgfs               2/2       Running            0          4h        10.42.39.16     ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-multicloud-multicloud-pike-8ff99dfb6-zmrw8                2/2       Running            0          4h        10.42.176.68    ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-multicloud-multicloud-vio-866cd4bf9b-g2zlx                2/2       Running            0          4h        10.42.64.228    ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-multicloud-multicloud-windriver-7775ddc4b9-4278w          2/2       Running            0          4h        10.42.87.19     ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-nbi-nbi-7595d4dbc-9m95t                                   1/1       Running            1          4h        10.42.94.19     ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-nbi-nbi-mariadb-65f556b8b7-fgjzx                          1/1       Running            0          4h        10.42.195.127   ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-nbi-nbi-mongo-0                                           1/1       Running            0          4h        10.42.184.20    ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-oof-cmso-db-0                                             1/1       Running            0          4h        10.42.193.70    ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-oof-music-cassandra-0                                     1/1       Running            1          4h        10.42.189.19    ip-172-31-42-33.us-east-2.compute.internal    <none>
onap          onap-oof-music-cassandra-1                                     1/1       Running            0          4h        10.42.163.171   ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-oof-music-cassandra-2                                     1/1       Running            0          4h        10.42.195.0     ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-oof-music-cassandra-job-config-7mb5f                      0/1       Completed          0          4h        10.42.38.249    ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-oof-music-tomcat-7974f8748c-4pgg8                         1/1       Running            0          4h        10.42.118.22    ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-oof-music-tomcat-7974f8748c-dxdds                         1/1       Running            0          4h        10.42.159.58    ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-oof-music-tomcat-7974f8748c-jh7jg                         1/1       Running            0          4h        10.42.64.7      ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-oof-oof-6dbf897c86-8d6bx                                  1/1       Running            0          4h        10.42.20.170    ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-oof-oof-cmso-service-7d54b6b478-djvdm                     1/1       Running            0          4h        10.42.192.55    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-oof-oof-has-api-774f9458cb-mtfg8                          1/1       Running            0          4h        10.42.208.73    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-oof-oof-has-controller-7998c7f7b6-vg6nz                   1/1       Running            0          4h        10.42.91.250    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-oof-oof-has-data-5dffb4c7df-qmjzz                         1/1       Running            0          4h        10.42.213.36    ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-oof-oof-has-healthcheck-rpst7                             0/1       Completed          0          4h        10.42.241.223   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-oof-oof-has-onboard-5bd2l                                 0/1       Completed          0          4h        10.42.205.75    ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-oof-oof-has-reservation-59759db679-dz4cs                  1/1       Running            0          4h        10.42.48.244    ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-oof-oof-has-solver-5bfc76f46-hfc88                        1/1       Running            0          4h        10.42.225.129   ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-oof-zookeeper-0                                           1/1       Running            0          4h        10.42.171.61    ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-oof-zookeeper-1                                           1/1       Running            0          4h        10.42.251.101   ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-oof-zookeeper-2                                           1/1       Running            0          4h        10.42.65.144    ip-172-31-37-24.us-east-2.compute.internal    <none>
onap          onap-policy-brmsgw-86dd8fc4f7-xtzk9                            1/1       Running            0          4h        10.42.56.63     ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-policy-drools-0                                           1/1       Running            0          4h        10.42.75.29     ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-policy-nexus-b7f484bdb-gmn76                              1/1       Running            0          4h        10.42.45.249    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-policy-pap-6555659676-77nl7                               2/2       Running            0          4h        10.42.87.216    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-policy-pdp-0                                              2/2       Running            0          4h        10.42.63.230    ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-policy-policy-apex-pdp-0                                  1/1       Running            0          4h        10.42.73.87     ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-policy-policy-distribution-79544b78c7-hk5mk               1/1       Running            0          4h        10.42.104.145   ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-policy-policydb-79cdf7cb7c-jb82c                          1/1       Running            0          4h        10.42.98.252    ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-pomba-pomba-aaictxbuilder-684d5977cd-sj8dm                2/2       Running            0          4h        10.42.170.23    ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-pomba-pomba-contextaggregator-779745878d-4s8c9            1/1       Running            0          4h        10.42.154.6     ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-pomba-pomba-data-router-59bb4fd899-dc5zc                  1/1       Running            0          4h        10.42.171.88    ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-pomba-pomba-elasticsearch-7b8bc5f864-5mjqk                1/1       Running            0          4h        10.42.111.181   ip-172-31-33-223.us-east-2.compute.internal   <none>

onap          onap-pomba-pomba-kibana-6dd8dbb68b-xltcm                       1/1       Running            0          4h        10.42.181.201   ip-172-31-45-152.us-east-2.compute.internal   <none>

onap          onap-pomba-pomba-networkdiscovery-d47f68c88-j2zw4              2/2       Running            0          4h        10.42.34.101    ip-172-31-41-30.us-east-2.compute.internal    <none>

onap          onap-pomba-pomba-networkdiscoveryctxbuilder-6978dc6c79-6z7zm   2/2       Running            0          4h        10.42.116.177   ip-172-31-41-229.us-east-2.compute.internal   <none>

onap          onap-pomba-pomba-sdcctxbuilder-fcbc7f96b-nj2c9                 1/1       Running            0          4h        10.42.87.59     ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-pomba-pomba-search-data-5b4d8f7dc6-k7qlx                  2/2       Running            0          4h        10.42.37.226    ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-pomba-pomba-servicedecomposition-99b445978-6crkz          2/2       Running            0          4h        10.42.46.61     ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-pomba-pomba-validation-service-5c8454484-t4kls            1/1       Running            0          4h        10.42.235.206   ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-portal-portal-app-67bc49d88-tpxbq                         2/2       Running            0          4h        10.42.25.53     ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-portal-portal-cassandra-75d4746d76-jn78x                  1/1       Running            0          4h        10.42.128.240   ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-portal-portal-db-7b768ffcc-92q8b                          1/1       Running            1          4h        10.42.187.56    ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-portal-portal-db-config-qshzn                             0/2       Completed          0          4h        10.42.112.46    ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-portal-portal-db-config-rk4m2                             0/2       Init:Error         0          4h        10.42.57.79     ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-portal-portal-sdk-59fbd4bd9d-6j9x9                        2/2       Running            0          4h        10.42.217.29    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-portal-portal-widget-79587fdf56-5dh2c                     1/1       Running            0          4h        10.42.12.249    ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-portal-portal-zookeeper-7c9444bfb5-9bvwq                  1/1       Running            0          4h        10.42.113.205   ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-robot-robot-86c5f986b7-kd257                              1/1       Running            0          4h        10.42.246.138   ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-be-7fdc576bfb-6q5bx                               2/2       Running            0          4h        10.42.142.158   ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-be-config-backend-2vw2q                           0/1       Completed          0          4h        10.42.87.181    ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-be-config-backend-k57lh                           0/1       Init:Error         0          4h        10.42.148.79    ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-cs-5b9bb5756d-28zhd                               1/1       Running            1          4h        10.42.232.24    ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-sdc-sdc-cs-config-cassandra-vgnz2                         0/1       Completed          0          4h        10.42.111.187   ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-es-8bdf86659-bvgj6                                1/1       Running            0          4h        10.42.21.223    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-es-config-elasticsearch-lkb9m                     0/1       Completed          0          4h        10.42.20.202    ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-fe-65c95cf8bf-2944f                               2/2       Running            0          4h        10.42.193.193   ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-kb-79bfffc4cf-6p5qn                               1/1       Running            0          4h        10.42.59.138    ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-onboarding-be-84466885d5-pwqpn                    2/2       Running            0          4h        10.42.163.142   ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-onboarding-be-cassandra-init-7zv5j                0/1       Completed          0          4h        10.42.218.1     ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-wfd-be-54f49cb9f-8wmkz                            1/1       Running            0          4h        10.42.69.253    ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-sdc-sdc-wfd-be-workflow-init-q8t7z                        0/1       Completed          0          4h        10.42.255.91    ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-sdc-sdc-wfd-fe-585bd84f8d-grwwh                           2/2       Running            0          4h        10.42.110.244   ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-sdnc-controller-blueprints-7c8f9dbfb7-tq4f5               1/1       Running            0          4h        10.42.200.245   ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-sdnc-controller-blueprints-db-0                           1/1       Running            0          4h        10.42.207.149   ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-sdnc-nengdb-0                                             1/1       Running            0          4h        10.42.211.255   ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-sdnc-network-name-gen-f4ff8b99f-8bxbh                     1/1       Running            0          4h        10.42.107.56    ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-sdnc-sdnc-0                                               2/2       Running            0          4h        10.42.220.124   ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-sdnc-sdnc-ansible-server-dffcf4c8b-jdpmh                  1/1       Running            0          4h        10.42.84.202    ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-sdnc-sdnc-db-0                                            2/2       Running            0          4h        10.42.45.121    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-sdnc-sdnc-dgbuilder-5496ff5f86-5l27s                      1/1       Running            0          4h        10.42.17.254    ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-sdnc-sdnc-dmaap-listener-54f5c69c7c-bxp2n                 1/1       Running            0          4h        10.42.42.177    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-sdnc-sdnc-portal-8ff4bb696-j9xz5                          1/1       Running            0          4h        10.42.250.3     ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-sdnc-sdnc-ueb-listener-5d5749f879-2pw5j                   1/1       Running            0          4h        10.42.93.48     ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-sniro-emulator-sniro-emulator-5bdc8cc4cb-qjbk8            1/1       Running            0          4h        10.42.207.251   ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-so-so-7f5f98949b-w85kc                                    1/1       Running            0          4h        10.42.110.129   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-so-so-bpmn-infra-74fcf888b4-9tnfc                         1/1       Running            0          4h        10.42.60.90     ip-172-31-37-24.us-east-2.compute.internal    <none>
onap          onap-so-so-catalog-db-adapter-5b8cb47dc-j44bt                  1/1       Running            0          4h        10.42.222.142   ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-so-so-mariadb-5b5bd695c4-jdx8v                            1/1       Running            0          4h        10.42.249.209   ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-so-so-monitoring-855456948b-fr24q                         1/1       Running            0          4h        10.42.14.54     ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-so-so-openstack-adapter-c44cd98dc-pk2gb                   1/1       Running            1          4h        10.42.25.240    ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-so-so-request-db-adapter-745dc6677-z7tcx                  1/1       Running            0          4h        10.42.186.49    ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-so-so-sdc-controller-67f8cbbc7c-kdfmr                     1/1       Running            0          4h        10.42.23.215    ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-so-so-sdnc-adapter-5cbc5f8dd8-6k22h                       1/1       Running            0          4h        10.42.14.50     ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-so-so-vfc-adapter-6c5996d748-5h2jj                        1/1       Running            0          4h        10.42.205.174   ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-uui-uui-d5ff69677-xkmq2                                   1/1       Running            0          4h        10.42.157.9     ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-uui-uui-server-7cc8bcb956-c28g5                           1/1       Running            0          4h        10.42.245.99    ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-vfc-vfc-catalog-66dd84cc97-htgsr                          2/2       Running            0          4h        10.42.248.74    ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-vfc-vfc-db-65d8fd5567-25z45                               1/1       Running            0          4h        10.42.181.50    ip-172-31-37-24.us-east-2.compute.internal    <none>
onap          onap-vfc-vfc-ems-driver-6f5fd887db-hrh7q                       1/1       Running            0          4h        10.42.28.44     ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-vfc-vfc-generic-vnfm-driver-7964b7f4cc-nd2d2              2/2       Running            1          4h        10.42.172.200   ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-vfc-vfc-huawei-vnfm-driver-6c5676758d-vs8fh               2/2       Running            0          4h        10.42.58.24     ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-vfc-vfc-juju-vnfm-driver-67b9c4bb4c-fhv45                 2/2       Running            0          4h        10.42.10.58     ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-vfc-vfc-multivim-proxy-5f6f455bfd-tg7m2                   1/1       Running            0          4h        10.42.51.118    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-vfc-vfc-nokia-v2vnfm-driver-6f7646f69d-22p88              1/1       Running            0          4h        10.42.158.31    ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-vfc-vfc-nokia-vnfm-driver-7c9d59597f-gkmct                2/2       Running            0          4h        10.42.184.15    ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-vfc-vfc-nslcm-c8478dcc8-cnlvd                             2/2       Running            0          4h        10.42.144.183   ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-vfc-vfc-resmgr-6d7d7bc78b-f5v88                           2/2       Running            0          4h        10.42.255.238   ip-172-31-37-24.us-east-2.compute.internal    <none>
onap          onap-vfc-vfc-vnflcm-fb656d967-p6wh2                            2/2       Running            0          4h        10.42.56.73     ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-vfc-vfc-vnfmgr-66d5cd85f7-ld4v4                           2/2       Running            0          4h        10.42.121.241   ip-172-31-34-207.us-east-2.compute.internal   <none>
onap          onap-vfc-vfc-vnfres-7757d4b486-bkqqz                           2/2       Running            0          4h        10.42.100.120   ip-172-31-34-111.us-east-2.compute.internal   <none>
onap          onap-vfc-vfc-workflow-68f85b8fcf-mxgwl                         1/1       Running            0          4h        10.42.72.65     ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-vfc-vfc-workflow-engine-7d76bb4b56-tj6ks                  1/1       Running            0          4h        10.42.48.13     ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-vfc-vfc-zte-sdnc-driver-58d4fc6d9f-hk4d9                  1/1       Running            0          4h        10.42.245.72    ip-172-31-41-30.us-east-2.compute.internal    <none>
onap          onap-vfc-vfc-zte-vnfm-driver-fc669cc8d-64k6j                   2/2       Running            1          4h        10.42.68.226    ip-172-31-41-229.us-east-2.compute.internal   <none>
onap          onap-vid-vid-5cb464799b-fw69j                                  2/2       Running            0          4h        10.42.21.193    ip-172-31-41-110.us-east-2.compute.internal   <none>
onap          onap-vid-vid-galera-config-4f274                               0/1       Completed          0          4h        10.42.80.200    ip-172-31-33-223.us-east-2.compute.internal   <none>
onap          onap-vid-vid-mariadb-galera-0                                  1/1       Running            0          4h        10.42.40.225    ip-172-31-45-152.us-east-2.compute.internal   <none>
onap          onap-vnfsdk-vnfsdk-6b45bfffb4-phcwx                            1/1       Running            0          4h        10.42.32.180    ip-172-31-38-95.us-east-2.compute.internal    <none>
onap          onap-vnfsdk-vnfsdk-init-postgres-lf659                         0/1       Completed          0          4h        10.42.238.204   ip-172-31-38-194.us-east-2.compute.internal   <none>
onap          onap-vnfsdk-vnfsdk-pgpool-6874f7b695-69njk                     1/1       Running            0          4h        10.42.239.14    ip-172-31-36-163.us-east-2.compute.internal   <none>
onap          onap-vnfsdk-vnfsdk-pgpool-6874f7b695-pjmwp                     1/1       Running            0          4h        10.42.171.177   ip-172-31-39-138.us-east-2.compute.internal   <none>
onap          onap-vnfsdk-vnfsdk-postgres-0                                  1/1       Running            0          4h        10.42.106.183   ip-172-31-37-59.us-east-2.compute.internal    <none>
onap          onap-vnfsdk-vnfsdk-postgres-1                                  1/1       Running            0          4h        10.42.47.66     ip-172-31-41-229.us-east-2.compute.internal   <none>
ubuntu@ip-172-31-40-250:~$ kubectl get nodes -o wide
NAME                                          STATUS    ROLES     AGE       VERSION            INTERNAL-IP      EXTERNAL-IP      OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
ip-172-31-33-223.us-east-2.compute.internal   Ready     <none>    5h        v1.11.2-rancher1   18.222.148.116   18.222.148.116   Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-34-111.us-east-2.compute.internal   Ready     <none>    5h        v1.11.2-rancher1   3.16.37.170      3.16.37.170      Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-34-207.us-east-2.compute.internal   Ready     <none>    5h        v1.11.2-rancher1   18.225.32.201    18.225.32.201    Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-36-163.us-east-2.compute.internal   Ready     <none>    5h        v1.11.2-rancher1   13.58.189.251    13.58.189.251    Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-37-24.us-east-2.compute.internal    Ready     <none>    5h        v1.11.2-rancher1   18.224.180.26    18.224.180.26    Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-37-59.us-east-2.compute.internal    Ready     <none>    5h        v1.11.2-rancher1   18.191.248.14    18.191.248.14    Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-38-194.us-east-2.compute.internal   Ready     <none>    4h        v1.11.2-rancher1   18.217.45.91     18.217.45.91     Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-38-95.us-east-2.compute.internal    Ready     <none>    4h        v1.11.2-rancher1   52.15.39.21      52.15.39.21      Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-39-138.us-east-2.compute.internal   Ready     <none>    4h        v1.11.2-rancher1   18.224.199.40    18.224.199.40    Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-41-110.us-east-2.compute.internal   Ready     <none>    4h        v1.11.2-rancher1   18.223.151.180   18.223.151.180   Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-41-229.us-east-2.compute.internal   Ready     <none>    5h        v1.11.2-rancher1   18.218.252.13    18.218.252.13    Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-41-30.us-east-2.compute.internal    Ready     <none>    4h        v1.11.2-rancher1   3.16.113.3       3.16.113.3       Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-42-33.us-east-2.compute.internal    Ready     <none>    5h        v1.11.2-rancher1   13.59.2.86       13.59.2.86       Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ip-172-31-45-152.us-east-2.compute.internal   Ready     <none>    4h        v1.11.2-rancher1   18.219.56.50     18.219.56.50     Ubuntu 16.04.1 LTS   4.4.0-1049-aws   docker://17.3.2
ubuntu@ip-172-31-40-250:~$ kubectl top nodes
NAME                                          CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
ip-172-31-33-223.us-east-2.compute.internal   852m         10%       13923Mi         90%       
ip-172-31-34-111.us-east-2.compute.internal   1160m        14%       11643Mi         75%       
ip-172-31-34-207.us-east-2.compute.internal   1101m        13%       7981Mi          51%      
ip-172-31-36-163.us-east-2.compute.internal   656m         8%        13377Mi         87%       
ip-172-31-37-24.us-east-2.compute.internal    401m         5%        8543Mi          55%       
ip-172-31-37-59.us-east-2.compute.internal    711m         8%        10873Mi         70%       
ip-172-31-38-194.us-east-2.compute.internal   1136m        14%       8195Mi          53%       
ip-172-31-38-95.us-east-2.compute.internal    1195m        14%       9127Mi          59%       
ip-172-31-39-138.us-east-2.compute.internal   296m         3%        10870Mi         70%       
ip-172-31-41-110.us-east-2.compute.internal   2586m        32%       10950Mi         71%       
ip-172-31-41-229.us-east-2.compute.internal   159m         1%        9138Mi          59%       
ip-172-31-41-30.us-east-2.compute.internal    180m         2%        9862Mi          64%       
ip-172-31-42-33.us-east-2.compute.internal    1573m        19%       6352Mi          41%       
ip-172-31-45-152.us-east-2.compute.internal   1579m        19%       10633Mi         69%  
ubuntu@ip-172-31-40-250:~/oom/kubernetes/robot$ sudo ./ete-k8s.sh onap health
./ete-k8s.sh: 22: [: onap: unexpected operator
./ete-k8s.sh: 22: [: health: unexpected operator
+ export NAMESPACE=onap
Testsuites                                                                    
==============================================================================
Testsuites.Health-Check :: Testing ecomp components are available via calls.  
==============================================================================
Basic A&AI Health Check                                               | PASS |
------------------------------------------------------------------------------
Basic AAF Health Check                                                | PASS |
------------------------------------------------------------------------------
Basic AAF SMS Health Check                                            | PASS |
------------------------------------------------------------------------------
Basic APPC Health Check                                               | PASS |
------------------------------------------------------------------------------
Basic CLI Health Check                                                | PASS |
------------------------------------------------------------------------------
Basic CLAMP Health Check                                              | PASS |
------------------------------------------------------------------------------
Basic DCAE Health Check                                               | PASS |
------------------------------------------------------------------------------
Basic DMAAP Data Router Health Check                                  [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f536960ea50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv
[ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f536972e450>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv
[ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5369617090>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv
| FAIL |
ConnectionError: HTTPConnectionPool(host='dmaap-dr-node.onap', port=8080): Max retries exceeded with url: /internal/fetchProv (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5369d21850>: Failed to establish a new connection: [Errno 111] Connection refused',))
------------------------------------------------------------------------------
Basic DMAAP Message Router Health Check                               | PASS |
------------------------------------------------------------------------------
Basic External API NBI Health Check                                   | PASS |
------------------------------------------------------------------------------
Basic Log Elasticsearch Health Check                                  | PASS |
------------------------------------------------------------------------------
Basic Log Kibana Health Check                                         | PASS |
------------------------------------------------------------------------------
Basic Log Logstash Health Check                                       | PASS |
------------------------------------------------------------------------------
Basic Microservice Bus Health Check                                   | PASS |
------------------------------------------------------------------------------
Basic Multicloud API Health Check                                     | PASS |
------------------------------------------------------------------------------
Basic Multicloud-ocata API Health Check                               | PASS |
------------------------------------------------------------------------------
Basic Multicloud-pike API Health Check                                | PASS |
------------------------------------------------------------------------------
Basic Multicloud-titanium_cloud API Health Check                      | PASS |
------------------------------------------------------------------------------
Basic Multicloud-vio API Health Check                                 | PASS |
------------------------------------------------------------------------------
Basic OOF-Homing Health Check                                         | PASS |
------------------------------------------------------------------------------
Basic OOF-SNIRO Health Check                                          | PASS |
------------------------------------------------------------------------------
Basic OOF-CMSO Health Check                                           | PASS |
------------------------------------------------------------------------------
Basic Policy Health Check                                             | PASS |
------------------------------------------------------------------------------
Basic Pomba AAI-context-builder Health Check                          | PASS |
------------------------------------------------------------------------------
Basic Pomba SDC-context-builder Health Check                          | PASS |
------------------------------------------------------------------------------
Basic Pomba Network-discovery-context-builder Health Check            | PASS |
------------------------------------------------------------------------------
Basic Portal Health Check                                             | PASS |
------------------------------------------------------------------------------
Basic SDC Health Check                                                (DMaaP:UP)| PASS |
------------------------------------------------------------------------------
Basic SDNC Health Check                                               | PASS |
------------------------------------------------------------------------------
Basic SO Health Check                                                 | PASS |
------------------------------------------------------------------------------
Basic UseCaseUI API Health Check                                      | PASS |
------------------------------------------------------------------------------
Basic VFC catalog API Health Check                                    | PASS |
------------------------------------------------------------------------------
Basic VFC emsdriver API Health Check                                  | PASS |
------------------------------------------------------------------------------
Basic VFC gvnfmdriver API Health Check                                | PASS |
------------------------------------------------------------------------------
Basic VFC huaweivnfmdriver API Health Check                           | PASS |
------------------------------------------------------------------------------
Basic VFC jujuvnfmdriver API Health Check                             | PASS |
------------------------------------------------------------------------------
Basic VFC multivimproxy API Health Check                              | PASS |
------------------------------------------------------------------------------
Basic VFC nokiavnfmdriver API Health Check                            | PASS |
------------------------------------------------------------------------------
Basic VFC nokiav2driver API Health Check                              | PASS |
------------------------------------------------------------------------------
Basic VFC nslcm API Health Check                                      | PASS |
------------------------------------------------------------------------------
Basic VFC resmgr API Health Check                                     | PASS |
------------------------------------------------------------------------------
Basic VFC vnflcm API Health Check                                     | PASS |
------------------------------------------------------------------------------
Basic VFC vnfmgr API Health Check                                     | PASS |
------------------------------------------------------------------------------
Basic VFC vnfres API Health Check                                     | PASS |
------------------------------------------------------------------------------
Basic VFC workflow API Health Check                                   | PASS |
------------------------------------------------------------------------------
Basic VFC ztesdncdriver API Health Check                              | PASS |
------------------------------------------------------------------------------
Basic VFC ztevnfmdriver API Health Check                              | PASS |
------------------------------------------------------------------------------
Basic VID Health Check                                                | PASS |
------------------------------------------------------------------------------
Basic VNFSDK Health Check                                             | PASS |
------------------------------------------------------------------------------
Basic Holmes Rule Management API Health Check                         | FAIL |
502 != 200
------------------------------------------------------------------------------
Basic Holmes Engine Management API Health Check                       | FAIL |
502 != 200
------------------------------------------------------------------------------
Testsuites.Health-Check :: Testing ecomp components are available ... | FAIL |
51 critical tests, 48 passed, 3 failed
51 tests total, 48 passed, 3 failed

ONAP Module Dependencies - Compile and Runtime

Existing Library Research

Portal/SDK

SO

AAI

Library Selected for Reference Implementation

Java

Python

Specification

Use Cases

Design Issues

Developer Guide

Testing Guide

...