DCAE Services - Logging Evaluation

Owned by Vijay Kumar
Oct 29, 2021
13 min read


Note: As with Jakarata release, all DCAE Platform components (Cloudify, Associated Handlers - PH, SCH, InventoryAPI, Dashboard, CBS) will be deprecated - below log reporting is focussed only for DCAE Services component.


DCAE ComponentsTechnologyPODNameSTDIO complaint?Logger libraryLogging TypeLogs Location Example (from STDOUT)
VESCollectorjavadev-dcae-ves-collector-XXYEScommons-logging-1.1.1.jar  log4j-api-2.13.3.jar  log4j-core-2.13.3.jar  log4j-jul-2.13.3.jar  log4j-slf4j-impl-2.13.3.jar  spring-boot-starter-log4j2-2.4.3.jarEELF + Custom + STDOUT
vescollector@dcae-ves-collector:/opt/app/VESCollector/logs$ find .
.
./input.log
./collector.log
./error.log
./eelf
./eelf/error.log
./eelf/audit.log
./eelf/metric.log
./eelf/debug.log
./output.log		2021-09-15 19:37:08.126  INFO 25 [or-http-epoll-4] o.o.d.c.ConfigParsing                    : Found following DMaaP configuration: '{"ves-pnfRegistration":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.VES_PNFREG_OUTPUT"},"type":"message_router"},"ves-3gpp-fault-supervision":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.SEC_3GPP_FAULTSUPERVISION_OUTPUT"},"type":"message_router"},"ves-notification":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.VES_NOTIFICATION_OUTPUT"},"type":"message_router"},"ves-3gpp-heartbeat":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT"},"type":"message_router"},"ves-3gpp-performance-assurance":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.SEC_3GPP_PERFORMANCEASSURANCE_OUTPUT"},"type":"message_router"},"ves-3gpp-provisioning":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.SEC_3GPP_PROVISIONING_OUTPUT"},"type":"message_router"},"ves-fault":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.SEC_FAULT_OUTPUT"},"type":"message_router"},"ves-heartbeat":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.SEC_HEARTBEAT_OUTPUT"},"type":"message_router"},"ves-other":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.SEC_OTHER_OUTPUT"},"type":"message_router"},"ves-measurement":{"dmaap_info":{"topic_url":"http://message-router:3904/events/unauthenticated.VES_MEASUREMENT_OUTPUT"},"type":"message_router"}}'
2021-09-15 19:42:07.855  INFO 25 [or-http-epoll-4] o.o.d.s.s.r.s.c.c.i.CbsClientImpl        : Got successful response from Config Binding Service
2021-09-15 19:47:07.852  INFO 25 [or-http-epoll-4] o.o.d.s.s.r.s.c.c.i.CbsClientImpl        : Got successful response from Config Binding Service
2021-09-15 19:52:07.851  INFO 25 [or-http-epoll-4] o.o.d.s.s.r.s.c.c.i.CbsClientImpl        : Got successful response from Config Binding Service
TCAgen2javadev-dcae-tcagen2-XXYEScustom EELF logger lib uses logbackEELF + STDOUT/opt/logs/dcae-analytics-tca $ ls -lrt *.log
-rw-r--r--    1 tca-gen2 tca-gen2         0 Sep 15 19:36 error.log
-rw-r--r--    1 tca-gen2 tca-gen2         0 Sep 15 19:36 audit.log
-rw-r--r--    1 tca-gen2 tca-gen2    236230 Oct  1 18:16 metrics.log
-rw-r--r--    1 tca-gen2 tca-gen2   2329766 Oct  1 18:16 debug.log		2021-09-15 19:51:42.551 DEBUG 1 --- [     parallel-1] o.o.d.s.s.r.s.c.c.impl.CbsClientImpl     : Calling https://config-binding-service:10443/service_component_all/dcae-tcagen2 for configuration
2021-09-15 19:51:42.552 DEBUG 1 --- [or-http-epoll-2] r.n.r.DefaultPooledConnectionProvider    : [id:3d81733d, L:/10.42.0.57:58002 - R:config-binding-service.onap.svc.cluster.local/10.43.242.254:10443] Channel acquired, now: 1 active connections, 0 inactive connections and 0 pending acquire requests.
2021-09-15 19:51:42.552 DEBUG 1 --- [or-http-epoll-2] r.netty.http.client.HttpClientConnect    : [id:3d81733d-15, L:/10.42.0.57:58002 - R:config-binding-service.onap.svc.cluster.local/10.43.242.254:10443] Handler is being applied: {uri=https://config-binding-service:10443/service_component_all/dcae-tcagen2, method=GET}
2021-09-15 19:51:42.552 DEBUG 1 --- [or-http-epoll-2] r.n.r.DefaultPooledConnectionProvider    : [id:3d81733d-15, L:/10.42.0.57:58002 - R:config-binding-service.onap.svc.cluster.local/10.43.242.254:10443] onStateChange(GET{uri=/service_component_all/dcae-tcagen2, connection=PooledConnection{channel=[id: 0x3d81733d, L:/10.42.0.57:58002 - R:config-binding-service.onap.svc.cluster.local/10.43.242.254:10443]}}, [request_prepared])
2021-09-15 19:51:42.552 DEBUG 1 --- [or-http-epoll-2] o.o.d.s.s.r.s.a.http.RxHttpClient        : Request: GET /service_component_all/dcae-tcagen2 DefaultHttpHeaders[X-InvocationID: d56b0bbe-0c65-4dfc-9afd-c85d252a64dd, X-RequestID: 32eedd3b-0185-428a-b3c3-7444889465c9, user-agent: ReactorNetty/1.0.5, host: config-binding-service:10443, accept: */*]
2021-09-15 19:51:42.553 DEBUG 1 --- [or-http-epoll-2] r.n.r.DefaultPooledConnectionProvider    : [id:3d81733d-15, L:/10.42.0.57:58002 - R:config-binding-service.onap.svc.cluster.local/10.43.242.254:10443] onStateChange(GET{uri=/service_component_all/dcae-tcagen2, connection=PooledConnection{channel=[id: 0x3d81733d, L:/10.42.0.57:58002 - R:config-binding-service.onap.svc.cluster.local/10.43.242.254:10443]}}, [request_sent])
2021-09-15 19:51:42.564 DEBUG 1 --- [or-http-epoll-2] r.n.http.client.HttpClientOperations     : [id:3d81733d-15, L:/10.42.0.57:58002 - R:config-binding-service.onap.svc.cluster.local/10.43.242.254:10443] Received response (auto-read:false) : [x-onap-requestid=d24f6c28-5f74-4167-b5b8-eaa61af7ddd1, Content-Type=application/json, Date=Wed, 15 Sep 2021 19:51:42 GMT, content-length=3707]
2021-09-15 19:51:42.565 DEBUG 1 --- [or-http-epoll-2] r.n.r.DefaultPooledConnectionProvider    : [id:3d81733d-15, L:/10.42.0.57:58002 - R:config-binding-service.onap.svc.cluster.local/10.43.242.254:10443] onStateChange(GET{uri=/service_component_all/dcae-tcagen2, connection=PooledConnection{channel=[id: 0x3d81733d, L:/10.42.0.57:58002 - R:config-binding-service.onap.svc.cluster.local/10.43.242.254:10443]}}, [response_received])
PRHjavadev-dcae-prh-XXYESlogback-spring.xmlSTDOUT/opt/app/prh/logs $ ls
/opt/app/prh/logs $ 		org.onap.dcaegen2.services.prh.tasks.ScheduledTasks     |2021-09-15T20:13:19.744Z     |INFO     |Init configs     |InvocationID=3c864a6b-9257-4070-8de7-c37cc11dadc4, InstanceUUID=feb59e2a-d68f-430e-b0d2-aebd01cecb0a     |     |INVOKE     |pool-3-thread-1     |
org.onap.dcaegen2.services.prh.tasks.ScheduledTasks     |2021-09-15T20:13:19.756Z     |INFO     |PRH tasks have been completed     |     |     |     |reactor-http-epoll-1     |
org.onap.dcaegen2.services.prh.tasks.ScheduledTasks     |2021-09-15T20:13:29.757Z     |INFO     |Init configs     |InvocationID=3c864a6b-9257-4070-8de7-c37cc11dadc4, InstanceUUID=c01ed037-82a2-4aeb-bffa-ce7cbaf2282e     |     |INVOKE     |pool-3-thread-1     |
org.onap.dcaegen2.services.prh.tasks.ScheduledTasks     |2021-09-15T20:13:29.769Z     |INFO     |PRH tasks have been completed     |     |     |     |reactor-http-epoll-1     |
org.onap.dcaegen2.services.prh.tasks.ScheduledTasks     |2021-09-15T20:13:39.776Z     |INFO     |Init configs     |InvocationID=3c864a6b-9257-4070-8de7-c37cc11dadc4, InstanceUUID=506f6d9c-cd4f-48d3-b1e0-ef1ddf19179b     |     |INVOKE     |pool-3-thread-1     |
org.onap.dcaegen2.services.prh.tasks.ScheduledTasks     |2021-09-15T20:13:39.782Z     |INFO     |PRH tasks have been completed     |     |     |     |reactor-http-epoll-1     |
org.onap.dcaegen2.services.prh.tasks.ScheduledTasks     |2021-09-15T20:13:49.782Z     |INFO     |Init configs     |InvocationID=3c864a6b-9257-4070-8de7-c37cc11dadc4, InstanceUUID=0d5bcc3b-e^C   |
org.onap.dcaegen2.services.prh.tasks.ScheduledTasks     |2021-09-15T20:16:30.005Z     |INFO     |PRH tasks have been completed     |     |     |     |reactor-http-epoll-1     |
HV-VESkotlindev-dcae-hv-ves-collector-XXYESlogback-classic-1.3.0-alpha4.jar  logback-core-1.3.0-alpha4.jarCustom + STDOUT (same)/var/log/ONAP/dcae-hv-ves-collector $ ls -lrt
total 12
-rw-r--r--    1 hvves    hvves         8747 Sep 15 19:36 hv-ves.log		 | 2021-09-15T19:35:56.616Z      | t.services.cbs.client.model.CbsClientConfiguration    | INFO  | CBS client will use http over TLS.    |       |       |       | main
 | 2021-09-15T19:35:56.753Z      | t.services.cbs.client.model.CbsClientConfiguration    | INFO  | Path to cert files: /etc/ves-hv/ssl/  |       |       |       | main
 | 2021-09-15T19:35:58.567Z      | ollectors.veshv.config.impl.HvVesCommandLineParser    | INFO  | Healthcheck port missing on command line, using default: 6060      |        |       |       | main
 | 2021-09-15T19:36:03.207Z      | r.core.util.internal.logging.InternalLoggerFactory    | DEBUG         | Using SLF4J as the default logging framework  |       |    |        | main
 | 2021-09-15T19:36:06.781Z      | ae.collectors.veshv.main.servers.HealthCheckServer    | INFO  | Health check server is up and listening on 0.0.0.0:6060       |    |        | InstanceID=4bfdc7a3-a319-49d0-a56a-ba0d6cabe96d, ServerFQDN=dcae-hv-ves-collector     | reactor-http-epoll-1
 | 2021-09-15T19:36:06.893Z      | ae.collectors.veshv.config.api.ConfigurationModule    | INFO  | Using base configuration file: /etc/ves-hv/configuration/base.json |        |       |       | main
 | 2021-09-15T19:36:06.990Z      | ae.collectors.veshv.config.api.ConfigurationModule    | INFO  | Successfully parsed configuration file to: PartialConfiguration(listenPort=Some(6061), idleTimeoutSec=Some(60), firstRequestDelaySec=Some(10), requestIntervalSec=Some(5), sslDisable=Some(true), keyStoreFile=None, keyStorePasswordFile=None, trustStoreFile=None, trustStorePasswordFile=None, logLevel=Some(INFO), streamPublishers=None)     |       |       |       | main
 | 2021-09-15T19:36:09.893Z      | rvices.sdk.rest.services.cbs.client.impl.CbsLookup    | INFO  | Config Binding Service address: config-binding-service:10443  |    |        |       | main
DFCjavadfc-dcae-datafile-collector-XXYESMDC + logback-spring.xmlCustom + STDOUT (same)/var/log/ONAP $ ls
application.log		org.onap.dcaegen2.services.sdk.services.common.FileReader     |2021-10-01T18:24:26.912Z     |ERROR     |Error while reading file. Filename:      |RequestID=875c9324-5161-4794-961b-49cc4a7a019c     |     |     |main     |
org.onap.dcaegen2.services.sdk.rest.services.cbs.client.impl.CbsClientConfigMap     |2021-10-01T18:24:26.923Z     |ERROR     |Error loading configuration from configMap file: Expected a com.google.gson.JsonObject but was com.google.gson.JsonNull     |RequestID=875c9324-5161-4794-961b-49cc4a7a019c     |     |     |main     |
org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks     |2021-10-01T18:24:27.014Z     |WARN     |No configuration loaded, skipping polling for messages     |     |     |     |pool-2-thread-1     |
org.onap.dcaegen2.collectors.datafile.tasks.ScheduledTasks     |2021-10-01T18:24:42.654Z     |ERROR     |Polling for file ready message failed, exception: java.lang.IllegalStateException: 404 Not Found\n{"mrstatus":3001,"helpURL":"http://onap.readthedocs.io","message":"No such topic exists.-[unauthenticated.VES_NOTIFICATION_OUTPUT]","status":404}     |RequestID=7e3b345a-02bf-4892-b03c-7c94a0f2757b     |     |     |reactor-http-epoll-3     |
PM-Mapperjavapm-dcae-pm-mapper-XXYESjboss-logging-3.4.1.Final.jar  logback-classic-1.2.3.jar      logback-core-1.2.3.jar         logging-slf4j-1.2.2.jarCustom + STDOUT (same)/var/log/ONAP/dcaegen2/services/pm-mapper $ ls
pm-mapper_output.log           pm-mapper_output_readable.log
2021-10-01T18:31:21.936Z        XNIO-1 I/O-2    INFO    org.onap.dcaegen2.services.pmmapper.healthcheck.HealthCheckHandler      InstanceUUID=7db54fca-bd45-40c8-a3f8-8c3889078d60, RequestID=b33e5eb2-31d7-4020-a03b-3104a5d8ebbc, ServiceName=/healthcheck, InvocationID=b5fd1ab0-2a52-4ab5-9692-a3f12432c9b6, InvokeTimestamp=2021-10-01T18:31:21.936895Z, PartnerName=, ClientIPAddress=/10.0.0.18, ServerFQDN=10.42.0.146                       ENTRY
2021-10-01T18:31:21.939Z        XNIO-1 I/O-2    INFO    org.onap.dcaegen2.services.pmmapper.healthcheck.HealthCheckHandler      InstanceUUID=7db54fca-bd45-40c8-a3f8-8c3889078d60, RequestID=b33e5eb2-31d7-4020-a03b-3104a5d8ebbc, ServiceName=/healthcheck, InvocationID=b5fd1ab0-2a52-4ab5-9692-a3f12432c9b6, InvokeTimestamp=2021-10-01T18:31:21.936895Z, PartnerName=, ClientIPAddress=/10.0.0.18, ServerFQDN=10.42.0.146       Healthcheck request successful
2021-10-01T18:31:21.939Z        XNIO-1 I/O-2    INFO    org.onap.dcaegen2.services.pmmapper.healthcheck.HealthCheckHandler      ResponseCode=, InstanceUUID=7db54fca-bd45-40c8-a3f8-8c3889078d60, RequestID=b33e5eb2-31d7-4020-a03b-3104a5d8ebbc, ServiceName=/healthcheck, ResponseDescription=, InvocationID=b5fd1ab0-2a52-4ab5-9692-a3f12432c9b6, Severity=, InvokeTimestamp=2021-10-01T18:31:21.936895Z, PartnerName=, ClientIPAddress=/10.0.0.18, ServerFQDN=10.42.0.146, StatusCode=                  EXIT
2021-10-01T18:31:26.208Z        Config-1        INFO    org.onap.dcaegen2.services.pmmapper.config.ConfigHandler                Fetching pm-mapper configuration from Configbinding Service
2021-10-01T18:31:26.209Z        Config-1        INFO    org.onap.dcaegen2.services.pmmapper.utils.RequestSender         2df2403b-6253-4179-999f-64724eca9d1f         INVOKE [ SYNCHRONOUS ]
2021-10-01T18:31:26.209Z        Config-1        INFO    org.onap.dcaegen2.services.pmmapper.utils.RequestSender         Sending GET request to http://10.43.242.254:10000/service_component/dcae-pm-mapper.
2021-10-01T18:31:26.220Z        Config-1        INFO    org.onap.dcaegen2.services.pmmapper.utils.RequestSender         Response code: 200, Server Response Received:\n{"aaf_identity": "dcae@dcae.onap.org", "aaf_password": *****, "dmaap_dr_delete_endpoint": "https://dmaap-dr-node:8443/delete", "enable_http": false, "enable_tls": true, "key_store_pass_path": "/opt/app/pm-mapper/etc/cert/jks.pass", "key_store_path": "/opt/app/pm-mapper/etc/cert/cert.jks", "pm-mapper-filter": "{ \"filters\":[] }", "streams_publishes": {"dmaap_publisher": {"dmaap_info": {"client_id": 1633112875004, "client_role": "org.onap.dcae.pmPublisher", "location": "san-francisco", "topic_url": "http://message-router:3904/events/org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS"}, "type": "message_router"}}, "streams_subscribes": {"dmaap_subscriber": {"dmaap_info": {"decompress": true, "delivery_url": "https://dcae-pm-mapper:8443/delivery", "location": "san-francisco", "password": *****, "privileged": true, "subscriber_id": 1, "username": "username"}, "type": "data_router"}}, "trust_store_pass_path": "/opt/app/pm-mapper/etc/cert/trust.pass", "trust_store_path": "/opt/app/pm-mapper/etc/cert/trust.jks"}
BBS-Eventprocessorjavabbs-dcae-bbs-eventprocessor-ms-XXYESLogback + slf4j.LoggerFactorySTDOUT/opt/app/bbs-event-processor/logs $ ls
/opt/app/bbs-event-processor/logs
2021-10-01 19:01:09.219 ERROR 1 --- [or-http-epoll-4] o.o.b.e.p.p.CpeAuthenticationPipeline    : DMaaP Consumer error: 404 Not Found
{"mrstatus":3001,"helpURL":"http://onap.readthedocs.io","message":"No such topic exists.-[unauthenticated.CPE_AUTHENTICATION]","status":404}
2021-10-01 19:01:09.219  INFO 1 --- [or-http-epoll-4] o.o.b.e.p.p.CpeAuthenticationPipeline    : CPE Authentication processing pipeline has been completed
2021-10-01 19:01:34.208  INFO 1 --- [pipeline-thrd-3] o.o.b.e.p.p.CpeAuthenticationPipeline    : Process next CPE Authentication events
2021-10-01 19:01:34.208  INFO 1 --- [pipeline-thrd-1] o.o.b.e.p.p.ReRegistrationPipeline       : Process next Re-Registration events
2021-10-01 19:01:34.214 ERROR 1 --- [or-http-epoll-4] t.DmaapCpeAuthenticationConsumerTaskImpl : DMaaP Consumption Exception: 404 Not Found
{"mrstatus":3001,"helpURL":"http://onap.readthedocs.io","message":"No such topic exists.-[unauthenticated.CPE_AUTHENTICATION]","status":404}
2021-10-01 19:01:34.214 ERROR 1 --- [or-http-epoll-4] o.o.b.e.p.p.CpeAuthenticationPipeline    : DMaaP Consumer error: 404 Not Found
{"mrstatus":3001,"helpURL":"http://onap.readthedocs.io","message":"No such topic exists.-[unauthenticated.CPE_AUTHENTICATION]","status":404}
2021-10-01 19:01:34.214  INFO 1 --- [or-http-epoll-4] o.o.b.e.p.p.CpeAuthenticationPipeline    : CPE Authentication processing pipeline has been completed
2021-10-01 19:01:34.223 ERROR 1 --- [or-http-epoll-3] .p.t.DmaapReRegistrationConsumerTaskImpl : DMaaP Consumption Exception: 404 Not Found
{"mrstatus":3001,"helpURL":"http://onap.readthedocs.io","message":"No such topic exists.-[unauthenticated.PNF_UPDATE]","status":404}
2021-10-01 19:01:34.224 ERROR 1 --- [or-http-epoll-3] o.o.b.e.p.p.ReRegistrationPipeline       : DMaaP Consumer error: 404 Not Found
{"mrstatus":3001,"helpURL":"http://onap.readthedocs.io","message":"No such topic exists.-[unauthenticated.PNF_UPDATE]","status":404}
2021-10-01 19:01:34.224  INFO 1 --- [or-http-epoll-3] o.o.b.e.p.p.ReRegistrationPipeline       : PNF Re-Registration processing pipeline has been completed
Heartbeat MSpython*dcae-heartbeat*YESCHK REPOSTDOUTTBA2021-07-21 19:49:51,700 | __main__ | misshtbtd | create_database | 65 |  INFO | ('MSHBT:Create_database:DB not exists? ', (False,))
2021-07-21 19:49:51,701 | __main__ | misshtbtd | create_database | 71 |  INFO | MSHBD:Database already exists
2021-07-21 19:49:51,701 | __main__ | misshtbtd | create_update_db | 306 |  INFO | ('MSHBT: DB parameters -', 'dcae-heartbeat-pg-primary', '5432', 'heartbeat', 'Zojl2,JapuTive', 'heartbeat')
2021-07-21 19:49:51,904 | __main__ | misshtbtd | create_update_vnf_table_1 | 152 |  INFO | MSHBT:Created vnf_table_1 table
2021-07-21 19:49:51,905 | __main__ | misshtbtd | create_update_vnf_table_1 | 180 |  DEBUG | Inserted new event_name = Heartbeat_vDNS into vnf_table_1
2021-07-21 19:49:51,984 | __main__ | misshtbtd | create_update_vnf_table_1 | 180 |  DEBUG | Inserted new event_name = Heartbeat_vFW into vnf_table_1
2021-07-21 19:49:51,985 | __main__ | misshtbtd | create_update_vnf_table_1 | 180 |  DEBUG | Inserted new event_name = Heartbeat_xx into vnf_table_1
2021-07-21 19:49:51,985 | __main__ | misshtbtd | create_update_vnf_table_1 | 189 |  INFO | MSHBT:Updated vnf_table_1 as per the json configuration file
2021-07-21 19:49:52,002 [onap_dcae_cbs_docker_client.client] DEBUG    opening config_path=/app-config/application_config.yaml
2021-07-21 19:49:52,002 | onap_dcae_cbs_docker_client.client | client | get_config | 217 |  DEBUG | opening config_path=/app-config/application_config.yaml
2021-07-21 19:49:52,008 [onap_dcae_cbs_docker_client.client] DEBUG    Returning config read from /app-config/application_config.yaml
2021-07-21 19:49:52,008 | onap_dcae_cbs_docker_client.client | client | get_config | 221 |  DEBUG | Returning config read from /app-config/application_config.yaml
2021-07-21 19:49:52,008 | misshtbtd | misshtbtd | fetch_json_file | 277 |  INFO | MSHBD:current config logged to : ../etc/download.json
2021-07-21 19:49:52,011 | misshtbtd | misshtbtd | fetch_json_file | 296 |  INFO | ('MSHBT: The json file is - ', '../etc/config.json')
2021-07-21 19:49:52,011 | misshtbtd | misshtbtd | read_hb_properties | 241 |  INFO | #### read_hb_properties.str.Config - {'CBS_polling_allowed': 'True', 
DL-Feederjava*dcae-datalake-feeder*YES./src/main/java/org/onap/datalake/des/controller/DataExposureController.javaSTDOUT/var/log/ONAP/dcaegen2/services/datalake $ ls -lrt
total 0		2021-10-01 19:13:17,911 |DEBUG| [     pool-3-thread-1] org.onap.datalake.feeder.service.Puller:144 - pulling...
2021-10-01 19:13:17,913 |DEBUG| [     pool-3-thread-2] org.onap.datalake.feeder.service.Puller:146 - done pulling.
2021-10-01 19:13:17,914 |DEBUG| [     pool-3-thread-2] org.onap.datalake.feeder.service.Puller:170 - no record from this polling.
2021-10-01 19:13:17,914 |DEBUG| [     pool-3-thread-2] org.onap.datalake.feeder.service.TopicConfigPollingService:96 - kafkaId=1 isActiveTopicsChanged=false, currentActiveTopicsVersion=1 local=1
2021-10-01 19:13:17,914 |DEBUG| [     pool-3-thread-2] org.onap.datalake.feeder.service.Puller:144 - pulling...
2021-10-01 19:13:17,994 |DEBUG| [     pool-3-thread-4] org.onap.datalake.feeder.service.DmaapService:150 - add effectiveTopics  org.onap.dmaap.mr.PNF_READY:[EffectiveTopic org.onap.dmaap.mr.PNF_READY (base Topic=Topic TopicName _DL_DEFAULT_ (enabled=true, dbs=[Db 1 (name=Couchbase 1, enabled=true), Db 2 (name=MongoDB 1, enabled=true), Db 3 (name=Druid, enabled=true), Db 4 (name=Hadoop Cluster, enabled=true)], kafkas=[Kafka 1 (name=main Kafka cluster, enabled=true)]))].
2021-10-01 19:13:17,995 |DEBUG| [     pool-3-thread-4] org.onap.datalake.feeder.service.DmaapService:146 - get topic setting from DB: org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS.
2021-10-01 19:13:18,186 |DEBUG| [     pool-3-thread-4] org.onap.datalake.feeder.service.DmaapService:150 - add effectiveTopics  org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS:[EffectiveTopic org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS (base Topic=Topic TopicName _DL_DEFAULT_ (enabled=true, dbs=[Db 1 (name=Couchbase 1, enabled=true), Db 2 (name=MongoDB 1, enabled=true), Db 3 (name=Druid, enabled=true), Db 4 (name=Hadoop Cluster, enabled=true)], kafkas=[Kafka 1 (name=main Kafka cluster, enabled=true)]))].
2021-10-01 19:13:18,187 |DEBUG| [     pool-3-thread-4] org.onap.datalake.feeder.service.DmaapService:146 - get topic setting from DB: org.onap.dmaap.mr.mirrormakeragent.
DL-AdminUI?*dcae-datalake-admin-ui*YES

node:alpine (builder) +  nginx:1.17.9 + uses .ts (js + angular)		STDOUT$ pwd
/var/log/ONAP/dcaegen2/services/datalake-admin-ui
$ ls -lrt
total 0
2021/10/01 19:15:11 [warn] 15#15: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1
10.0.0.18 - - [01/Oct/2021:19:22:35 +0000] "GET / HTTP/1.1" 200 1972 "-" "kube-probe/1.20" "-"
10.0.0.18 - - [01/Oct/2021:19:22:45 +0000] "GET / HTTP/1.1" 200 1972 "-" "kube-probe/1.20" "-"
10.0.0.18 - - [01/Oct/2021:19:22:55 +0000] "GET / HTTP/1.1" 200 1972 "-" "kube-probe/1.20" "-"
10.0.0.18 - - [01/Oct/2021:19:23:05 +0000] "GET / HTTP/1.1" 200 1972 "-" "kube-probe/1.20" "-"
10.0.0.18 - - [01/Oct/2021:19:23:15 +0000] "GET / HTTP/1.1" 200 1972 "-" "kube-probe/1.20" "-"
10.0.0.18 - - [01/Oct/2021:19:23:25 +0000] "GET / HTTP/1.1" 200 1972 "-" "kube-probe/1.20" "-"
DL-DESJava*dcae-datalake-des*YESlogback + slf4j.Logger, slf4j.LoggerFactorySTDOUT/var/log/ONAP/dcaegen2/services/datalake $ ls -lrt
total 0
2021-10-01 19:19:04,613 |INFO | [                main] org.onap.datalake.des.DesApplication:50  - Starting DesApplication on dcae-datalake-des with PID 13 (/datalake/des-1.1.1-execute.jar started by datalake in /datalake)
2021-10-01 19:19:04,699 |DEBUG| [                main] org.onap.datalake.des.DesApplication:53  - Running with Spring Boot v2.1.0.RELEASE, Spring v5.1.2.RELEASE
2021-10-01 19:19:04,703 |INFO | [                main] org.onap.datalake.des.DesApplication:675 - No active profile set, falling back to default profiles: default
2021-10-01 19:19:32,489 |INFO | [                main] org.onap.datalake.des.DesApplication:59  - Started DesApplication in 30.571 seconds (JVM running for 33.285)
2021-10-01 19:21:52,009 |INFO | [http-nio-1681-exec-1] org.onap.datalake.des.service.DataExposureService:76 - DataExposureList is not null
2021-10-01 19:23:20,985 |INFO | [http-nio-1681-exec-2] org.onap.datalake.des.service.DataExposureService:76 - DataExposureList is not null
2021-10-01 19:24:50,990 |INFO | [http-nio-1681-exec-3] org.onap.datalake.des.service.DataExposureService:76 - DataExposureList is not null
KPI-MSJava*dcae-kpi-ms*YESlogback + slf4j.Logger, slf4j.LoggerFactorySTDOUT
/var/log/ONAP/dcaegen2/services/kpims $ ls -lrt
total 0		18:32:21.923 [main] INFO  o.s.s.c.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
18:32:21.997 [pool-2-thread-1] WARN  c.att.nsa.apiClient.http.HttpClient - Error executing HTTP request. Connect to message-router:3905 [message-router/10.43.110.175] failed: Connection refused (Connection refused); blacklisting for 2 minutes
18:32:22.393 [main] INFO  o.s.s.c.ThreadPoolTaskScheduler - Initializing ExecutorService 'taskScheduler'
18:32:22.420 [main] INFO  io.undertow - starting server: Undertow - 2.0.30.Final
18:32:22.488 [main] INFO  org.xnio - XNIO version 3.3.8.Final
18:32:22.511 [main] INFO  org.xnio.nio - XNIO NIO Implementation Version 3.3.8.Final
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.xnio.nio.NioXnio$2 (jar:file:/app/app.jar!/BOOT-INF/lib/xnio-nio-3.3.8.Final.jar!/) to constructor sun.nio.ch.EPollSelectorProvider()
WARNING: Please consider reporting this to the maintainers of org.xnio.nio.NioXnio$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
18:32:22.825 [main] INFO  o.s.b.w.e.undertow.UndertowWebServer - Undertow started on port(s) 8080 (http)
18:32:22.888 [main] INFO  org.onap.dcaegen2.kpi.Application - Started Application in 8.586 seconds (JVM running for 20.423)
18:32:37.797 [XNIO-1 task-1] INFO  io.undertow.servlet - Initializing Spring DispatcherServlet 'dispatcherServlet'
18:32:37.797 [XNIO-1 task-1] INFO  o.s.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
18:32:37.807 [XNIO-1 task-1] INFO  o.s.web.servlet.DispatcherServlet - Completed initialization in 10 ms
18:32:40.727 [pool-2-thread-1] INFO  c.a.n.c.c.impl.CambriaConsumerImpl - UEB GET /events/org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS/kpi-cg/kpi-cid
18:32:40.727 [pool-2-thread-1] WARN  c.a.nsa.apiClient.http.HostSelector - All hosts were blacklisted; reverting to full set of hosts.
18:32:40.727 [pool-2-thread-1] INFO  c.att.nsa.apiClient.http.HttpClient - GET https://message-router:3905/events/org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS/kpi-cg/kpi-cid (anonymous) ...
18:32:41.740 [pool-2-thread-1] WARN  c.att.nsa.apiClient.http.HttpClient - Error executing HTTP request. Connect to message-router:3905 [message-router/10.43.110.175] failed: Connection refused (Connection refused); blacklisting for 2 minutes
18:33:00.727 [pool-2-thread-2] INFO  c.a.n.c.c.impl.CambriaConsumerImpl - UEB GET /events/org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS/kpi-cg/kpi-cid
18:33:00.727 [pool-2-thread-2] WARN  c.a.nsa.apiClient.http.HostSelector - All hosts were blacklisted; reverting to full set of hosts.
18:33:00.727 [pool-2-thread-2] INFO  c.att.nsa.apiClient.http.HttpClient - GET https://message-router:3905/events/org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS/kpi-cg/kpi-cid (anonymous) ...
18:33:01.740 [pool-2-thread-2] WARN  c.att.nsa.apiClient.http.HttpClient - Error executing HTTP request. Connect to message-router:3905 [message-router/10.43.110.175] failed: Connection refused (Connection refused); blacklisting for 2 minutes
18:33:10.731 [reactor-http-epoll-2] INFO  o.o.d.s.s.r.s.c.c.impl.CbsClientImpl - Got successful response from Config Binding Service
PMSHPython*dcae-pmsh*YESonaplogging.mdcContext import MDC (
onappylog==1.0.9 +
custom log_config.yaml)		Custom + STDOUT (same)/var/log/ONAP/dcaegen2/services/pmsh $ ls -lrt
total 272
-rw-r--r--    1 onap     nogroup     271844 Oct  7 18:41 application.log		2021-10-07T18:35:36+0000 | sub_handler_thread | 140470855510816 | INFO | pmsh_utils | _get_pmsh_config | ServiceName=dcae-pmsh | RequestID=5cfaa7b2-279d-11ec-bd5b-ca5d874accd1 | InvocationID=5cfaa8e8-279d-11ec-bd5b-ca5d874accd1 | Attempting to fetch PMSH Configuration from CBS.
2021-10-07T18:35:36+0000 | sub_handler_thread | 140470855510816 | INFO | pmsh_utils | _get_pmsh_config | ServiceName=dcae-pmsh | RequestID=5cfaa7b2-279d-11ec-bd5b-ca5d874accd1 | InvocationID=5cfaa8e8-279d-11ec-bd5b-ca5d874accd1 | Successfully fetched PMSH config from CBS: {'config': {'aaf_identity': 'dcae@dcae.onap.org', 'aaf_password': 'demo123456!', 'ca_cert_path': '/opt/app/pmsh/etc/certs/cacert.pem', 'cert_path': '/opt/app/pmsh/etc/certs/cert.pem', 'control_loop_name': 'pmsh-control-loop', 'enable_tls': True, 'key_path': '/opt/app/pmsh/etc/certs/key.pem', 'operational_policy_name': 'pmsh-operational-policy', 'pmsh_policy': {'subscription': {'administrativeState': 'LOCKED', 'fileBasedGP': 15, 'fileLocation': '/pm/pm.xml', 'measurementGroups': [{'measurementGroup': {'managedObjectDNsBasic': [{'DN': 'dna'}, {'DN': 'dnb'}], 'measurementTypes': [{'measurementType': 'countera'}, {'measurementType': 'counterb'}]}}, {'measurementGroup': {'managedObjectDNsBasic': [{'DN': 'dnc'}, {'DN': 'dnd'}], 'measurementTypes': [{'measurementType': 'counterc'}, {'measurementType': 'counterd'}]}}], 'nfFilter': {'modelInvariantIDs': [], 'modelNames': [], 'modelVersionIDs': [], 'nfNames': ['^pnf.*', '^vnf.*']}, 'subscriptionName': 'ExtraPM-All-gNB-R2B'}}, 'streams_publishes': {'policy_pm_publisher': {'dmaap_info': {'topic_url': 'https://message-router:3905/events/unauthenticated.DCAE_CL_OUTPUT'}, 'type': 'message_router'}}, 'streams_subscribes': {'aai_subscriber': {'dmaap_info': {'topic_url': 'https://message-router:3905/events/AAI-EVENT'}, 'type': 'message_router'}, 'policy_pm_subscriber': {'dmaap_info': {'topic_url': 'https://message-router:3905/events/unauthenticated.PMSH_CL_INPUT'}, 'type': 'message_router'}}}}
2021-10-07T18:35:36+0000 | sub_handler_thread | 140470855510816 | INFO | pmsh_utils | refresh_config | ServiceName=dcae-pmsh | RequestID=5cfaa7b2-279d-11ec-bd5b-ca5d874accd1 | InvocationID=5cfaa8e8-279d-11ec-bd5b-ca5d874accd1 | AppConfig data has been refreshed
2021-10-07T18:35:36+0000 | sub_handler_thread | 140470855510816 | INFO | subscription_handler | execute | ServiceName=dcae-pmsh | RequestID=5cfaa7b2-279d-11ec-bd5b-ca5d874accd1 | InvocationID=5cfaa8e8-279d-11ec-bd5b-ca5d874accd1 | Administrative State did not change in the app config: LOCKED
10.0.0.18 - - [07/Oct/2021 18:35:38] "GET /healthcheck HTTP/1.1" 200 -
2021-10-07T18:35:44+0000 | policy_event_thread | 140470856571680 | INFO | policy_response_handler | poll_policy_topic | ServiceName=dcae-pmsh | RequestID=5221edf0-279d-11ec-bd5b-ca5d874accd1 | InvocationID=5221efbc-279d-11ec-bd5b-ca5d874accd1 | Polling MR for XNF activation/deactivation policy response events.
2021-10-07T18:35:44+0000 | policy_event_thread | 140470856571680 | INFO | pmsh_utils | get_from_topic | ServiceName=dcae-pmsh | RequestID=61a3c384-279d-11ec-bd5b-ca5d874accd1 | InvocationID=61a3c56e-279d-11ec-bd5b-ca5d874accd1 | Fetching messages from MR topic: https://message-router:3905/events/unauthenticated.PMSH_CL_INPUT
2021-10-07T18:35:45+0000 | policy_event_thread | 140470856571680 | ERROR | pmsh_utils | get_from_topic | ServiceName=dcae-pmsh | RequestID=61a3c384-279d-11ec-bd5b-ca5d874accd1 | InvocationID=61a3c56e-279d-11ec-bd5b-ca5d874accd1 | Failed to fetch message from MR: HTTPSConnectionPool(host='message-router', port=3905): Max retries exceeded with url: /events/unauthenticated.PMSH_CL_INPUT/dcae_pmsh_cg/dcae_pmsh_policy_cl_input?timeout=1000 (Ca
RESTConfJava*dcae-restconf*YEScommons-logging-1.1.1.jar      log4j-api-2.12.1.jar   log4j-jul-2.12.1.jar
jboss-logging-3.4.1.Final.jar  log4j-core-2.12.1.jar  spring-boot-starter-log4j2-2.1.5.RELEASE.jar		Custom + STDOUT (same)
restconfcollector@dcae-restconf-collector:/opt/app/restconfcollector/logs$ ls -lrt
total 340
-rw-r--r-- 1 restconfcollector restconfcollector 341113 Oct  7 18:52 console.txt		2021-10-07 18:49:29.588  INFO 29 --- [pool-3-thread-1] o.onap.dcae.common.RestapiCallNodeUtil   : Parameter trustStorePassword: [Au109vBr3O;;7+WZQXY4hqZs]
2021-10-07 18:49:29.588  INFO 29 --- [pool-3-thread-1] o.onap.dcae.common.RestapiCallNodeUtil   : Parameter keyStoreFileName: [/opt/app/dcae-certificate/cert.jks]
2021-10-07 18:49:29.588  INFO 29 --- [pool-3-thread-1] o.onap.dcae.common.RestapiCallNodeUtil   : Parameter keyStorePassword: [ofu3%tPpJJ??DUWHu#c4$}]K]
2021-10-07 18:49:29.588  INFO 29 --- [pool-3-thread-1] o.onap.dcae.common.RestapiCallNodeUtil   : Parameter dumpHeaders: [false]
2021-10-07 18:49:29.588  INFO 29 --- [pool-3-thread-1] o.onap.dcae.common.RestapiCallNodeUtil   : Parameter returnRequestPayload: [false]
2021-10-07 18:49:29.588  INFO 29 --- [pool-3-thread-1] o.onap.dcae.common.RestapiCallNodeUtil   : Parameter disableSsl: [true]
2021-10-07 18:49:29.588  INFO 29 --- [pool-3-thread-1] o.onap.dcae.common.RestapiCallNodeUtil   : org.onap.dcae.common.Parameters@250d47f3
2021-10-07 18:49:29.588  INFO 29 --- [pool-3-thread-1] org.onap.dcae.common.RestapiCallNode     : p.templateFileName ./etc/access-token.json
2021-10-07 18:49:29.588  INFO 29 --- [pool-3-thread-1] org.onap.dcae.common.RestapiCallNode     : Building JSON started
2021-10-07 18:49:29.588  INFO 29 --- [pool-3-thread-1] org.onap.dcae.common.RestapiCallNode     : Building JSON completed. Time: 0
2021-10-07 18:49:29.589  INFO 29 --- [pool-3-thread-1] org.onap.dcae.common.RestapiCallNode     : Properties:
2021-10-07 18:49:29.623  INFO 29 --- [pool-3-thread-1] org.onap.dcae.common.RestapiCallNode     : Sending request:
2021-10-07 18:49:29.624  INFO 29 --- [pool-3-thread-1] org.onap.dcae.common.RestapiCallNode     : {
  "grantType" : "password",
  "userName" : "swisscom",
  "value" : "Test_12345"
}

2021-10-07 18:49:29.624  INFO 29 --- [pool-3-thread-1] org.onap.dcae.common.RestapiCallNode     : URL: https://172.30.0.55:26335/rest/plat/smapp/v1/oauth/token method PUT Custom header null
2021-10-07 18:49:34.628  INFO 29 --- [pool-3-thread-1] c.s.j.api.client.filter.LoggingFilter    : 1 * Client out-bound request
1 > PUT https://172.30.0.55:26335/rest/plat/smapp/v1/oauth/token
1 > Accept: application/json
1 > Content-Type: application/json
{
  "grantType" : "password",
  "userName" : "swisscom",
  "value" : "Test_12345"
}


2021-10-07 18:49:34.629 ERROR 29 --- [pool-3-thread-1] org.onap.dcae.common.RestapiCallNode     : Error sending the request: Exception while sending http request to client java.net.SocketTimeoutException: connect timed out
SliceAnalysis MSJava*slice-analysis-ms*YESlogback + slf4j.Logger, slf4j.LoggerFactorySTDOUT$ cd /var/log/ONAP/dcaegen2/services/sliceanalysisms
$ ls		18:49:13.409 [task-1] INFO  o.h.jpa.internal.util.LogHelper - HHH000204: Processing PersistenceUnitInfo [
        name: default
        ...]
18:49:13.904 [task-1] INFO  org.hibernate.Version - HHH000412: Hibernate Core {5.3.7.Final}
18:49:13.906 [task-1] INFO  org.hibernate.cfg.Environment - HHH000206: hibernate.properties not found
18:49:14.402 [pool-2-thread-1] INFO  c.a.n.c.c.impl.CambriaConsumerImpl - UEB GET /events/org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS/sliceanalysisms-cg/sliceanalysisms-cid
18:49:14.403 [pool-3-thread-1] INFO  c.a.n.c.c.impl.CambriaConsumerImpl - UEB GET /events/DCAE_CL_RSP/sliceanalysisms-cg/sliceanalysisms-cid
18:49:14.403 [pool-4-thread-1] INFO  c.a.n.c.c.impl.CambriaConsumerImpl - UEB GET /events/unauthenticated.ML_RESPONSE_TOPIC/sliceanalysisms-cg/sliceanalysisms-cid
18:49:14.506 [pool-3-thread-1] INFO  c.att.nsa.apiClient.http.HttpClient - GET https://message-router:3905/events/DCAE_CL_RSP/sliceanalysisms-cg/sliceanalysisms-cid (anonymous) ...
18:49:14.506 [pool-4-thread-1] INFO  c.att.nsa.apiClient.http.HttpClient - GET https://message-router:3905/events/unauthenticated.ML_RESPONSE_TOPIC/sliceanalysisms-cg/sliceanalysisms-cid (anonymous) ...
18:49:14.506 [pool-2-thread-1] INFO  c.att.nsa.apiClient.http.HttpClient - GET https://message-router:3905/events/org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS/sliceanalysisms-cg/sliceanalysisms-cid (anonymous) ...
18:49:14.607 [main] INFO  o.h.validator.internal.util.Version - HV000001: Hibernate Validator 6.0.14.Final
18:49:15.097 [task-1] INFO  o.h.annotations.common.Version - HCANN000001: Hibernate Commons Annotations {5.0.4.Final}
18:49:15.789 [task-1] INFO  org.hibernate.dialect.Dialect - HHH000400: Using dialect: org.hibernate.dialect.PostgreSQL9Dialect
18:49:15.790 [main] WARN  o.s.b.a.o.j.JpaBaseConfiguration$JpaWebConfiguration - spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
18:49:15.825 [pool-3-thread-1] WARN  c.att.nsa.apiClient.http.HttpClient - Error executing HTTP request. Connect to message-router:3905 [message-router/10.43.110.175] failed: Connection refused (Connection refused); blacklisting for 2 minutes
18:49:15.885 [pool-4-thread-1] WARN  c.att.nsa.apiClient.http.HttpClient - Error executing HTTP request. Connect to message-router:3905 [message-router/10.43.110.175] failed: Connection refused (Connection refused); blacklisting for 2 minutes
18:49:15.887 [pool-2-thread-1] WARN  c.att.nsa.apiClient.http.HttpClient - Error executing HTTP request. Connect to message-router:3905 [message-router/10.43.110.175] failed: Connection refused (Connection refused); blacklisting for 2 minutes
18:49:15.907 [task-1] INFO  o.h.e.j.e.i.LobCreatorBuilderImpl - HHH000422: Disabling contextual LOB creation as connection was null
SNMPTRap CollectorPython*dcae-snmptrap-collector*NO (DCAEGEN2-2957)custom py loggerEELF/opt/app/snmptrap/logs2021-10-07T19:16:43.098||sw_log_metrics|snmptrapd||||WARN|100||total traps: 6, interval in seconds: 60, average traps-per-second: 0
2021-10-07T19:16:43.099||sw_log_metrics|snmptrapd||||WARN|100||agent: 127.0.0.1, notifications: 6, interval in seconds: 60, percent of total traps: 100
2021-10-07T19:16:44.140||post_dmaap|snmptrapd||||WARN|100||OS exception while attempting to post 1b2eef9a-27a3-11ec-b986-c240818df74b attempt 0: (None) None HTTPConnectionPool(host='message-router', port=3904): Max retries exceeded with url: /events/unauthenticated.ONAP-COLLECTOR-SNMPTRAP (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000000038b5168>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-10-07T19:16:44.141||post_dmaap|snmptrapd||||WARN|100||sleeping 0.7500 seconds and retrying
2021-10-07T19:16:45.933||post_dmaap|snmptrapd||||WARN|100||OS exception while attempting to post 1b2eef9a-27a3-11ec-b986-c240818df74b attempt 1: (None) None HTTPConnectionPool(host='message-router', port=3904): Max retries exceeded with url: /events/unauthenticated.ONAP-COLLECTOR-SNMPTRAP (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000000038b51a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-10-07T19:16:45.934||post_dmaap|snmptrapd||||WARN|100||sleeping 0.7500 seconds and retrying
2021-10-07T19:16:47.692||post_dmaap|snmptrapd||||WARN|100||OS exception while attempting to post 1b2eef9a-27a3-11ec-b986-c240818df74b attempt 2: (None) None HTTPConnectionPool(host='message-router', port=3904): Max retries exceeded with url: /events/unauthenticated.ONAP-COLLECTOR-SNMPTRAP (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000000038b51d8>: Failed to establish a new connection: [Errno 111] Connection refused'))
2021-10-07T19:16:47.694||post_dmaap|snmptrapd||||ERROR|100||ALL publish attempts failed for traps 1b2eef9a-27a3-11ec-b986-c240818df74b to URL http://message-router:3904/events/unauthenticated.ONAP-COLLECTOR-SNMPTRAP
SON-HandlerJava*dcae-son-handler*YESCHK REPOSTDOUT$ cd /var/log/ONAP/dcaegen2/services/sonhms
$ ls -lrt
total 0		] failed: Connection refused (Connection refused); blacklisting for 2 minutes
19:27:42.735 [pool-6-thread-1] WARN  c.att.nsa.apiClient.http.HttpClient - Error executing HTTP request. Connect to message-router:3904 [message-router/10.43.110.175] failed: Connection refused (Connection refused); blacklisting for 2 minutes
19:27:44.748 [pool-5-thread-2] WARN  c.att.nsa.apiClient.http.HttpClient - Error executing HTTP request. Connect to message-router:3904 [message-router/10.43.110.175] failed: Connection refused (Connection refused); blacklisting for 2 minutes
19:28:01.706 [pool-3-thread-2] INFO  c.a.n.c.c.impl.CambriaConsumerImpl - UEB GET /events/PCI-NOTIF-TOPIC-NGHBR-LIST-CHANGE-INFO/sonhms-cg/sonhms-cid
19:28:01.706 [pool-3-thread-2] WARN  c.a.nsa.apiClient.http.HostSelector - All hosts were blacklisted; reverting to full set of hosts.
19:28:01.707 [pool-3-thread-2] INFO  c.att.nsa.apiClient.http.HttpClient - GET http://message-router:3904/events/PCI-NOTIF-TOPIC-NGHBR-LIST-CHANGE-INFO/sonhms-cg/sonhms-cid (anonymous) ...
19:28:01.707 [pool-4-thread-1] INFO  c.a.n.c.c.impl.CambriaConsumerImpl - UEB GET /events/unauthenticated.SEC_FAULT_OUTPUT/sonhms-cg/sonhms-cid
19:28:01.707 [pool-4-thread-1] WARN  c.a.nsa.apiClient.http.HostSelector - All hosts were blacklisted; reverting to full set of hosts.
19:28:01.707 [pool-4-thread-1] INFO  c.att.nsa.apiClient.http.HttpClient - GET http://message-router:3904/events/unauthenticated.SEC_FAULT_OUTPUT/sonhms-cg/sonhms-cid (anonymous) ...
19:28:01.709 [pool-6-thread-2] INFO  c.a.n.c.c.impl.CambriaConsumerImpl - UEB GET /events/DCAE_CL_RSP/sonhms-cg/sonhms-cid
19:28:01.713 [pool-6-thread-2] WARN  c.a.nsa.apiClient.http.HostSelector - All hosts were blacklisted; reverting to full set of hosts.
19:28:01.713 [pool-6-thread-2] INFO  c.att.nsa.apiClient.http.HttpClient - GET http://message-router:3904/events/DCAE_CL_RSP/sonhms-cg/sonhms-cid (anonymous) ...
19:28:01.722 [pool-5-thread-1] INFO  c.a.n.c.c.impl.CambriaConsumerImpl - UEB GET /events/unauthenticated.VES_MEASUREMENT_OUTPUT/sonhms-cg/sonhms-cid
19:28:01.722 [pool-5-thread-1] WARN  c.a.nsa.apiClient.http.HostSelector - All hosts were blacklisted; reverting to full set of hosts.
19:28:01.722 [pool-5-thread-1] INFO  c.att.nsa.apiClient.http.HttpClient - GET http://message-router:3904/events/una
VES-MapperJava*dcae-ves-mapper*YES
apache-log4j-extras-1.2.17.jar  jboss-logging-annotations-1.2.0.Beta1.jar  log4j-core-2.13.3.jar        logback-classic-1.2.3.jar
commons-logging-1.1.1.jar       log4j-1.2.17.jar                           log4j-over-slf4j-1.7.25.jar  logback-core-1.2.3.jar
jboss-logging-3.3.2.Final.jar   log4j-api-2.13.3.jar                       log4j-to-slf4j-2.11.1.jar    spring-boot-starter-logging-2.1.1.RELEASE.jar		EELF + STDOUT/opt/app/VESAdapter/logs
# ls -lrt
total 20
-rw-r--r-- 1 root root   726 Oct  7 19:33 metrics.log
-rw-r--r-- 1 root root   707 Oct  7 19:33 error.log
-rw-r--r-- 1 root root 10584 Oct  7 19:35 debug.log		9:33:39.810 [http-nio-8085-exec-1] INFO  debugLogger - Created new DMaaP MR Publisher Instance. Publisher creation time: Thu Oct 07 19:33:39 UTC 2021
19:33:39.819 [http-nio-8085-exec-1] INFO  debugLogger - Created scriber topic:unauthenticated.DCAE_RCC_OUTPUTpublisher topic:unauthenticated.VES_PNFREG_OUTPUT
19:33:39.821 [rcc-notification] INFO  metricsLogger - fetch and publish from and to Dmaap started:rcc-notification
19:33:39.822 [rcc-notification] INFO  debugLogger - The Polling Interval in Milli Second is :{}10000
19:33:39.822 [rcc-notification] INFO  debugLogger - starting subscriber & publisher thread:rcc-notification
19:33:39.826 [rcc-notification] DEBUG debugLogger - DMaaP MR Subscriber Authentication is disabled as username or password is not present.
19:33:39.894 [http-nio-8085-exec-1] INFO  debugLogger - Created new instance of DMaaP MR Factory
19:33:39.895 [http-nio-8085-exec-1] DEBUG debugLogger - Creating new DMaaP MR Subscriber Instance with configuration: org.onap.universalvesadapter.configs.DMaaPMRSubscriberConfig@873f02bb
19:33:39.896 [http-nio-8085-exec-1] INFO  debugLogger - Created DMaaP MR Subscriber URI: http://message-router:3904/events/unauthenticated.ONAP-COLLECTOR-SNMPTRAP/grp2/con2
19:33:39.896 [http-nio-8085-exec-1] INFO  debugLogger - Created new DMaaP MR Subscriber Instance. Subscriber creation time: Thu Oct 07 19:33:39 UTC 2021
19:33:39.906 [http-nio-8085-exec-1] INFO  debugLogger - Created new instance of DMaaP MR Factory
19:33:39.909 [http-nio-8085-exec-1] DEBUG debugLogger - Creating new DMaaP MR Publisher Instance with configuration: org.onap.universalvesadapter.configs.DMaaPMRPublisherConfig@1ea00ecf
19:33:39.913 [http-nio-8085-exec-1] DEBUG debugLogger - Creating Instance of DMaaP Publisher Queue. BatchQueueSize: 1, RecoveryQueueSize: 100000
19:33:39.915 [http-nio-8085-exec-1] INFO  debugLogger - Created DMaaP MR Publisher URI: http://message-router:3904/events/unauthenticated.SEC_FAULT_OUTPUT
19:33:39.915 [http-nio-8085-exec-1] INFO  debugLogger - Created new DMaaP MR Publisher Instance. Publisher creation time: Thu Oct 07 19:33:39 UTC 2021
19:33:39.916 [http-nio-8085-exec-1] INFO  debugLogger - Created scriber topic:unauthenticated.ONAP-COLLECTOR-SNMPTRAPpublisher topic:unauthenticated.SEC_FAULT_OUTPUT
19:33:39.988 [snmp-notification] INFO  metricsLogger - fetch and publish from and to Dmaap started:snmp-notification
19:33:39.989 [snmp-notification] INFO  debugLogger - The Polling Interval in Milli Second is :{}10000
19:33:39.989 [snmp-notification] INFO  debugLogger - starting subscriber & publisher thread:snmp-notification
19:33:39.989 [snmp-notification] DEBUG debugLogger - DMaaP MR Subscriber Authentication is disabled as username or password is not present.
19:33:40.020 [reactor-http-epoll-2] DEBUG debugLogger - VESAdapterInitializer:: fetchResultFromDestination :: END
19:33:40.909 [rcc-notification] ERROR errorLogger - IO Exception while fetching messages from DMaaP Topic. Exception org.apache.http.conn.HttpHostConnectException: Connect to message-router:3904 [message-router/10.43.110.175] failed: Connection refused (Connection refused)
19:33:41.008 [snmp-notification] ERROR errorLogger - IO Exception while fetching messages from DMaaP Topic. Exception org.apache.http.conn.HttpHostConnectException: Connect to message-router:3904 [message-router/10.43.110.175] failed: Connection refused (Connection refused)
19:34:35.986 [reactor-http-epoll-2] INFO  o.o.d.s.s.r.s.c.c.impl.CbsClientImpl - Got successful response from Config Binding Service
DCAE MS HealthcheckJS*healthcheck*NO (DCAEGEN2-2958)node:14.17.0-alpine3.13 + js
NA