see: LOG-66 - Getting issue details... STATUS LOG-49 - Getting issue details... STATUS
Quickstart - getting your ELK Dashboard up
Logging Analytics Dashboards (Kibana)
Enabling the ELK stack
Target the VM in your cluster that is running log-kibana - in the live CD system for example it is usually on http://master3.onap.info:30253/
The index needs to be enabled - this is currently being automated in LOG-152 - Getting issue details... STATUS for now you can flip the following dropdown.
You will be able to search on logs - separate from also using the kibana dashoards.
Troubleshooting:
# check indices to verify data exists in Elasticsearch
root@k8s:~# curl -u elastic:changeme http://127.0.0.1:30254/_cat/indices
red open .monitoring-es-6-2017.10.04 X9nA9PHsR92u9VA1EjvrjA 1 1
yellow open .monitoring-es-6-2017.10.05 GEWfbYV8Qu632ILcOaGufg 1 1 70825 324 49mb 49mb
yellow open onaplogs-2017.10.05 qOErjm_zR1yJbES2A94GuA 5 1 417450 0 146.9mb 146.9mb
yellow open .triggered_watches sJWEmGgYRmuOsOa0ouWgaw 1 1 0 0 365.7kb 365.7kb
yellow open onaplogs-2017.10.04 NoEhHdK3ToeUC2KGqo3XlA 5 1 270744 0 103.1mb 103.1mb
yellow open .monitoring-alerts-6 84pDDo6HQOGIqgJQ5nMriA 1 1 1 0 6.3kb 6.3kb
yellow open .watcher-history-3-2017.10.04 lZ_1PBP-RiuevffPV3a1-g 1 1 2612 0 2.6mb 2.6mb
yellow open .watcher-history-3-2017.10.05 IDLSTtl9Thq0pTuTyzngNQ 1 1 4544 0 3.2mb 3.2mb
yellow open .kibana WWyQNR5HTzCRsqEQkPR3YA 1 1 1 0 3.2kb 3.2kb
yellow open .watches vS6TCNdiTwSL9JJOx_N8QQ 1 1 4 0 63.4kb 63.4kb
Configuration
search on log* and select a "time filter field name" to @timestamp
onap | split rows | aggregation=terms | field=source.keyword | size=100, play
# filter by aai
October 5th 2017, 11:10:39.087 source:/var/log/onap/aai/aai-ml/error.log offset:605,688 DistributionClientResultImpl .responseStatus:ASDC_SERVER_PROBLEM, responseMessage=ASDC server problem] 2017-10-05T15:10:37.393Z input_type:log message: MDLSVC2001E|MDLSVC2001E Unable to register with ASDC: Failed to initialize distribution client: ASDC server problem| type:log Logger:org.onap.aai.modelloader.service.SdcConnectionJob Timestamp:October 5th 2017, 11:10:37.347 Thread:[Timer-0] INFO org.openecomp.sdc.impl.DistributionClientImpl DistributionClient - init 2017-10-05T15:10:37.393Z [Timer-0] ERROR org.openecomp.sdc.http.AsdcConnectorClient status from ASDC is org.openecomp.sdc.http.HttpAsdcResponse@2c523e67 2017-10-05T15:10:37.393Z [Timer-0] ERROR org.open
ONAP Kibana Dashboard
Logging Design
Repositories
| Repo | Directory | Details |
|---|---|---|
| oom | Deployment yamls | |
| oom | configs | |
| logging-analytics | docker image configs | |
| logging-analytics | reference logback.xml configs | |
| (various) aai.... | runtime logback.xml/log4j configs |
Logging API
Discuss shared library approach to logging (Java only for now)
Logging DevOps
Logging Framework Deployment
The ELK stack containers are under the onap-log namespace in the OOM deployment of ONAP. They should be started by default - if not you can start them manually.
Kibana is on port 30253, Elasticsearch is on port 30254
# look for log containers
root@ip-172-31-82-46:~# kubectl get pods --all-namespaces -a | grep onap-log
onap-log elasticsearch-2934588800-st9j3 1/1 Running 0 3h
onap-log kibana-3372627750-ff8rv 1/1 Running 0 3h
onap-log logstash-1708188010-703pk 1/1 Running 0 3h
# start the ELK stack if required
root@ip-172-31-82-46:~/oom/kubernetes/oneclick# ./createAll.bash -n onap -a log
# check access ports (30254 and 30253)
root@ip-172-31-82-46:~/oom/kubernetes/oneclick# kubectl get services --all-namespaces -a | grep onap-log
onap-log elasticsearch 10.43.86.120 <nodes> 9200:30254/TCP
onap-log kibana 10.43.165.215 <nodes> 5601:30253/TCP
onap-log logstash 10.43.72.107 <none> 5044/TCP
# check for pods with filebeat containers (will be 2 per pod)
root@kos1001:/dockerdata-nfs/onap/robot# kubectl get pods --all-namespaces -a | grep 2/2
onap-aai aai-resources-338473047-8k6vr 2/2 Running 0 7h
onap-aai aai-traversal-2033243133-6cr9v 2/2 Running 0 7h
onap-aai model-loader-service-3356570452-25fjp 2/2 Running 0 7h
onap-aai search-data-service-2366687049-jt0nb 2/2 Running 0 7h
onap-aai sparky-be-3141964573-f2mhr 2/2 Running 0 7h
onap-appc appc-1335254431-v1pcs 2/2 Running 0 7h
onap-mso mso-3911927766-bmww7 2/2 Running 0 7h
onap-policy drools-2302173499-t0zmt 2/2 Running 0 7h
onap-policy pap-1954142582-vsrld 2/2 Running 0 7h
onap-policy pdp-4137191120-qgqnj 2/2 Running 0 7h
onap-portal portalapps-4168271938-4kp32 2/2 Running 0 7h
onap-portal portaldb-2821262885-0t32z 2/2 Running 0 7h
onap-sdc sdc-be-2986438255-sdqj6 2/2 Running 0 7h
onap-sdc sdc-fe-1573125197-7j3gp 2/2 Running 0 7h
onap-sdnc sdnc-3858151307-w9h7j 2/2 Running 0 7h
onap-vid vid-server-1837290631-x4ttc 2/2 Running 0 7h
Triage Log Capture
The following are procedures to determine the state of logs traversing from the container under use via the filebeat, logstash, elasticsearch pipeline.
Originating Container
Filebeat Sidecar Container
Logstash DaemonSet
Elasticsearch container
Kibana container
Enabling Debug Logs
Via URL
| MSO | Debuglog successfully enabled. |
Via Logback.xml
Logging Access
Robot logs
after a
/dockerdata-nfs/onap/robot# ./demo-k8s.sh init_customer
http://host:30209/logs/demo/InitCustomer/report.html
user:robot pass:robot
Host VM logs
/dockerdata-nfs/onap/aai/aai-traversal/logs/
/dockerdata-nfs/onap/sdc/logs/ASDC/ASDC-BE/
ELK Logs
Training Videos
| Video | Details |
|---|---|
| Configuring a new ONAP install's ELK stack |