...
Discuss shared library approach to logging (Java only for now)
Logging DevOps
ONAP Development#KubernetesDevOps
Logging Framework Deployment
The ELK stack containers are under the onap-log namespace in the OOM deployment of ONAP. They should be started by default - if not you can start them manually.
Kibana is on port 30253, Elasticsearch is on port 30254
| Code Block |
|---|
# look for log containers
root@ip-172-31-82-46:~# kubectl get pods --all-namespaces -a | grep onap-log
onap-log elasticsearch-2934588800-st9j3 1/1 Running 0 3h
onap-log kibana-3372627750-ff8rv 1/1 Running 0 3h
onap-log logstash-1708188010-703pk 1/1 Running 0 3h
# start the ELK stack if required
root@ip-172-31-82-46:~/oom/kubernetes/oneclick# ./createAll.bash -n onap -a log
# check access ports (30254 and 30253)
root@ip-172-31-82-46:~/oom/kubernetes/oneclick# kubectl get services --all-namespaces -a | grep onap-log
onap-log elasticsearch 10.43.86.120 <nodes> 9200:30254/TCP
onap-log kibana 10.43.165.215 <nodes> 5601:30253/TCP
onap-log logstash 10.43.72.107 <none> 5044/TCP
# check for pods with filebeat containers (will be 2 per pod)
root@kos1001:/dockerdata-nfs/onap/robot# kubectl get pods --all-namespaces -a | grep 2/2
onap-aai aai-resources-338473047-8k6vr 2/2 Running 0 7h
onap-aai aai-traversal-2033243133-6cr9v 2/2 Running 0 7h
onap-aai model-loader-service-3356570452-25fjp 2/2 Running 0 7h
onap-aai search-data-service-2366687049-jt0nb 2/2 Running 0 7h
onap-aai sparky-be-3141964573-f2mhr 2/2 Running 0 7h
onap-appc appc-1335254431-v1pcs 2/2 Running 0 7h
onap-mso mso-3911927766-bmww7 2/2 Running 0 7h
onap-policy drools-2302173499-t0zmt 2/2 Running 0 7h
onap-policy pap-1954142582-vsrld 2/2 Running 0 7h
onap-policy pdp-4137191120-qgqnj 2/2 Running 0 7h
onap-portal portalapps-4168271938-4kp32 2/2 Running 0 7h
onap-portal portaldb-2821262885-0t32z 2/2 Running 0 7h
onap-sdc sdc-be-2986438255-sdqj6 2/2 Running 0 7h
onap-sdc sdc-fe-1573125197-7j3gp 2/2 Running 0 7h
onap-sdnc sdnc-3858151307-w9h7j 2/2 Running 0 7h
onap-vid vid-server-1837290631-x4ttc 2/2 Running 0 7h
|
Enabling Debug Logs
Via URL
...
http://cd.onap.info:30223/mso/logging/debug
...
Debuglog successfully enabled.Via Logback.xml
Logging Access
Robot logs
after a
/dockerdata-nfs/onap/robot# ./demo-k8s.sh init_customer
http://host:30209/logs/demo/InitCustomer/report.html
user:robot pass:robot
Host VM logs
/dockerdata-nfs/onap/aai/aai-traversal/logs/
/dockerdata-nfs/onap/sdc/logs/ASDC/ASDC-BE/Triage Log Capture
The following are procedures to determine the state of logs traversing from the container under use via the filebeat, logstash, elasticsearch pipeline.
I would start with the deployment.yaml and verify the filebeat section is the same as other working pods that are listed in the last check I did for pairwise testing. I would expect logs from for example portal in portal-app and portal-sdk – at least error logs.
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
After this I would check the logstash service to see if it is receiving logs its port – I’ll add instructions to the wiki.
We can also check the docker pv for the 2 portal containers – the emptydir should be in /var/lib/docker – this is a secondary place we can pickup/verify the logs besides filebeat.
Originating Container
Filebeat Sidecar Container
Filebeat sidecar container setup and configuration in OOM
Logstash DaemonSet
Elasticsearch container
Kibana container
Enabling Debug Logs
Via URL
| MSO | Debuglog successfully enabled. |
Via Logback.xml
Logging Access
Robot logs
after a
/dockerdata-nfs/onap/robot# ./demo-k8s.sh init_customer
http://host:30209/logs/demo/InitCustomer/report.html
user:robot pass:robot
Host VM logs
/dockerdata-nfs/onap/aai/aai-traversal/logs/
/dockerdata-nfs/onap/sdc/logs/ASDC/ASDC-BE/
| Code Block | ||
|---|---|---|
| ||
ubuntu@ip-172-31-15-18:/dockerdata-nfs/onap$ ls -la aai/data-router/logs/AAI-DR/
-rw-r--r-- 1 root root 0 Aug 16 01:11 audit.log
-rw-r--r-- 1 root root 0 Aug 16 01:11 debug.log
-rw-r--r-- 1 root root 106459 Aug 17 00:01 error.2018-08-16.log.zip
-rw-r--r-- 1 root root 5562166 Aug 17 19:50 error.log
-rw-r--r-- 1 root root 536 Aug 16 01:12 metrics.log |
ELK Logs
Training Videos
| Video | Details | ||||||
|---|---|---|---|---|---|---|---|
| Configuring a new ONAP install's ELK stack |
...