WIP
This page is the design and reference site for the Logging RI. The following topics are being provided to the rest of the ONAP community.
- How to use the EELF Logging Library
- How to adhere to the logging guidelines
- How to implement the filebeat sidecar container to push logs to the ELK stack pod
- How to use the EELF AOP wrapping library
- How to ensure requestID tracking during distributed transactions
Questions on onap-discuss from the teams
https://lists.onap.org/pipermail/onap-discuss/2018-January/007415.html
LOG-118 - Getting issue details... STATUS
Purpose
The goals of the logging project are the following
provide for standardized logs across all ONAP components
provide ELK framework for log capture, indexing and presentation/search
provide framework for requestID tracking of distributed transactions (correlation ID's in p.162 of Building Microservices 2015 by Sam Newman) via HTTP ETags
Architecture
Microservices
NBI - Northbound Interface (1/1)
Logging Demo WAR
PN - Processing Node (scaled N/N)
Processing Node WAR
Project Structure
| Project | repo | Dependencies | Artifact | Docker |
|---|---|---|---|---|
| logging-reference | logging-reference | pom | ||
| logging-demo | logging-demo | WAR | ||
| logging-mock-service | logging-mock-service | logging-library | WAR | |
| Logging Library | logging-library | JAR | ||
| Kubernetes demo spec | kubernetes/log-mock-demo | yaml, config | ||
| Kubernetes service spec | kubernetes/log-mock-service | yaml, config | ||
| logging-docker-root | pom | |||
| logging-docker-demo | logging-demo | DockerFile | ||
| logging-docker-service | logging-mock-service | DockerFile |
API
Deployment Diagram
Quickstart
WIP (as of 20180121) - will notify when complete
Goto your IDE workspace
Clone the logging repo
https://gerrit.onap.org/r/#/admin/projects/logging-analytics
https://git.onap.org/logging-analytics/tree/
goto the reference folder and load the pom.xml project
obrienbiometrics:wse_sts392b michaelobrien$ git clone ssh://michaelobrien@gerrit.onap.org:29418/logging-analytics Cloning into 'logging-analytics'... Warning: Permanently added '[gerrit.onap.org]:29418,[198.145.29.92]:29418' (RSA) to the list of known hosts. remote: Total 374 (delta 0), reused 374 (delta 0) Receiving objects: 100% (374/374), 1001.91 KiB | 1.22 MiB/s, done. Resolving deltas: 100% (120/120), done. obrienbiometrics:wse_sts392b michaelobrien$ cd logging-analytics/reference/ obrienbiometrics:reference michaelobrien$ ls\ logging-demo logging-library logging-mock-service pom.xml # build the project to verify obrienbiometrics:reference michaelobrien$ mvn clean install -U -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dadditionalparam=-Xdoclint:none [INFO] logging-reference .................................. SUCCESS [ 3.069 s] [INFO] logging-library .................................... SUCCESS [ 1.313 s] [INFO] logging-mock-service ............................... SUCCESS [ 0.395 s] [INFO] logging-demo ....................................... SUCCESS [ 2.329 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 9.232 s [INFO] Finished at: 2018-01-21T09:29:15-05:00
Audits
Existing EELF implementations
Please use the latest 1.0.0 from https://github.com/att/EELF - nexus artifacts pending
| app | project | repo | version |
|---|---|---|---|
| aai | aai-common | 1.0.0 | |
| aai | aai-core | 1.0.0 | |
| aai | aai-service (deprecated) | 0.0.1 (not in use) | |
| aai | aai-traversal | 1.0.0 | |
| aai | ajsc-aai | 1.0.0 | |
| aai | eelf-logging | logging-service | 0.0.1 (not in use) |
| appc | (pax.logging = wrapper on log4j, slf4j, commons-logging 1.2 and logback 1.1.1) needs this RI - provide a demo to the appc team | 0.0.1 (upgrade?) | |
| ccsdk | nbapi | ccsdk/platform/nbapi | 1.0.0 |
| dmaap | buscontroller | dmaap/buscontroller | 0.0.1 |
| ui | dcae_dmaap_webapp | dmaapbc | 1.0.0 |
| ui | dmaap-bc-app-common | 1.0.0 | |
| ui | dmaap-bc-app-parent | 1.0.0 | |
| policy | common-modules/ | common | 1.0.0 |
| policy | drools-pdp | 1.0.0 | |
| portal | ecompportal-be-common | portal | <eelf.version>1.0.0 |
| policy | feature-eelf | drools-pdp | 1.0.0 |
| policy | PolicyEngineSuite/ONAP-logging | common | 0.0.1 (not in use) |
| policy | ONAP-SDK-APP | engine | 0.0.1 (not in use) |
| sdc | ? | ||
| so | common | so | 0.0.1 (upgrade ?) |
| vid | epsdk-app-onap | vid | 1.0.0 |
| vid | vid-app-common | vid | 1.0.0 |
| open-o | |||
| multicloud | python based | MULTICLOUD-151 - Getting issue details... STATUS |
ONAP components with a Filebeat container
Projects with a filebeat container as of 20170114
ubuntu@ip-172-31-49-200:~$ kubectl get pods --all-namespaces | grep 2/2 onap-aai aai-resources-1039856271-3ph1d 2/2 Running 0 37m onap-aai aai-traversal-143081090-f4cls 2/2 Running 0 37m onap-aai model-loader-service-4144225433-0jr21 2/2 Running 0 37m onap-aai search-data-service-3842430948-53j1w 2/2 Running 0 37m onap-aai sparky-be-4222608366-966p4 2/2 Running 0 37m onap-appc appc-1828810488-mhsf7 2/2 Running 0 37m onap-mso mso-736294529-c2p8v 2/2 Running 0 37m onap-policy drools-2600956298-dp7bq 2/2 Running 0 37m onap-policy pap-466625067-n23bp 2/2 Running 0 37m onap-policy pdp-2354817903-kmxv5 2/2 Running 0 37m onap-portal portalapps-1783099045-79lcq 2/2 Running 0 37m onap-sdc sdc-be-2336519847-p6p5l 2/2 Running 0 37m onap-sdc sdc-fe-2862673798-xcpnk 2/2 Running 0 37m onap-sdnc sdnc-0 2/2 Running 0 37m onap-sdnc sdnc-dbhost-0 2/2 Running 0 37m onap-vid vid-server-3026751708-rpmq4 2/2 Running 0 37m
Current Logging Libraries
Current Framework usage impacting Logging
Logging in VNFs
Design Issues
DI 1: 20171231: Initial Maven Project Structure
https://gerrit.onap.org/r/#/admin/projects/logging-analytics
git clone ssh://michaelobrien@gerrit.onap.org:29418/logging-analytics
see
obrienbiometrics:logging-analytics michaelobrien$ git status On branch master Your branch is up-to-date with 'origin/master'. Changes to be committed: (use "git reset HEAD <file>..." to unstage) new file: reference/logging-demo/.gitignore new file: reference/logging-demo/pom.xml new file: reference/logging-demo/src/main/java/org/onap/logging/LoggingReference.java new file: reference/logging-library/.gitignore new file: reference/logging-library/pom.xml new file: reference/logging-library/src/main/java/org/onap/logging/LoggingReference.java new file: reference/pom.xml Untracked files: (use "git add <file>..." to include in what will be committed) reference/.project reference/.settings/ reference/logging-demo/.checkstyle reference/logging-demo/.classpath reference/logging-demo/.project reference/logging-demo/.settings/ reference/logging-library/.checkstyle reference/logging-library/.classpath reference/logging-library/.project reference/logging-library/.settings/ reference/target/ obrienbiometrics:logging-analytics michaelobrien$ git commit -m "Initial logging lib/demo proj struct" [master 9e42247] Initial logging lib/demo proj struct 7 files changed, 306 insertions(+) create mode 100644 reference/logging-demo/.gitignore create mode 100644 reference/logging-demo/pom.xml create mode 100644 reference/logging-demo/src/main/java/org/onap/logging/LoggingReference.java create mode 100644 reference/logging-library/.gitignore create mode 100644 reference/logging-library/pom.xml create mode 100644 reference/logging-library/src/main/java/org/onap/logging/LoggingReference.java create mode 100644 reference/pom.xml obrienbiometrics:logging-analytics michaelobrien$ git review -s obrienbiometrics:logging-analytics michaelobrien$ git commit -s --amend [master c11cd50] Initial logging lib/demo proj struct Date: Sun Dec 31 13:30:04 2017 -0500 7 files changed, 306 insertions(+) create mode 100644 reference/logging-demo/.gitignore create mode 100644 reference/logging-demo/pom.xml create mode 100644 reference/logging-demo/src/main/java/org/onap/logging/LoggingReference.java create mode 100644 reference/logging-library/.gitignore create mode 100644 reference/logging-library/pom.xml create mode 100644 reference/logging-library/src/main/java/org/onap/logging/LoggingReference.java create mode 100644 reference/pom.xml obrienbiometrics:logging-analytics michaelobrien$ git review Warning: Permanently added '[gerrit.onap.org]:29418,[198.145.29.92]:29418' (RSA) to the list of known hosts. remote: Processing changes: new: 1, refs: 1, done remote: remote: New Changes: remote: https://gerrit.onap.org/r/27195 Initial logging lib/demo proj struct remote: To ssh://gerrit.onap.org:29418/logging-analytics * [new branch] HEAD -> refs/publish/master
For eclipse you will need to make sure the spring nature in the .classpath is set correctly - in order for maven build to also work in eclipse for war projects - this will show up in the project build path
<classpathentry exported="true" kind="con" path="org.eclipse.m2e.MAVEN2_CLASSPATH_CONTAINER"> <attributes> <attribute name="maven.pomderived" value="true"/> <attribute name="org.eclipse.jst.component.nondependency" value=""/> <attribute name="org.eclipse.jst.component.dependency" value="/WEB-INF/lib"/> </attributes> </classpathentry>
DI 2: 20171231: Audit: EELF implementors
DI 3: 20171231: Audit: Mixed version of EELF dependency
LOG-110 - Getting issue details... STATUS
Figuring out the reason for the different dependencies
aai-service, ajsc-aai, common-modules/common-logging
<dependency>
<groupId>com.att.eelf</groupId>
<artifactId>eelf-core</artifactId>
<version>0.0.1</version>
</dependency>
aai-common, eelf-logging, ONAP-logging, ONAP-SDK-APP
<dependency>
<groupId>com.att.eelf</groupId>
<artifactId>eelf-core</artifactId>
<version>1.0.0</version>
</dependency>
DI 4: 20170104: ETag propagation
Passing UUID based requestID's via rest calls
https://en.wikipedia.org/wiki/HTTP_ETag
DI 5: 20170120: Docker image design
PV for logback.xml,
container for mock-log-nbi
container for mock-log-proc (the scalable TX substrate)
DockerFile
FROM tomcat:8.0.48-jre8 COPY target/logging-demo-1.2.0-SNAPSHOT.war /usr/local/tomcat/webapps/logging-demo.war
build.sh
Building, tagging and pushing the docker image so we can consume it in Kubernetes
mkdir target cp ../../logging-demo/target/*.war target docker build -t obrienlabs/logging-demo-nbi -f DockerFile . docker images | grep logging-demo-nbi docker tag obrienlabs/logging-demo-nbi obrienlabs/logging-demo-nbi:1 docker login docker push obrienlabs/logging-demo-nbi:1 #docker run -d -it --rm -p 8888:8080 obrienlabs/logging-demo-nbi:latest
DI 6: 20170120: Docker wrapper of Logging Library
LOG-136 - Getting issue details... STATUS
obrienbiometrics:docker michaelobrien$ vi DockerFile obrienbiometrics:docker michaelobrien$ docker build -t mock-logging-tomcat -f DockerFile . Sending build context to Docker daemon 18.04MB Step 1/2 : FROM tomcat:8.0.48-jre8 ---> e072422ca96f Step 2/2 : COPY logging-demo-1.2.0-SNAPSHOT.war /usr/local/tomcat/webapps/logging-demo.war ---> f0d234622af6 Successfully built f0d234622af6 Successfully tagged mock-logging-tomcat:latest obrienbiometrics:docker michaelobrien$ docker images | grep mock-logging mock-logging-tomcat latest f0d234622af6 9 seconds ago 576MB obrienbiometrics:docker michaelobrien$ docker run -d -it --rm -p 8888:8080 mock-logging-tomcat:latest ef5bdffb0e0aef9f185a51be2a915eae3d72fdb9a2125e1bb373516137c205b4 obrienbiometrics:docker michaelobrien$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ef5bdffb0e0a mock-logging-tomcat:latest "catalina.sh run" 5 seconds ago Up 4 seconds 0.0.0.0:8888->8080/tcp adoring_hawking obrienbiometrics:docker michaelobrien$ docker logs -f adoring_hawking 20-Jan-2018 04:30:26.092 INFO [localhost-startStop-1] org.springframework.web.context.ContextLoader.initWebApplicationContext Root WebApplicationContext: initialization completed in 1352 ms 20-Jan-2018 04:30:26.592 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/local/tomcat/webapps/logging-demo.war has finished in 3,652 ms 20-Jan-2018 04:30:26.841 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 3943 ms ^C obrienbiometrics:docker michaelobrien$ curl http://localhost:8888/logging-demo/rest/read/test testing: org.onap.logging.ApplicationService@433552a obrienbiometrics:docker michaelobrien$ docker build -t obrienlabs/mock-logging-tomcat -f DockerFile . Sending build context to Docker daemon 18.04MB Step 1/2 : FROM tomcat:8.0.48-jre8 ---> e072422ca96f Step 2/2 : COPY logging-demo-1.2.0-SNAPSHOT.war /usr/local/tomcat/webapps/logging-demo.war ---> Using cache ---> f0d234622af6 Successfully built f0d234622af6 Successfully tagged obrienlabs/mock-logging-tomcat:latest obrienbiometrics:docker michaelobrien$ docker tag obrienlabs/mock-logging-tomcat obrienlabs/mock-logging-tomcat:1 obrienbiometrics:docker michaelobrien$ docker login Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one. Username (michaelobrien): obrienlabs Password: Login Succeeded obrienbiometrics:docker michaelobrien$ docker push obrienlabs/mock-logging-tomcat:1 The push refers to a repository [docker.io/obrienlabs/mock-logging-tomcat] 8bb560dc792d: Pushed 7a5faefa0b46: Pushed d1a0f91a1ad8: Pushed c4d99fd86e94: Pushed 04178c827c65: Pushed c571db474d75: Pushed 93742a1ae069: Pushed 7ffe9646653b: Pushed 6ef532e39c1e: Pushed a4d7b0ac0438: Pushed 06f4de5fefea: Pushed 851f3e348c69: Pushed e27a10675c56: Pushed 1: digest: sha256:a583f08001b21635e74e205f130ffbd23d933032686345be50f39a46b537e11b size: 3048 run obrienbiometrics:docker michaelobrien$ docker run -d -it --rm -p 8888:8080 obrienlabs/mock-logging-tomcat:1 316023a8abed5911000a1c366b7c897a33abcd7e9e70f48a4e2c2611f4c98d93 obrienbiometrics:docker michaelobrien$ curl http://localhost:8888/logging-demo/rest/read/test testing: org.onap.logging.ApplicationService@75f0964b obrienbiometrics:docker michaelobrien$ test on separate machine ubuntu@ip-172-31-52-161:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ubuntu@ip-172-31-52-161:~$ docker run -d -it -p 8888:8080 obrienlabs/mock-logging-tomcat:1 Unable to find image 'obrienlabs/mock-logging-tomcat:1' locally 1: Pulling from obrienlabs/mock-logging-tomcat 723254a2c089: Pull complete abe15a44e12f: Pull complete 409a28e3cc3d: Pull complete a9511c68044a: Pull complete 9d1b16e30bc8: Pull complete 0fc5a09c9242: Pull complete d34976006493: Pull complete 3b70003f0c10: Pull complete bc7887582e2e: Pull complete d2ab4f165865: Pull complete 7a0257d5eec9: Pull complete a23d5c1d9825: Pull complete 42f73552ced4: Pull complete Digest: sha256:a583f08001b21635e74e205f130ffbd23d933032686345be50f39a46b537e11b Status: Downloaded newer image for obrienlabs/mock-logging-tomcat:1 4262ca54a570a177185b646ec92b440f37250f7ee2922e018aec5d226c5b4f2a ubuntu@ip-172-31-52-161:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4262ca54a570 obrienlabs/mock-logging-tomcat:1 "catalina.sh run" 9 seconds ago Up 8 seconds 0.0.0.0:8888->8080/tcp kickass_hoover ubuntu@ip-172-31-52-161:~$ curl http://dev.onap.info:8888/logging-demo/rest/read/test testing: org.onap.logging.ApplicationService@86a2339
DI 7: 20180126: Config for Logging Demo Containers
Implementation
Deploying the RI
WIP
LOG-137 - Getting issue details... STATUS
Deploy a Kubernetes cluster as normal via (No ONAP yet, just the K8S cluster) - ONAP on Kubernetes#InstallDocker/Rancher/Helm/Kubectl
Run the config and log pods
# clone oom git clone ssh://michaelobrien@gerrit.onap.org:29418/oom # clone logging-analytics git clone ssh://michaelobrien@gerrit.onap.org:29418/logging-analytics # ./createConfig.sh -n onap ./createAll.bash -n onap -a log # temp create config dirs ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ sudo mkdir /dockerdata-nfs/onap/logdemo ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ sudo mkdir /dockerdata-nfs/onap/logdemo/eteshare # copy the logdemo folder into the oom folder ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ cp -R ../../../logdemo/ ../ ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ ls ../logdemo/ #create the log demo ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ ./createAll.bash -n onap -a logdemo ********** Creating instance 1 of ONAP with port range 30200 and 30399 ********** Creating ONAP: ********** Creating deployments for logdemo ********** Creating namespace ********** namespace "onap-logdemo" created Creating service account ********** clusterrolebinding "onap-logdemo-admin-binding" created Creating registry secret ********** secret "onap-docker-registry-key" created Creating deployments and services ********** NAME: onap-logdemo LAST DEPLOYED: Mon Jan 29 01:07:01 2018 NAMESPACE: onap STATUS: DEPLOYED RESOURCES: ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE logdemo NodePort 10.43.195.134 <none> 8080:30299/TCP 0s ==> v1beta1/Deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE logdemo 1 1 1 0 0s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE logdemo-1873248906-7jgm9 0/1 ContainerCreating 0 0s #check ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ kubectl get pods --all-namespaces | grep logdemo onap-logdemo logdemo-4091951930-p0zcl 1/1 Running 0 43s ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ kubectl get services --all-namespaces -o wide | grep logdemo onap-logdemo logdemo NodePort 10.43.98.104 <none> 8080:30299/TCP 57s app=logdemo # test in-container and out-container ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ kubectl -n onap-logdemo exec -it logdemo-4091951930-zwd53 bash root@logdemo-4091951930-zwd53:/usr/local/tomcat# ls webapps/ ROOT docs examples host-manager logging-demo logging-demo.war manager root@logdemo-4091951930-zwd53:/usr/local/tomcat# curl http://127.0.0.1:8080/logging-demo/rest/read/test testing: org.onap.demo.logging.ApplicationService@7a4fa9b outside ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ curl http://dev.onap.info:30299/logging-demo/rest/read/test testing: org.onap.demo.logging.ApplicationService@52772382
http://dev.onap.info:30299/logging-demo/rest/read/test
Kubernetes deployment of the RI
"kind": "Deployment",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "mock-logging-tomcat",
"namespace": "default",
"selfLink": "/apis/extensions/v1beta1/namespaces/default/deployments/mock-logging-tomcat",
"uid": "740b40f3-fda1-11e7-a4b4-02e171d3f6c2",
"resourceVersion": "677",
"generation": 1,
"creationTimestamp": "2018-01-20T05:19:13Z",
"labels": {
"app": "mock-logging-tomcat",
"version": "1"
},
"annotations": {
"deployment.kubernetes.io/revision": "1"
}
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"app": "mock-logging-tomcat",
"version": "1"
}
},
"template": {
"metadata": {
"name": "mock-logging-tomcat",
"creationTimestamp": null,
"labels": {
"app": "mock-logging-tomcat",
"version": "1"
}
},
"spec": {
"containers": [
{
"name": "mock-logging-tomcat",
"image": "obrienlabs/mock-logging-tomcat:1",
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"privileged": false
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"schedulerName": "default-scheduler"
}
},
"strategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"maxUnavailable": 1,
"maxSurge": 1
}
}
},
"status": {
"observedGeneration": 1,
"replicas": 1,
"updatedReplicas": 1,
"readyReplicas": 1,
"availableReplicas": 1,
"conditions": [
{
"type": "Available",
"status": "True",
"lastUpdateTime": "2018-01-20T05:19:13Z",
"lastTransitionTime": "2018-01-20T05:19:13Z",
"reason": "MinimumReplicasAvailable",
"message": "Deployment has minimum availability."
}
]
}
}
Local Debugging via Eclipse
LOG-153 - Getting issue details... STATUS
Depending on how you set your port overrides for Tomcat 8.x
http://localhost:8988/logging-demo/rest/read/test
Testing
http://127.0.0.1:8080/logging-demo/rest/read/test
Links
EELF
git clone https://github.com/att/EELF.git
https://tomcat.apache.org/download-80.cgi
https://spring.io/tools/sts/all
https://www.sitepoint.com/inside-java-9-part-ii/#performanceimprovements