Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 38 Next »

WIP

This page is the design and reference site for the Logging RI. The following topics are being provided to the rest of the ONAP community.

  • How to use the  EELF Logging Library
  • How to adhere to the logging guidelines
  • How to implement the filebeat sidecar container to push logs to the ELK stack pod
  • How to use the EELF AOP wrapping library
  • How to ensure requestID tracking during distributed transactions

Questions on onap-discuss from the teams
https://lists.onap.org/pipermail/onap-discuss/2018-January/007415.html

LOG-118 - Getting issue details... STATUS

Purpose

The goals of the logging project are the following

provide for standardized logs across all ONAP components

provide ELK framework for log capture, indexing and presentation/search

provide framework for requestID tracking of distributed transactions (correlation ID's in p.162 of Building Microservices 2015 by Sam Newman) via HTTP ETags

Architecture

Microservices

NBI - Northbound Interface (1/1)

Logging Demo WAR

PN - Processing Node (scaled N/N)

Processing Node WAR


Project Structure

ProjectrepoDependenciesArtifactDocker
logging-referencelogging-reference
pom
logging-demologging-demo
WAR
logging-mock-servicelogging-mock-servicelogging-libraryWAR
Logging Librarylogging-library
JAR
Kubernetes demo speckubernetes/log-mock-demo
yaml, config
Kubernetes service speckubernetes/log-mock-service
yaml, config

logging-docker-root
pom

logging-docker-demologging-demoDockerFile

logging-docker-servicelogging-mock-serviceDockerFile


API

Deployment Diagram

Quickstart

WIP (as of 20180121) - will notify when complete 

Goto your IDE workspace

Clone the logging repo

https://gerrit.onap.org/r/#/admin/projects/logging-analytics

https://git.onap.org/logging-analytics/tree/

goto the reference folder and load the pom.xml project

obrienbiometrics:wse_sts392b michaelobrien$ git clone ssh://michaelobrien@gerrit.onap.org:29418/logging-analytics
Cloning into 'logging-analytics'...
Warning: Permanently added '[gerrit.onap.org]:29418,[198.145.29.92]:29418' (RSA) to the list of known hosts.
remote: Total 374 (delta 0), reused 374 (delta 0)
Receiving objects: 100% (374/374), 1001.91 KiB | 1.22 MiB/s, done.
Resolving deltas: 100% (120/120), done.
obrienbiometrics:wse_sts392b michaelobrien$ cd logging-analytics/reference/
obrienbiometrics:reference michaelobrien$ ls\
logging-demo		logging-library		logging-mock-service	pom.xml


# build the project to verify
obrienbiometrics:reference michaelobrien$ mvn clean install -U -DskipTests=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dadditionalparam=-Xdoclint:none
[INFO] logging-reference .................................. SUCCESS [  3.069 s]
[INFO] logging-library .................................... SUCCESS [  1.313 s]
[INFO] logging-mock-service ............................... SUCCESS [  0.395 s]
[INFO] logging-demo ....................................... SUCCESS [  2.329 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.232 s
[INFO] Finished at: 2018-01-21T09:29:15-05:00





Audits

Existing EELF implementations

Please use the latest 1.0.0 from https://github.com/att/EELF - nexus artifacts pending

appprojectrepoversion
aaiaai-common
1.0.0
aaiaai-core
1.0.0
aaiaai-service (deprecated)
0.0.1 (not in use)
aaiaai-traversal
1.0.0
aaiajsc-aai
1.0.0
aaieelf-logginglogging-service0.0.1 (not in use)
appc

(pax.logging = wrapper on log4j, slf4j, commons-logging 1.2 and logback 1.1.1)

needs this RI - provide a demo to the appc team


0.0.1 (upgrade?)
ccsdknbapiccsdk/platform/nbapi1.0.0
dmaapbuscontrollerdmaap/buscontroller0.0.1
uidcae_dmaap_webappdmaapbc1.0.0
uidmaap-bc-app-common
1.0.0
uidmaap-bc-app-parent
1.0.0
policycommon-modules/common1.0.0
policydrools-pdp
1.0.0
portalecompportal-be-commonportal
<eelf.version>1.0.0
</eelf.version>
policyfeature-eelfdrools-pdp1.0.0
policyPolicyEngineSuite/ONAP-loggingcommon0.0.1 (not in use)
policyONAP-SDK-APPengine0.0.1 (not in use)
sdc

?
socommonso0.0.1 (upgrade ?)
videpsdk-app-onapvid1.0.0
vidvid-app-commonvid1.0.0
open-o


multicloudpython based

MULTICLOUD-151 - Getting issue details... STATUS

ONAP components with a Filebeat container

Projects with a filebeat container as of 20170114

ubuntu@ip-172-31-49-200:~$ kubectl get pods --all-namespaces | grep 2/2
onap-aai              aai-resources-1039856271-3ph1d                2/2       Running        0          37m
onap-aai              aai-traversal-143081090-f4cls                 2/2       Running        0          37m
onap-aai              model-loader-service-4144225433-0jr21         2/2       Running        0          37m
onap-aai              search-data-service-3842430948-53j1w          2/2       Running        0          37m
onap-aai              sparky-be-4222608366-966p4                    2/2       Running        0          37m
onap-appc             appc-1828810488-mhsf7                         2/2       Running        0          37m
onap-mso              mso-736294529-c2p8v                           2/2       Running        0          37m
onap-policy           drools-2600956298-dp7bq                       2/2       Running        0          37m
onap-policy           pap-466625067-n23bp                           2/2       Running        0          37m
onap-policy           pdp-2354817903-kmxv5                          2/2       Running        0          37m
onap-portal           portalapps-1783099045-79lcq                   2/2       Running        0          37m
onap-sdc              sdc-be-2336519847-p6p5l                       2/2       Running        0          37m
onap-sdc              sdc-fe-2862673798-xcpnk                       2/2       Running        0          37m
onap-sdnc             sdnc-0                                        2/2       Running        0          37m
onap-sdnc             sdnc-dbhost-0                                 2/2       Running        0          37m
onap-vid              vid-server-3026751708-rpmq4                   2/2       Running        0          37m

Current Logging Libraries

Current Framework usage impacting Logging

Logging in VNFs

Design Issues

DI 1: 20171231: Initial Maven Project Structure

https://gerrit.onap.org/r/#/admin/projects/logging-analytics

git clone ssh://michaelobrien@gerrit.onap.org:29418/logging-analytics

see

obrienbiometrics:logging-analytics michaelobrien$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)
	new file:   reference/logging-demo/.gitignore
	new file:   reference/logging-demo/pom.xml
	new file:   reference/logging-demo/src/main/java/org/onap/logging/LoggingReference.java
	new file:   reference/logging-library/.gitignore
	new file:   reference/logging-library/pom.xml
	new file:   reference/logging-library/src/main/java/org/onap/logging/LoggingReference.java
	new file:   reference/pom.xml
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	reference/.project
	reference/.settings/
	reference/logging-demo/.checkstyle
	reference/logging-demo/.classpath
	reference/logging-demo/.project
	reference/logging-demo/.settings/
	reference/logging-library/.checkstyle
	reference/logging-library/.classpath
	reference/logging-library/.project
	reference/logging-library/.settings/
	reference/target/
obrienbiometrics:logging-analytics michaelobrien$ git commit -m "Initial logging lib/demo proj struct"
[master 9e42247] Initial logging lib/demo proj struct
 7 files changed, 306 insertions(+)
 create mode 100644 reference/logging-demo/.gitignore
 create mode 100644 reference/logging-demo/pom.xml
 create mode 100644 reference/logging-demo/src/main/java/org/onap/logging/LoggingReference.java
 create mode 100644 reference/logging-library/.gitignore
 create mode 100644 reference/logging-library/pom.xml
 create mode 100644 reference/logging-library/src/main/java/org/onap/logging/LoggingReference.java
 create mode 100644 reference/pom.xml
obrienbiometrics:logging-analytics michaelobrien$ git review -s
obrienbiometrics:logging-analytics michaelobrien$ git commit -s --amend
[master c11cd50] Initial logging lib/demo proj struct
 Date: Sun Dec 31 13:30:04 2017 -0500
 7 files changed, 306 insertions(+)
 create mode 100644 reference/logging-demo/.gitignore
 create mode 100644 reference/logging-demo/pom.xml
 create mode 100644 reference/logging-demo/src/main/java/org/onap/logging/LoggingReference.java
 create mode 100644 reference/logging-library/.gitignore
 create mode 100644 reference/logging-library/pom.xml
 create mode 100644 reference/logging-library/src/main/java/org/onap/logging/LoggingReference.java
 create mode 100644 reference/pom.xml
obrienbiometrics:logging-analytics michaelobrien$ git review
Warning: Permanently added '[gerrit.onap.org]:29418,[198.145.29.92]:29418' (RSA) to the list of known hosts.
remote: Processing changes: new: 1, refs: 1, done            
remote: 
remote: New Changes:        
remote:   https://gerrit.onap.org/r/27195 Initial logging lib/demo proj struct        
remote: 
To ssh://gerrit.onap.org:29418/logging-analytics
 * [new branch]      HEAD -> refs/publish/master

For eclipse you will need to make sure the spring nature in the .classpath is set correctly - in order for maven build to also work in eclipse for war projects - this will show up in the project build path

	<classpathentry exported="true" kind="con" path="org.eclipse.m2e.MAVEN2_CLASSPATH_CONTAINER">
		<attributes>
			<attribute name="maven.pomderived" value="true"/>
			<attribute name="org.eclipse.jst.component.nondependency" value=""/>
			<attribute name="org.eclipse.jst.component.dependency" value="/WEB-INF/lib"/>
		</attributes>
	</classpathentry>


DI 2: 20171231: Audit: EELF implementors


DI 3: 20171231: Audit: Mixed version of EELF dependency

LOG-110 - Getting issue details... STATUS

Figuring out the reason for the different dependencies

aai-service, ajsc-aai, common-modules/common-logging

<dependency> 

<groupId>com.att.eelf</groupId> 

<artifactId>eelf-core</artifactId> 

<version>0.0.1</version> 

</dependency> 


aai-common, eelf-logging, ONAP-logging, ONAP-SDK-APP

        <dependency>

            <groupId>com.att.eelf</groupId>

            <artifactId>eelf-core</artifactId>

            <version>1.0.0</version>

        </dependency>

DI 4: 20170104: ETag propagation

Passing UUID based requestID's via rest calls

https://en.wikipedia.org/wiki/HTTP_ETag

DI 5: 20170120: Docker image design

PV for logback.xml,

container for mock-log-nbi

container for mock-log-proc (the scalable TX substrate)

DockerFile

FROM tomcat:8.0.48-jre8
COPY target/logging-demo-1.2.0-SNAPSHOT.war /usr/local/tomcat/webapps/logging-demo.war

build.sh

Building, tagging and pushing the docker image so we can consume it in Kubernetes

mkdir target
cp ../../logging-demo/target/*.war target
docker build -t obrienlabs/logging-demo-nbi -f DockerFile .
docker images | grep logging-demo-nbi
docker tag obrienlabs/logging-demo-nbi obrienlabs/logging-demo-nbi:1
docker login
docker push obrienlabs/logging-demo-nbi:1
#docker run -d -it --rm -p 8888:8080 obrienlabs/logging-demo-nbi:latest


DI 6: 20170120: Docker wrapper of Logging Library

LOG-136 - Getting issue details... STATUS

obrienbiometrics:docker michaelobrien$ vi DockerFile
obrienbiometrics:docker michaelobrien$ docker build -t mock-logging-tomcat -f DockerFile .
Sending build context to Docker daemon  18.04MB
Step 1/2 : FROM tomcat:8.0.48-jre8
---> e072422ca96f
Step 2/2 : COPY logging-demo-1.2.0-SNAPSHOT.war /usr/local/tomcat/webapps/logging-demo.war
---> f0d234622af6
Successfully built f0d234622af6
Successfully tagged mock-logging-tomcat:latest
obrienbiometrics:docker michaelobrien$ docker images | grep mock-logging
mock-logging-tomcat                                           latest                            f0d234622af6        9 seconds ago       576MB
obrienbiometrics:docker michaelobrien$ docker run -d -it --rm -p 8888:8080 mock-logging-tomcat:latest
ef5bdffb0e0aef9f185a51be2a915eae3d72fdb9a2125e1bb373516137c205b4
obrienbiometrics:docker michaelobrien$ docker ps
CONTAINER ID        IMAGE                        COMMAND             CREATED             STATUS              PORTS                    NAMES
ef5bdffb0e0a        mock-logging-tomcat:latest   "catalina.sh run"   5 seconds ago       Up 4 seconds        0.0.0.0:8888->8080/tcp   adoring_hawking
obrienbiometrics:docker michaelobrien$ docker logs -f adoring_hawking

20-Jan-2018 04:30:26.092 INFO [localhost-startStop-1] org.springframework.web.context.ContextLoader.initWebApplicationContext Root WebApplicationContext: initialization completed in 1352 ms
20-Jan-2018 04:30:26.592 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/local/tomcat/webapps/logging-demo.war has finished in 3,652 ms
20-Jan-2018 04:30:26.841 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 3943 ms
^C
obrienbiometrics:docker michaelobrien$ curl http://localhost:8888/logging-demo/rest/read/test
testing: org.onap.logging.ApplicationService@433552a
obrienbiometrics:docker michaelobrien$ docker build -t obrienlabs/mock-logging-tomcat -f DockerFile .
Sending build context to Docker daemon  18.04MB
Step 1/2 : FROM tomcat:8.0.48-jre8
---> e072422ca96f
Step 2/2 : COPY logging-demo-1.2.0-SNAPSHOT.war /usr/local/tomcat/webapps/logging-demo.war
---> Using cache
---> f0d234622af6
Successfully built f0d234622af6
Successfully tagged obrienlabs/mock-logging-tomcat:latest
obrienbiometrics:docker michaelobrien$ docker tag obrienlabs/mock-logging-tomcat obrienlabs/mock-logging-tomcat:1
obrienbiometrics:docker michaelobrien$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username (michaelobrien): obrienlabs
Password:
Login Succeeded
obrienbiometrics:docker michaelobrien$ docker push obrienlabs/mock-logging-tomcat:1
The push refers to a repository [docker.io/obrienlabs/mock-logging-tomcat]
8bb560dc792d: Pushed
7a5faefa0b46: Pushed
d1a0f91a1ad8: Pushed
c4d99fd86e94: Pushed
04178c827c65: Pushed
c571db474d75: Pushed
93742a1ae069: Pushed
7ffe9646653b: Pushed
6ef532e39c1e: Pushed
a4d7b0ac0438: Pushed
06f4de5fefea: Pushed
851f3e348c69: Pushed
e27a10675c56: Pushed
1: digest: sha256:a583f08001b21635e74e205f130ffbd23d933032686345be50f39a46b537e11b size: 3048

run

obrienbiometrics:docker michaelobrien$ docker run -d -it --rm -p 8888:8080 obrienlabs/mock-logging-tomcat:1
316023a8abed5911000a1c366b7c897a33abcd7e9e70f48a4e2c2611f4c98d93
obrienbiometrics:docker michaelobrien$ curl http://localhost:8888/logging-demo/rest/read/test
testing: org.onap.logging.ApplicationService@75f0964b
obrienbiometrics:docker michaelobrien$


test on separate machine

ubuntu@ip-172-31-52-161:~$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
ubuntu@ip-172-31-52-161:~$ docker run -d -it  -p 8888:8080 obrienlabs/mock-logging-tomcat:1
Unable to find image 'obrienlabs/mock-logging-tomcat:1' locally
1: Pulling from obrienlabs/mock-logging-tomcat
723254a2c089: Pull complete
abe15a44e12f: Pull complete
409a28e3cc3d: Pull complete
a9511c68044a: Pull complete
9d1b16e30bc8: Pull complete
0fc5a09c9242: Pull complete
d34976006493: Pull complete
3b70003f0c10: Pull complete
bc7887582e2e: Pull complete
d2ab4f165865: Pull complete
7a0257d5eec9: Pull complete
a23d5c1d9825: Pull complete
42f73552ced4: Pull complete
Digest: sha256:a583f08001b21635e74e205f130ffbd23d933032686345be50f39a46b537e11b
Status: Downloaded newer image for obrienlabs/mock-logging-tomcat:1
4262ca54a570a177185b646ec92b440f37250f7ee2922e018aec5d226c5b4f2a
ubuntu@ip-172-31-52-161:~$ docker ps
CONTAINER ID        IMAGE                              COMMAND             CREATED             STATUS              PORTS                    NAMES
4262ca54a570        obrienlabs/mock-logging-tomcat:1   "catalina.sh run"   9 seconds ago       Up 8 seconds        0.0.0.0:8888->8080/tcp   kickass_hoover
ubuntu@ip-172-31-52-161:~$ curl http://dev.onap.info:8888/logging-demo/rest/read/test
testing: org.onap.logging.ApplicationService@86a2339

DI 7: 20180126: Config for Logging Demo Containers

OOM-628 - Getting issue details... STATUS

DI 8: 20180204: NodePort port reservation framework required

OOM-656 - Getting issue details... STATUS


Implementation

Deploying the RI

WIP

LOG-137 - Getting issue details... STATUS

Deploy a Kubernetes cluster as normal via (No ONAP yet, just the K8S cluster) - ONAP on Kubernetes#InstallDocker/Rancher/Helm/Kubectl

Run the config and log pods

# clone oom
git clone ssh://michaelobrien@gerrit.onap.org:29418/oom
# clone logging-analytics
git clone ssh://michaelobrien@gerrit.onap.org:29418/logging-analytics

# 
./createConfig.sh -n onap
./createAll.bash -n onap -a log
# temp create config dirs
ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ sudo mkdir /dockerdata-nfs/onap/logdemo
ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ sudo mkdir /dockerdata-nfs/onap/logdemo/eteshare
# copy the logdemo folder into the oom folder
root@obriensystemsu0:~/logging-analytics/reference/logging-kubernetes# cp -R logdemo ~/oom/kubernetes/
# make sure the 3 ELK log containers are up
root@obriensystemsu0:~/oom/kubernetes/oneclick# ./createAll.bash -n onap -a log
root@obriensystemsu0:~/oom/kubernetes/oneclick# kubectl get pods --all-namespaces -a
onap-log      elasticsearch-3138115406-mlwjf         1/1       Running     0          12m
onap-log      kibana-3372627750-260f2                1/1       Running     0          12m
onap-log      logstash-1708188010-bhlqf              1/1       Running     0          12m

# create the log demo
root@obriensystemsu0:~/oom/kubernetes/oneclick# ./createAll.bash -n onap -a logdemo

********** Creating instance 1 of ONAP with port range 30200 and 30399

********** Creating ONAP: 


********** Creating deployments for logdemo ********** 

Creating namespace **********
namespace "onap-logdemo" created

Creating service account **********
clusterrolebinding "onap-logdemo-admin-binding" created

Creating registry secret **********
secret "onap-docker-registry-key" created

Creating deployments and services **********
NAME:   onap-logdemo
LAST DEPLOYED: Mon Jan 29 11:25:44 2018
NAMESPACE: onap
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME     TYPE      CLUSTER-IP   EXTERNAL-IP  PORT(S)         AGE
logdemo  NodePort  10.43.26.45  <none>       8080:30299/TCP  0s

==> v1beta1/Deployment
NAME     DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
logdemo  1        1        1           0          0s

==> v1/Pod(related)
NAME                      READY  STATUS             RESTARTS  AGE
logdemo-3133092390-g4qx7  0/1    ContainerCreating  0         0s





#check
root@obriensystemsu0:~/oom/kubernetes/oneclick#  kubectl get pods --all-namespaces | grep logdemo
onap-logdemo   logdemo-3133092390-g4qx7               1/1       Running   0          5m
root@obriensystemsu0:~/oom/kubernetes/oneclick# kubectl get services  --all-namespaces -o wide | grep logdemo
onap-logdemo   logdemo                NodePort    10.43.26.45     <none>        8080:30299/TCP   6m        app=logdemo


root@obriensystemsu0:~/logging-analytics/reference# kubectl -n onap-logdemo logs -f logdemo-3133092390-g4qx7
29-Jan-2018 17:28:03.456 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 11584 ms

# test in-container and out-container
ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ kubectl -n onap-logdemo exec -it logdemo-4091951930-zwd53 bash
root@logdemo-4091951930-zwd53:/usr/local/tomcat# ls webapps/
ROOT  docs  examples  host-manager  logging-demo  logging-demo.war  manager

root@logdemo-4091951930-zwd53:/usr/local/tomcat# curl http://127.0.0.1:8080/logging-demo/rest/read/test
testing: org.onap.demo.logging.ApplicationService@7a4fa9b

outside
ubuntu@ip-172-31-52-161:~/oom/kubernetes/oneclick$ curl http://dev.onap.info:30299/logging-demo/rest/read/test
testing: org.onap.demo.logging.ApplicationService@52772382


http://dev.onap.info:30299/logging-demo/rest/read/test

Kubernetes deployment of the RI

"kind": "Deployment",
"apiVersion": "extensions/v1beta1",
"metadata": {
"name": "mock-logging-tomcat",
"namespace": "default",
"selfLink": "/apis/extensions/v1beta1/namespaces/default/deployments/mock-logging-tomcat",
"uid": "740b40f3-fda1-11e7-a4b4-02e171d3f6c2",
"resourceVersion": "677",
"generation": 1,
"creationTimestamp": "2018-01-20T05:19:13Z",
"labels": {
"app": "mock-logging-tomcat",
"version": "1"
},
"annotations": {
"deployment.kubernetes.io/revision": "1"
}
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"app": "mock-logging-tomcat",
"version": "1"
}
},
"template": {
"metadata": {
"name": "mock-logging-tomcat",
"creationTimestamp": null,
"labels": {
"app": "mock-logging-tomcat",
"version": "1"
}
},
"spec": {
"containers": [
{
"name": "mock-logging-tomcat",
"image": "obrienlabs/mock-logging-tomcat:1",
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent",
"securityContext": {
"privileged": false
}
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"schedulerName": "default-scheduler"
}
},
"strategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"maxUnavailable": 1,
"maxSurge": 1
}
}
},
"status": {
"observedGeneration": 1,
"replicas": 1,
"updatedReplicas": 1,
"readyReplicas": 1,
"availableReplicas": 1,
"conditions": [
{
"type": "Available",
"status": "True",
"lastUpdateTime": "2018-01-20T05:19:13Z",
"lastTransitionTime": "2018-01-20T05:19:13Z",
"reason": "MinimumReplicasAvailable",
"message": "Deployment has minimum availability."
}
]
}
}

Local Debugging via Eclipse

LOG-153 - Getting issue details... STATUS

Depending on how you set your port overrides for Tomcat 8.x

http://localhost:8988/logging-demo/rest/read/test

Testing

http://127.0.0.1:8080/logging-demo/rest/read/test

Links

EELF

git clone https://github.com/att/EELF.git

https://tomcat.apache.org/download-80.cgi

https://spring.io/tools/sts/all

https://www.sitepoint.com/inside-java-9-part-ii/#performanceimprovements




  • No labels