Add Nexus proxy procedure for ONAP deployments - LF Nexus3 is timing out periodically
Description
Attachments
- 22 Mar 2018, 09:03 PM
is blocked by
Confluence content
Activity
Former user December 18, 2018 at 6:36 PMEdited
Stats today on nexus3.onap.org - small images work fine but 1G images like ccsdk are taking 10+ min on the last 2 layers
ubuntu@ip-172-31-10-98:~$ sudo docker pull nexus3.onap.org:10001/onap/aaf/sms:3.0.1
3.0.1: Pulling from onap/aaf/sms
c67f3896b22c: Pull complete
134fac3103be: Pull complete
c7fdcaf7f439: Pull complete
590a963725eb: Pull complete
4df2c296b645: Pull complete
15dd1c6ebaf1: Pull complete
76eeb922b789: Pull complete
Digest: sha256:d5b64947edb93848acacaa9820234aa29e58217db9f878886b7bafae00fdb436
Status: Downloaded newer image for nexus3.onap.org:10001/onap/aaf/sms:3.0.1
ubuntu@ip-172-31-10-98:~$ sudo docker pull nexus3.onap.org:10001/onap/ccsdk-dgbuilder-image:0.3.2
0.3.2: Pulling from onap/ccsdk-dgbuilder-image
18d680d61657: Already exists
0addb6fece63: Already exists
78e58219b215: Already exists
eb6959a66df2: Already exists
4eeb11d61ff9: Downloading [> ] 5.266 MB/345.8 MB
0f81686fe963: Downloading [===> ] 5.266 MB/71.15 MB
4 min later
ubuntu@ip-172-31-10-98:~$ sudo docker pull nexus3.onap.org:10001/onap/ccsdk-dgbuilder-image:0.3.2
0.3.2: Pulling from onap/ccsdk-dgbuilder-image
18d680d61657: Already exists
0addb6fece63: Already exists
78e58219b215: Already exists
eb6959a66df2: Already exists
4eeb11d61ff9: Downloading [=> ] 10.54 MB/345.8 MB
0f81686fe963: Downloading [=======> ] 10.54 MB/71.15 MBFormer user December 18, 2018 at 6:20 PM
implementing as per
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy
buntu@ip-172-31-10-98:~$ sudo ps -ef | grep docker
root 4022 1 0 16:53 ? 00:00:25 /usr/bin/dockerd -H fd://
root 4030 4022 0 16:53 ? 00:00:03 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 4363 4022 0 17:05 ? 00:00:06 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5000 -container-ip 172.17.0.2 -container-port 5000
root 4367 4030 0 17:05 ? 00:00:00 docker-containerd-shim 7f9b0e97eb7f484b75629238e2b4ba01acecdaed70ac3c1e7e58f37d0061ff3c /var/run/docker/libcontainerd/7f9b0e97eb7f484b75629238e2b4ba01acecdaed70ac3c1e7e58f37d0061ff3c docker-runc
root 4384 4367 0 17:05 ? 00:00:28 registry serve /etc/docker/registry/config.yml
root 4602 1434 0 18:18 pts/0 00:00:00 sudo nohup ./docker_prepull.sh
root 4603 4602 0 18:18 pts/0 00:00:00 /bin/bash ./docker_prepull.sh
root 4633 4603 0 18:18 pts/0 00:00:00 sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8
root 4634 4633 0 18:18 pts/0 00:00:00 docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8
ubuntu 4670 1434 0 18:18 pts/0 00:00:00 grep --color=auto dockerFormer user August 16, 2018 at 11:04 PM
Just use the registry in docker - as Gary has posted
https://lists.onap.org/g/onap-discuss/topic/setting_up_local_repository/24568685?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,24568685
apt-get install docker.io
docker run -d -p 5000:5000 --restart=unless-stopped --name registry -e REGISTRY_PROXY_REMOTEURL=https://nexus3.onap.org:10001 registry:2
Former user April 13, 2018 at 8:31 PM
No, but as of yesterday - it directly affects OOM healthcheck - as both cd systems had 0/33 healthcheck during the latest nexus3 slowdown - the TSC is questioning the stability of the deploy.
Ran a pullPolicy: Never to get a better baseline
we need to drive the best practice for this issue
Nexus3.onap.org has been experiencing a slowdown since 10am yesterday.
There are 2 ways around this – use the prepull script and set pullPolicy: to Never (I am currently testing IfNotPresent) – in all values.yaml in amsterdam or the root oom/kubernetes/onap/values.yaml in master/Beijing.
The other way is to swap out the nexus3 repo with a local nexus proxy
Former user March 28, 2018 at 3:07 PM
@Former user is this an OOM issue?
Details
Assignee
Former userFormer user(Deactivated)Reporter
Former userFormer user(Deactivated)Labels
Sprint
NoneFix versions
Priority
Medium
Details
Details
Assignee
Reporter
Labels
Sprint
Fix versions
Priority
Medium
Provide a template for bringing up a local nexus repo so we can avoid the usually 30 min pull of the 90 containers stretching to 2+ hours
There are 3+ repos
nexus3
dockerhub for oomk8s and aai
:plus: 20181218
implementing as per
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy
https://lists.onap.org/g/onap-discuss/topic/onap_helpdesk_65794_nexus3/28794221?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,28794221
buntu@ip-172-31-10-98:~$ sudo ps -ef | grep docker root 4022 1 0 16:53 ? 00:00:25 /usr/bin/dockerd -H fd:// root 4030 4022 0 16:53 ? 00:00:03 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc root 4363 4022 0 17:05 ? 00:00:06 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5000 -container-ip 172.17.0.2 -container-port 5000 root 4367 4030 0 17:05 ? 00:00:00 docker-containerd-shim 7f9b0e97eb7f484b75629238e2b4ba01acecdaed70ac3c1e7e58f37d0061ff3c /var/run/docker/libcontainerd/7f9b0e97eb7f484b75629238e2b4ba01acecdaed70ac3c1e7e58f37d0061ff3c docker-runc root 4384 4367 0 17:05 ? 00:00:28 registry serve /etc/docker/registry/config.yml root 4602 1434 0 18:18 pts/0 00:00:00 sudo nohup ./docker_prepull.sh root 4603 4602 0 18:18 pts/0 00:00:00 /bin/bash ./docker_prepull.sh root 4633 4603 0 18:18 pts/0 00:00:00 sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 root 4634 4633 0 18:18 pts/0 00:00:00 docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 ubuntu 4670 1434 0 18:18 pts/0 00:00:00 grep --color=auto docker