Any LF showstopper?
Nexus3.onap.org:10001 experiencing a serious routing? issue not the older 3 hour slowdown from 4-6 months ago that was fixed. Normally prepull takes 30 min - it now takes 120+ hours for a full prepull since 20181217 - and 80x slowdown on image downloads.
Q) why jenkins has no issue with nexus3 - A) they are on the same domain - and don't go through a bell.ca exchange
| Code Block |
|---|
|
;; ANSWER SECTION:
jenkins.onap.org. 60 IN CNAME cloud.onap.org.
cloud.onap.org. 60 IN A 199.204.45.137
;; ANSWER SECTION:
nexus3.onap.org. 60 IN CNAME cloud.onap.org.
cloud.onap.org. 47 IN A 199.204.45.137 |
run a traceroute and notice
| Code Block |
|---|
|
# this is from an AWS EC2 instance in us-east-2
ubuntu@ip-172-31-10-98:~$ traceroute nexus3.onap.org
traceroute to nexus3.onap.org (199.204.45.137), 30 hops max, 60 byte packets
16 tcore3-ashburnbk_hundredgige0-1-0-0.net.bell.ca (64.230.125.182) 25.657 ms tcore4-ashburnbk_hundredgige0-1-0-0.net.bell.ca (64.230.125.184) 30.723 ms 30.673 ms |
discussions/helpdesk tickets
https://lists.onap.org/g/onap-discuss/topic/aaf_cm_image_not_found/28803606?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,28803606
https://lists.onap.org/g/onap-discuss/topic/onap_helpdesk_65866/28800220?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,28800220
https://lists.onap.org/g/onap-discuss/topic/onap_helpdesk_65794_nexus3/28794897?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,20,28794897
Effect: anyone bringing up a clean ONAP system casablanca, master, 3.0.0-ONAP - all will take 35+ hours to come up depending on what is deployed - for example LOG uses dockerhub images it will come up fast - but AAF or any other pod that has images over 1G will each take a couple hours. Once casablanca is pulled you are good - they don't change - but for master - everytime you redeploy on a different day - all or some part of the images will need to be pulled.
Temp Workaround
nexus3.onap.cloud:5000 alternate proxy (have the LF's back when on vacation) up on 20181218 - taking 2 days to saturate with casablanca images 128G FS ETA late Friday - as of 20h of pulliing - see 21 images of https://git.onap.org/integration/tree/version-manifest/src/main/resources/docker-manifest.csv?h=casablanca
20181219:1700EDT nexus3.onap.cloud status : 16h of pulls - 26 images
| Code Block |
|---|
|
ubuntu@a-nexus3:~$ sudo docker images | wc -l
26
ubuntu@a-nexus3:~$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nexus3.onap.cloud:5000/onap/aai-resources 1.3.4 723d184670e2 3 weeks ago 515 MB
nexus3.onap.cloud:5000/onap/aai-graphadmin 1.0.1 ed643f4a192c 4 weeks ago 526 MB
nexus3.onap.cloud:5000/onap/aai-traversal 1.3.3 4e5784b9e283 4 weeks ago 526 MB
nexus3.onap.cloud:5000/onap/appc-cdt-image 1.4.3 10b6b253e1a9 4 weeks ago 160 MB
nexus3.onap.cloud:5000/onap/appc-image 1.4.3 888018330bf5 4 weeks ago 2.88 GB
nexus3.onap.cloud:5000/onap/babel 1.3.2 b4012e79495e 4 weeks ago 625 MB
nexus3.onap.cloud:5000/onap/aaf/aaf_service 2.1.8 6eb295fed110 5 weeks ago 1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_oauth 2.1.8 74dcdce76094 5 weeks ago 1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_locate 2.1.8 2a4eaa6275ff 5 weeks ago 1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_hello 2.1.8 495a01176053 5 weeks ago 1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_gui 2.1.8 8caa6dc681f0 5 weeks ago 1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_fs 2.1.8 3d663698534d 5 weeks ago 1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_cm 2.1.8 0ba25c4ec3fb 5 weeks ago 1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_agent 2.1.8 090b326a7f11 5 weeks ago 1.14 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_config 2.1.8 6506ac785cb5 5 weeks ago 1.14 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_cass 2.1.8 4b91e9b0b43f 5 weeks ago 323 MB
nexus3.onap.cloud:5000/onap/aaf/smsquorumclient 3.0.1 f8cf701eadc3 7 weeks ago 18.2 MB
nexus3.onap.cloud:5000/onap/aaf/sms 3.0.1 02363fccc6c7 7 weeks ago 35.4 MB
nexus3.onap.cloud:5000/onap/aai/esr-server 1.2.1 00c9c28e8936 7 weeks ago 521 MB
nexus3.onap.cloud:5000/onap/aai/esr-gui 1.2.1 4bd7ab7ae54a 7 weeks ago 512 MB
nexus3.onap.cloud:5000/onap/aaf/testcaservice 3.0.0 fc717d0b071c 2 months ago 1.17 GB
nexus3.onap.cloud:5000/onap/aaf/abrmd 3.0.0 00a91d2dc09d 2 months ago 1.15 GB
nexus3.onap.cloud:5000/onap/aaf/distcenter 3.0.0 d8d9137ef2d3 2 months ago 1.09 GB
nexus3.onap.cloud:5000/onap/aai-cacher 1.0.0 8ec3df246a35 2 months ago 466 MB |
slightly-not-useful-random graphs
for reference my private AWS proxy after 46h of pulls has 60 images of 180 - predicting 140h or 5.8 days
| Code Block |
|---|
|
ubuntu@ip-172-31-10-98:~$ sudo docker images | wc -l
52
# at
nexus3.onap.info:5000/onap/externalapi/nbi 3.0.1 1ebc02237c1c 2 months ago 122 MB |
similar incoming traffic profile from nexus3.onap.org
nexus4.onap.cloud:5000 alternate proxy for master - larger FS 200G
- access instructions, cert, installation on Cloud Native Deployment
I take this tooooo seriously