Duration 90 minutes
Agenda Items | Presented By | Time | Notes/Links | JIRA Task |
|---|---|---|---|---|
| Casablanca Maintenance Release | 5 mins | |||
| Any Infrastructure Improvement/Plan | Linux Foundation | 10 mins | Any LF showstopper? #65866 - Nexus3 proxy verified 80-100x faster downloads since 20181217 #65794 - Nexus3 timing out - still getting 0.4MB/sec - not the usual 10+MB/sec #65809 - Nexus3 slowdown 10X - docker pulls very slow in openlab example normal speed on the nexux3.onap.info:5000 proxy example pull - 5 sec sudo docker pull nexus3.onap.info:5000/onap/appc-cdt-image:1.4.3 this will take minutes sudo docker pull nexus3.onap.org:10001/onap/appc-cdt-image:1.4.3 Nexus3.onap.org:10001 experiencing a serious routing? issue not the older 3 hour slowdown from 4-6 months ago that was fixed. Normally prepull takes 30 min - it now takes 120+ hours for a full prepull since 20181217 - and 80x slowdown on image downloads. Q) why jenkins has no issue with nexus3 - A) they are on the same domain - and don't go through a bell.ca exchange ;; ANSWER SECTION: jenkins.onap.org. 60 IN CNAME cloud.onap.org. cloud.onap.org. 60 IN A 199.204.45.137 ;; ANSWER SECTION: nexus3.onap.org. 60 IN CNAME cloud.onap.org. cloud.onap.org. 47 IN A 199.204.45.137 run a traceroute and notice # this is from an AWS EC2 instance in us-east-2 ubuntu@ip-172-31-10-98:~$ traceroute nexus3.onap.org traceroute to nexus3.onap.org (199.204.45.137), 30 hops max, 60 byte packets 16 tcore3-ashburnbk_hundredgige0-1-0-0.net.bell.ca (64.230.125.182) 25.657 ms tcore4-ashburnbk_hundredgige0-1-0-0.net.bell.ca (64.230.125.184) 30.723 ms 30.673 ms discussions/helpdesk tickets Effect: anyone bringing up a clean ONAP system casablanca, master, 3.0.0-ONAP - all will take 35+ hours to come up depending on what is deployed - for example LOG uses dockerhub images it will come up fast - but AAF or any other pod that has images over 1G will each take a couple hours. Once casablanca is pulled you are good - they don't change - but for master - everytime you redeploy on a different day - all or some part of the images will need to be pulled. Temp Workaround nexus3.onap.cloud:5000 alternate proxy (have the LF's back when on vacation) up on 20181218 - taking 2 days to saturate with casablanca images 128G FS ETA late Friday - as of 20h of pulliing - see 21 images of https://git.onap.org/integration/tree/version-manifest/src/main/resources/docker-manifest.csv?h=casablanca 20181219:1700EDT nexus3.onap.cloud status : 16h of pulls - 26 images ubuntu@a-nexus3:~$ sudo docker images | wc -l 26 ubuntu@a-nexus3:~$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE nexus3.onap.cloud:5000/onap/aai-resources 1.3.4 723d184670e2 3 weeks ago 515 MB nexus3.onap.cloud:5000/onap/aai-graphadmin 1.0.1 ed643f4a192c 4 weeks ago 526 MB nexus3.onap.cloud:5000/onap/aai-traversal 1.3.3 4e5784b9e283 4 weeks ago 526 MB nexus3.onap.cloud:5000/onap/appc-cdt-image 1.4.3 10b6b253e1a9 4 weeks ago 160 MB nexus3.onap.cloud:5000/onap/appc-image 1.4.3 888018330bf5 4 weeks ago 2.88 GB nexus3.onap.cloud:5000/onap/babel 1.3.2 b4012e79495e 4 weeks ago 625 MB nexus3.onap.cloud:5000/onap/aaf/aaf_service 2.1.8 6eb295fed110 5 weeks ago 1.16 GB nexus3.onap.cloud:5000/onap/aaf/aaf_oauth 2.1.8 74dcdce76094 5 weeks ago 1.16 GB nexus3.onap.cloud:5000/onap/aaf/aaf_locate 2.1.8 2a4eaa6275ff 5 weeks ago 1.16 GB nexus3.onap.cloud:5000/onap/aaf/aaf_hello 2.1.8 495a01176053 5 weeks ago 1.16 GB nexus3.onap.cloud:5000/onap/aaf/aaf_gui 2.1.8 8caa6dc681f0 5 weeks ago 1.16 GB nexus3.onap.cloud:5000/onap/aaf/aaf_fs 2.1.8 3d663698534d 5 weeks ago 1.16 GB nexus3.onap.cloud:5000/onap/aaf/aaf_cm 2.1.8 0ba25c4ec3fb 5 weeks ago 1.16 GB nexus3.onap.cloud:5000/onap/aaf/aaf_agent 2.1.8 090b326a7f11 5 weeks ago 1.14 GB nexus3.onap.cloud:5000/onap/aaf/aaf_config 2.1.8 6506ac785cb5 5 weeks ago 1.14 GB nexus3.onap.cloud:5000/onap/aaf/aaf_cass 2.1.8 4b91e9b0b43f 5 weeks ago 323 MB nexus3.onap.cloud:5000/onap/aaf/smsquorumclient 3.0.1 f8cf701eadc3 7 weeks ago 18.2 MB nexus3.onap.cloud:5000/onap/aaf/sms 3.0.1 02363fccc6c7 7 weeks ago 35.4 MB nexus3.onap.cloud:5000/onap/aai/esr-server 1.2.1 00c9c28e8936 7 weeks ago 521 MB nexus3.onap.cloud:5000/onap/aai/esr-gui 1.2.1 4bd7ab7ae54a 7 weeks ago 512 MB nexus3.onap.cloud:5000/onap/aaf/testcaservice 3.0.0 fc717d0b071c 2 months ago 1.17 GB nexus3.onap.cloud:5000/onap/aaf/abrmd 3.0.0 00a91d2dc09d 2 months ago 1.15 GB nexus3.onap.cloud:5000/onap/aaf/distcenter 3.0.0 d8d9137ef2d3 2 months ago 1.09 GB nexus3.onap.cloud:5000/onap/aai-cacher 1.0.0 8ec3df246a35 2 months ago 466 MB slightly-not-useful-random graphs
for reference my private AWS proxy after 46h of pulls has 60 images of 180 - predicting 140h or 5.8 days ubuntu@ip-172-31-10-98:~$ sudo docker images | wc -l 52 # at nexus3.onap.info:5000/onap/externalapi/nbi 3.0.1 1ebc02237c1c 2 months ago 122 MB similar incoming traffic profile from nexus3.onap.org
nexus4.onap.cloud:5000 alternate proxy for master - larger FS 200G - access instructions, cert, installation on Cloud Native Deployment windriver lab also has a network issue (for example if i pull from nexus3.onap.cloud:5000 (azure) into an aws EC2 instance - 45 sec for 1.1G - If I pull the same in an openlab VM - on the order of 10+ min) - therefore you need a local nexus3 proxy if you are inside the openstack lab - I have registered nexus3windriver.onap.cloud:5000 to a nexus3 proxy in my logging tenant - cert above I take this tooooo seriously | TSC-79 - Getting issue details... STATUS |
| Task Force Update | 60 mins | E2E Process Automation TODO: review the vFW automation in https://github.com/garyiwu/onap-lab-ci - thanks Yang Xu | TSC-53 - Getting issue details... STATUS | |
| Task Force Update | 15 mins | Pair Wise Activities Update | TSC-42 - Getting issue details... STATUS | |
| TSC Activities and Deadlines | TSC Vice-Chair Election - Congratulations Lingli ! | TSC-3 - Getting issue details... STATUS | ||
| Incoming ONAP Events | 5 mins | Jan 8-11 - Dublin Release F2F Developer Design Forum (France): https://wiki.lfnetworking.org/pages/viewpage.action?pageId=8257579 Feel free to request your VISA: http://events.linuxfoundation.org/visa-request Submit your proposal: https://wiki.lfnetworking.org/display/LN/OPNFV-ONAP+January+2019+Session+Proposals Reminder - No TSC Call on December 27th, 2018 |
