Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Agenda Items

Presented ByTimeNotes/LinksJIRA Task
Casablanca Maintenance Release5 mins

View file
nameONAP Casablanca TSC Dec20Version1.pdf
height250


Any Infrastructure Improvement/Plan

Linux Foundation

Michael O'Brien

10

mins

Any LF showstopper?

#65866 - Nexus3 proxy verified 80-100x faster downloads since 20181217

#65794 - Nexus3 timing out - still getting 0.4MB/sec - not the usual 10+MB/sec

#65809 - Nexus3 slowdown 10X - docker pulls very slow in openlab

example normal speed on the nexux3.onap.info:5000 proxy

View file
namenexus3_onap_info_normal_download.mov
height250

example pull - 5 sec

sudo docker pull nexus3.onap.info:5000/onap/appc-cdt-image:1.4.3

this will take minutes

sudo docker pull nexus3.onap.org:10001/onap/appc-cdt-image:1.4.3

Nexus3.onap.org:10001 experiencing a serious routing? issue not the older 3 hour slowdown from 4-6 months ago that was fixed. Normally prepull takes 30 min - it now takes 120+ hours for a full prepull since 20181217 - and 80x slowdown on image downloads.

Q) why jenkins has no issue with nexus3 - A) they are on the same domain - and don't go through a bell.ca exchange

Code Block
themeRDark
;; ANSWER SECTION:
jenkins.onap.org.	60	IN	CNAME	cloud.onap.org.
cloud.onap.org.		60	IN	A	199.204.45.137
;; ANSWER SECTION:
nexus3.onap.org.	60	IN	CNAME	cloud.onap.org.
cloud.onap.org.		47	IN	A	199.204.45.137

run a traceroute and notice

Code Block
themeRDark
# this is from an AWS EC2 instance in us-east-2
ubuntu@ip-172-31-10-98:~$ traceroute nexus3.onap.org
traceroute to nexus3.onap.org (199.204.45.137), 30 hops max, 60 byte packets
16  tcore3-ashburnbk_hundredgige0-1-0-0.net.bell.ca (64.230.125.182)  25.657 ms tcore4-ashburnbk_hundredgige0-1-0-0.net.bell.ca (64.230.125.184)  30.723 ms  30.673 ms 

discussions/helpdesk tickets

https://lists.onap.org/g/onap-discuss/topic/aaf_cm_image_not_found/28803606?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,28803606

https://lists.onap.org/g/onap-discuss/topic/onap_helpdesk_65866/28800220?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,28800220

https://lists.onap.org/g/onap-discuss/topic/onap_helpdesk_65794_nexus3/28794897?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,20,28794897

https://lists.onap.org/g/onap-discuss/topic/onap_beijing_installation/28810052?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,28810052

Effect: anyone bringing up a clean ONAP system casablanca, master, 3.0.0-ONAP - all will take 35+ hours to come up depending on what is deployed - for example LOG uses dockerhub images it will come up fast - but AAF or any other pod that has images over 1G will each take a couple hours. Once casablanca is pulled you are good - they don't change - but for master - everytime you redeploy on a different day - all or some part of the images will need to be pulled.

Temp Workaround

nexus3.onap.cloud:5000 alternate proxy (have the LF's back when on vacation) up on 20181218 - taking 2 days to saturate with casablanca images 128G FS ETA late Friday - as of 20h of pulliing - see 21 images of https://git.onap.org/integration/tree/version-manifest/src/main/resources/docker-manifest.csv?h=casablanca

20181219:1700EDT nexus3.onap.cloud status : 16h of pulls - 26 images

Code Block
themeMidnight
ubuntu@a-nexus3:~$ sudo docker images | wc -l
26
ubuntu@a-nexus3:~$ sudo docker images 
REPOSITORY                                        TAG                 IMAGE ID            CREATED             SIZE
nexus3.onap.cloud:5000/onap/aai-resources         1.3.4               723d184670e2        3 weeks ago         515 MB
nexus3.onap.cloud:5000/onap/aai-graphadmin        1.0.1               ed643f4a192c        4 weeks ago         526 MB
nexus3.onap.cloud:5000/onap/aai-traversal         1.3.3               4e5784b9e283        4 weeks ago         526 MB
nexus3.onap.cloud:5000/onap/appc-cdt-image        1.4.3               10b6b253e1a9        4 weeks ago         160 MB
nexus3.onap.cloud:5000/onap/appc-image            1.4.3               888018330bf5        4 weeks ago         2.88 GB
nexus3.onap.cloud:5000/onap/babel                 1.3.2               b4012e79495e        4 weeks ago         625 MB
nexus3.onap.cloud:5000/onap/aaf/aaf_service       2.1.8               6eb295fed110        5 weeks ago         1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_oauth         2.1.8               74dcdce76094        5 weeks ago         1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_locate        2.1.8               2a4eaa6275ff        5 weeks ago         1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_hello         2.1.8               495a01176053        5 weeks ago         1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_gui           2.1.8               8caa6dc681f0        5 weeks ago         1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_fs            2.1.8               3d663698534d        5 weeks ago         1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_cm            2.1.8               0ba25c4ec3fb        5 weeks ago         1.16 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_agent         2.1.8               090b326a7f11        5 weeks ago         1.14 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_config        2.1.8               6506ac785cb5        5 weeks ago         1.14 GB
nexus3.onap.cloud:5000/onap/aaf/aaf_cass          2.1.8               4b91e9b0b43f        5 weeks ago         323 MB
nexus3.onap.cloud:5000/onap/aaf/smsquorumclient   3.0.1               f8cf701eadc3        7 weeks ago         18.2 MB
nexus3.onap.cloud:5000/onap/aaf/sms               3.0.1               02363fccc6c7        7 weeks ago         35.4 MB
nexus3.onap.cloud:5000/onap/aai/esr-server        1.2.1               00c9c28e8936        7 weeks ago         521 MB
nexus3.onap.cloud:5000/onap/aai/esr-gui           1.2.1               4bd7ab7ae54a        7 weeks ago         512 MB
nexus3.onap.cloud:5000/onap/aaf/testcaservice     3.0.0               fc717d0b071c        2 months ago        1.17 GB
nexus3.onap.cloud:5000/onap/aaf/abrmd             3.0.0               00a91d2dc09d        2 months ago        1.15 GB
nexus3.onap.cloud:5000/onap/aaf/distcenter        3.0.0               d8d9137ef2d3        2 months ago        1.09 GB
nexus3.onap.cloud:5000/onap/aai-cacher            1.0.0               8ec3df246a35        2 months ago        466 MB

slightly-not-useful-random graphs

for reference my private AWS proxy after 46h of pulls has 60 images of 180 - predicting 140h or 5.8 days

Code Block
themeMidnight
ubuntu@ip-172-31-10-98:~$ sudo docker images | wc -l
52
# at
nexus3.onap.info:5000/onap/externalapi/nbi              3.0.1               1ebc02237c1c        2 months ago        122 MB

similar incoming traffic profile from nexus3.onap.org

nexus4.onap.cloud:5000 alternate proxy for master - larger FS 200G

- access instructions, cert, installation on Cloud Native Deployment

windriver lab also has a network issue (for example if i pull from nexus3.onap.cloud:5000 (azure) into an aws EC2 instance - 45 sec for 1.1G - If I pull the same in an openlab VM - on the order of 10+ min) - therefore you need a local nexus3 proxy if you are inside the openstack lab - I have registered nexus3windriver.onap.cloud:5000 to a nexus3 proxy in my logging tenant - cert above

I take this tooooo seriously

Jira Legacy
serverSystem Jira
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyTSC-79
Task Force Update60 mins

E2E Process Automation

View file
nameONAP-E2E-Automation-v3.pptx
height250

TODO: review the vFW automation in https://github.com/garyiwu/onap-lab-ci - thanks Yang Xu

Jira Legacy
serverSystem Jira
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyTSC-53

Task Force Update15 mins

Pair Wise Activities Update

Dublin Pair-Wise Testing

Jira Legacy
serverSystem Jira
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyTSC-42

TSC Activities and Deadlines

TSC Vice-Chair Election - Congratulations Lingli !

https://lists.onap.org/g/onap-tsc/topic/congratulations_lingli/28803393?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,28803393

Jira Legacy
serverSystem Jira
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyTSC-3

Incoming ONAP Events

5

mins

Jan 8-11 - Dublin Release F2F Developer Design Forum (France): https://wiki.lfnetworking.org/pages/viewpage.action?pageId=8257579   

Feel free to request your VISA: http://events.linuxfoundation.org/visa-request

Submit your proposal: https://wiki.lfnetworking.org/display/LN/OPNFV-ONAP+January+2019+Session+Proposals

Reminder - No TSC Call on December 27th, 2018


...