F2F: ONAP CI/CD using OOM Kubernetes
Description
Attachments
- 07 Dec 2017, 08:21 PM
- 12 Nov 2017, 04:39 PM
- 06 Nov 2017, 03:26 AM
blocks
is blocked by
relates to
Activity
Former user December 9, 2017 at 5:15 PM
Shane I have another item we can create a dashboard fragment for - to add to yours - robot init (composed of cloud-region AAU put - customer create, model distribute - will add a couple more rest calls - with actual output - this is the vFW just before service creation in VD
[12:08]
http://jenkins.onap.info/job/oom-cd/659/console
[12:08]
16:46:36 run partial vFW
16:46:36 Starting Xvfb on display :89 with res 1280x1024x24
16:46:36 Executing robot tests at log level TRACE
16:46:38 ==============================================================================
16:46:38 OpenECOMP ETE
16:46:38 ==============================================================================
16:46:38 OpenECOMP ETE.Robot
16:46:38 ==============================================================================
16:46:38 OpenECOMP ETE.Robot.Testsuites
16:46:38 ==============================================================================
16:46:40 OpenECOMP ETE.Robot.Testsuites.Demo :: Executes the VNF Orchestration Test ...
16:46:40 ==============================================================================
16:46:43 Initialize Customer And Models | PASS |
16:46:43 ------------------------------------------------------------------------------
16:46:43 OpenECOMP ETE.Robot.Testsuites.Demo :: Executes the VNF Orchestrat... | PASS |
[12:13]
https://github.com/obrienlabs/onap-root/commit/cda829a5bd9ae7f53442aebdff6b6aa868590940
Former user December 7, 2017 at 8:22 PM
@Former user has added an excellent dashboard at
live
((col:1,id:AWAts77k3NTXK5mX2kuM,panelIndex:1,row:1,size_x:8,size_y:3,type:visualization),(col:9,id:AWAtuTVI3NTXK5mX2kuP,panelIndex:2,row:1,size_x:4,size_y:3,type:visualization),(col:1,id:AWAtuBTY3NTXK5mX2kuO,panelIndex:3,row:7,size_x:6,size_y:3,type:visualization),(col:1,id:AWAttmqB3NTXK5mX2kuN,panelIndex:4,row:4,size_x:6,size_y:3,type:visualization),(col:7,id:AWAtvHtY3NTXK5mX2kuR,panelIndex:6,row:4,size_x:6,size_y:6,type:visualization)),query:(match_all)),timeRestore:!f,title:'CD%20Health%20Check',uiState),viewMode:view)
Former user November 17, 2017 at 5:06 PM
Source data – a full build of the system occurs every hour for now – in the future by commit trigger from a checkin – when I setup a mirror repository
http://jenkins.onap.info/job/oom-cd/
http://jenkins.onap.info/job/oom-cd/164/console
Logs from above pumped live into
last 7 days – 5 second refresh
http://kibana.onap.info:5601/app/kibana#/discover?_g=(refreshInterval'$$hashKey':'object:817',display:'5%20seconds',pause:
(_source),index:AV-vxHzpUcDKD8zJ1zbn,interval:auto,query:(match_all)),sort:!('@timestamp',desc))
Dashboard 5 sec refresh – query is “message: PASS” – we count the number of health check passes per hour
((col:1,id:AV-xBtr0UcDKD8zJ15sl,panelIndex:1,row:1,size_x:6,size_y:3,type:visualization),(col:7,id:AV-xCajhUcDKD8zJ15y-,panelIndex:2,row:1,size_x:6,size_y:3,type:visualization)),query:(match_all)),timeRestore:!t,title:'dash%20pass',uiState:(P-1:(vis:(params:(sort:(columnIndex:!n,direction:!n))))),viewMode:view)
Former user November 11, 2017 at 6:20 PMEdited
20171111_cd_onap_info ami-3b67df41 for cd.onap.info
ubuntu@ip-172-31-85-6:~$ history
2 curl https://releases.rancher.com/install-docker/1.12.sh | sh
3 sudo usermod -aG docker ubuntu
5 git clone http://gerrit.onap.org/r/oom
6 curl https://jira.onap.org/secure/attachment/10666/prepull_docker.sh > prepull_docker.sh
7 chmod 777 prepull_docker.sh
8 nohup ./prepull_docker.sh &
10 tail -f nohup.out
11 ps -ef | grep docker | grep pull
13 curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
14 chmod +x ./kubectl
16 sudo mv ./kubectl /usr/local/bin/kubectl
17 mkdir ~/.kube
23 docker run -d --restart=unless-stopped -p 8880:8080 rancher/server
25 sudo docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.6 http://cd.onap.info:8880/v1/scripts/2C253E95BFB5300782DF:1483142400000:BMhoRm5Tygl4DJe7rh8fqgR9Bs
26 vi ~/.kube/config
27 kubectl get pods --all-namespaces
40 wget http://storage.googleapis.com/kubernetes-helm/helm-v2.3.0-linux-amd64.tar.gz
41 tar -zxvf helm-v2.3.0-linux-amd64.tar.gz
43 sudo mv linux-amd64/helm /usr/local/bin/helmDetails
Assignee
Former userFormer user(Deactivated)Reporter
Former userFormer user(Deactivated)Labels
Time tracking
3d 3h loggedSprint
NoneFix versions
Priority
High
Details
Details
Assignee
Former userReporter
Former userLabels
Time tracking
Sprint
Fix versions
Priority
High
1. Session Title
ONAP CI/CD using OOM Kubernetes
2. Email address
frank.obrien@amdocs.com
3. Speaker Full Name
Michael O'Brien
4. Speaker Company
Amdocs
5. Speaker Biography (900 characters maximum)
https://wiki.onap.org/display/~michaelobrien
6. Select the Talk Type ( Presentation / Panel Discussion / Hands On)
Presentation
7. Title for Your Proposal
ONAP CI/CD using OOM Kubernetes
8. Session Abstract (900 character Maximum)
Development at ONAP occurs on the master branch. The ONAP project is therefore in need of an automated end-to-end integration sanity to declare a particular tagged build sane and working at some defined level of functionality. This tagged build would represent a snapshot of war/jar artifacts and docker images - usually a nightly or commit triggered tag.
Automatically determining the status of the master master branch can be accomplished by running a continuous deployment framework after CI finishes and builds the artifacts in response to a git merge trigger callback.
This presentation will go over a particular implementation of a short regression test (as opposed to a full long regression) - that exercises an extended health check and the automated subset of the vFirewall use case demo to mark a particular tagged build as good. Developers can then consume the build/pull and update their environments with only builds marked as OK (a TBD level). This CD job can also verify that a commit did not break integration with other components - as much as the sanity can provide maximum spanning coverage for.
The presentation will run from triggered Jenkins jobs that deploy for example an Amazon AWS EC2 spot VM, deploy OOM, run health and the vFirewall, collect logs/results and tear down the temporary VM. The process from the Jenkins artifact creation trigger to VM teardown is between 20-60 min per trigger - depending on docker image preload. A combination of several pre-recordings and parallel live/background demos will compress the actual time to demonstrate ONAP CD in several passes.
Northbound clients like the OPNFV Auto project and CNCF can also mark builds as ready to work with by subscribing to the CD result stream.
In summary:
Automatically deploy ONAP OOM based on git submission artifact generation
Run extended healthcheck and automated parts of vFirewall E2E sanity use case
Report back on whether specific tag (code, artifacts, docker image repos) is sane and ready for developer and northbound (CNCF, OPNFV) consumption
By default tear down OOM deployment after testing - or keep a rolling set - up for testing, or provide saved tagged AMI for later reproduction
The CD deploy will - depending on commit/merge load test 1 or more merges per CD job
See redeploy sh script on https://lf-onap.atlassian.net/browse/INT-120, https://lf-onap.atlassian.net/browse/OOM-150 - Deployment Integration Jenkins job using OOM-K8S - to validate branch
Rotating live deployments in a public lab on AWS EC2
Feeds directly into the development process, could show benefits vs Docker Compose/Vagrant
Discuss https://lf-onap.atlassian.net/browse/OOM-357 - CNCF CrossCloud Deployment/CI Integration with ONAP
Discuss https://wiki.onap.org/display/DW/OPNFV+Integration - OPNFV Northbound client integration with ONAP
Discuss results of OPNFV Plugfest - https://wiki.opnfv.org/display/EVNT/Plugfest+Planning+Page 1 week before our F2F
See prototype on https://wiki.onap.org/display/DW/ONAP+on+Amazon+EC2
See Tracking Wiki at https://wiki.onap.org/display/DW/ONAP+on+Kubernetes
See Tracking JIRA at https://lf-onap.atlassian.net/browse/OOM-393 and https://lf-onap.atlassian.net/browse/INT-120
Recommended Prerequisites:
"An Introduction to the ONAP Operations Manager (OOM)"
"Deploy ONAP/OOM on any Environment"
Recommended Follow Up Presentations:
"ONAP/OOM for Developers"