- PTL
- Integration committers cleanup after release:
- Open nominations for Integration Project PTL - Michał Jagiełło is going to step down from that role (but still being committer)
- Lab status
- A&AI fails on both Orange and DT daily
- Maciej Wereski Probably race condition on onap-aai-graphadmin-create-db-schema job
- Radoslaw Chmiel found a workaround?
- Gating 4 was failing on
apt update- Disk was full
- Remove docker artifacts with
docker system prune -a
- Gating failing on another step: deploy_full (#2532722215) · Jobs · Orange-OpenSource / lfn / onap / onap_oom_automatic_installation · GitLab
- Gating is going to be down due to Changes to GitLab.com public project CI/CD minute quotas | GitLab - gating pipelines are running on Orange (as we have still no GitLab CI on ONAP group) so we are going to reach the limit very soon
- A&AI fails on both Orange and DT daily
- Tests on the UNH
- LF created project ONAP / integration / chained-ci · GitLab
- still no Integration project committers as members
- University New Hampshire lab - Developer Wiki - Confluence (onap.org) ← wiki to follow the status of running tests on UNH lab
- GitLab runner and jumphost servers up and running
- GitLab runner not registered - there is no place to register it
- Release of base Python/Java(?) images is needed
- Lab status
- still failing holmes-healthcheck on Orange lab
- [HOLMES] Fixed the Healthcheck Problem (I64d5f326) · Gerrit Code Review (onap.org) OOM change merged
- k8s version on gating (where we don't have that issue) is different than on Orange daily - should be checked if change version on Orange fix that issue
- a lot of failures on security tests
- Fiachra Corcoran - waivers directory gerrit.onap Code Review - integration/seccom.git/tree - waivers/
- good results on stability: both instantiation and onboarding
- gating s3 error on tests results upload - no space left on the device
- still failing holmes-healthcheck on Orange lab
- Tests on the UNH
- still no LF agree on https://gitlab.com/onap/integration creation
- we have an integration tenant on UNH lab
- other projects as well and some of them already created their own ONAP instances (Policy for sure)
- plan is to create a GitLab runner on UNH Infrastructure tenant and run Orange pipelines there
- all integration commiters should have an access to UNH lab following that guide: ONAP IOL-UNH Lab - Developer Wiki - Confluence
- Lab status
- weekly tests Xtesting ONAP Results
- versions and stability tests ran manually
- MariaDB issues Lukasz Rajewski
- several changes:
- [MARIADB] Fix wsrep software error (Ib7d8be50) · Gerrit Code Review (onap.org) - downgrade to still acceptable version by SECCOM
- [MARIADB] Back to istanbul version of mariaDB image (I23b31db0) · Gerrit Code Review (onap.org) - back to the Istanbul version of MariaDB
- [SO][MARIADB] Improve Mariadb connection (I3dc20f27) · Gerrit Code Review (onap.org) - WIP: Use sequential jbdc connection on SO
- Wait for tests results to check if there is any improvement
- several changes:
- E2E network slicing automation
- Maciej Lisowski provide a change to automate A&AI
- marcin krasowski works on SDC part automation
- Ahila P works on the CPS TBDMT part
- Lab status
- Rearrange tests execution order:
- Shoulnd't security tests be executed on the beginning? Some tests on failure don't cleanup properly
- onap-k8s and onap-k8s-teardown false-positives
- Jakarta Integration Blocking points - Developer Wiki - Confluence (onap.org)
- Maciej Wereski mentioned about ETCD issue on version recommended by SECCOM
- Issue: Inconsistent revision and data occurs · Issue #13766 · etcd-io/etcd (github.com)
- SECCOM recommendation: gerrit.onap Code Review - integration/seccom.git/blob - recommended_versions.yaml
- Amy Zwarico Paweł Pawlak take a look please
- Lab status
- Lack of stability mostly due to SDC onboarding issues
- There is an idea to stop executing tests if previous stage failed. We have currently 4 stages:
- Infra healthcheck
- Healthcheck (Splitted into 2 on Orange)
- Smoke tests (Splitted into 2 on Orange)
- Onap security
- So if stage 1 failed (Infra healthcheck) we don't execute any other tests
- There is also an idea to reduce the number of tests to execute only tests which are not a part of another, e.g. basic_onboard tests is only onboarding service into SDC, but for many other tests we do the same.
- Morgan Richomme Q&A
- Volunteers needed to manage gating and testing
- Needs to ask TSC
- Failing CSIT tests:
- vfc-gvnfm-vnflcm-master-csit-sanity-check #3345 [Jenkins] (onap.org)
- vfc-gvnfm-vnflcm-master-csit-sanity-check [Jenkins] (onap.org)
- vfc-gvnfm-vnfres-master-csit-sanity-check [Jenkins] (onap.org)
- vfc-gvnfm-vnfres-master-csit-sanity-check [Jenkins] (onap.org)
- vfc-nfvo-driver-vnfm-gvnfm-master-csit-sanity-check [Jenkins] (onap.org)
- vfc-nfvo-driver-vnfm-gvnfm-master-csit-sanity-check [Jenkins] (onap.org)
- vfc-nfvo-lcm-master-csit-sanity-check [Jenkins] (onap.org)
- vfc-nfvo-lcm-master-csit-sanity-check [Jenkins] (onap.org)
- NOT OUR ISSUE
- Jakarta Integration Blocking points - Developer Wiki - Confluence (onap.org)
- Lab status
- Regression candidate; Tests results - Summary (onap.org)
- Krzysztof Kuzmicki is working on:
- cmpv2 - daily result cmpv2 (#2214007103) · Jobs · Orange-OpenSource / lfn / onap / xtesting-onap · GitLab Krzysztof Kuzmicki
- hv-ves
- Feedback on New Q&A review & Demo weekly stability tests Morgan Richomme & Demo kubescape
- how to run stability tests:
- link to the script:
- link to the onap_bench library:
- chaos tests:
- link to the python package used in resiliency/chaos testing:
- kubescape tests:
- almost ready
- link to the python package used in kubescape tests:
- if we want to get more details we should create an ONAP account on kubescape to push results there (or create own kubescape instance)
- exchange with seccom Michał Jagiełło
- how to run stability tests:
- Use case automation status:
- Maciej Lisowski marcin krasowski any update?
- What's next with the robot:
- testsuite/python-testing-utils · Gerrit Code Review (onap.org) is Python 3.X compatible (no Python 3.10, but it's more with testing framework there and no code)
- xtesting-smoke-usecases-robot docker image has no Python 2.7 installed
- nexus3.onap.org:10001/onap/testsuite:1.8.0 image has Ubuntu installed and Python 2.7, bash etc installed there
- Do not trust license scanning tools: [INT-2086] onap/integration-python:9.1.0 contains libuuid which is GPL-3.0 - ONAP JIRA
- tern marked libuuid as GPLv3
- ScanCode.io marked libuuid as GPLv3
- libuuid has BSD license COPYING « libuuid - util-linux/util-linux.git - The util-linux code repository. (kernel.org)
- Thomas Kulik reported we have an empty doc directory in pythonsdk-tests, We should:
- provide tests documentation, or
- remove that directory
- Michał Jagiełło is going to create a ticket for Morgan Richomme to provide some doc there
- Lab status
- Two new-old pipelines
- ONAP istanbul daily Index of /onap-integration/daily/onap-daily-dt-oom-istanbul
- ONAP master daily Index of /onap-integration/daily/onap-daily-dt-oom-master
- Executed on DT lab - thanks Andreas Geißler !!!
- basic_vm regression - probably that change [AAI] Updates for AAI Jakarta to release latest versions (I9a068590) · Gerrit Code Review (onap.org) provides it - needs to be discussed with OOM team
- good resutls on Istanbul daily Xtesting ONAP Results
- Krzysztof Kuzmicki works on the CMPv2 tests
- Two new-old pipelines
- New Q&A review Morgan Richomme
- Few Q&A wiki pages from Morgan about ONAP integration related daily tasks:
- Till the end of the March it will be a fixed point on our agends: Morgan Q&A
- One more wiki page prepared by Sylvain Desbureaux : Daily Deployments and gating
- Use case automation status:
- No update
- Run tests on UNH lab
- Lab is ready to use
- I created an epic for that [INT-2085] Run ONAP tests on UNH lab - ONAP JIRA
- New committers: marcin krasowski Maciej Lisowski congratulations!
- Lab status
- Istanbul
- daily istanbul is back (problem due python dependency in k8s kubespray installer, increase of mino version 1.15→1.16 foxed the issue)
- results are very good (as for the release validation): https://logs.onap.org/onap-integration/daily/onap_daily_pod4_istanbul/2022-03/02_04-01/
- note it includes the log4j changes (istanbul = istanbul oom branch)
- problem solved also on DT side, daily shall be back with new name soon
- basic_cnf_macro not integrated in istanbul - not in istanbul xtesting dockers. But test done on istanbul with master xtesting docker shows that basic_cnf_macro is PASS (no regression) Lukasz Rajewski
- master
- daily master error seem reproducible
- regressions on basic_vm (so cleanup of volume) => needs to be investigated AP: Michał Jagiełło
- errors on *_macro due to bad cleanup => Michał Jagiełło patch merged today
- weekly restarted thsi week end
- tern results: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2022-02/25_20-26/legal/tern/index.html. Former user (Deleted) asks if tern shall be removed (JIRA for that). unitl scancode is not fullyu integrate, no problem to keep tern in weekly (not run so frequently..)
- versions results: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2022-02/25_20-26/security/versions/versions.html + Amy Zwarico Paweł Wieczorek rouzaut
- internal certificates: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2022-02/25_20-26/infrastructure-healthcheck/internal_check_certs/internal_check_certs/certificates.html
- in addition of robot, issue with ejbca and multicloud-fcaps
- stability tests not run due to basic_vm regression
- daily master error seem reproducible
- Istanbul
- Use case automation status:
- DCAE related use cases
- Update on 5Gbulkpm: Krzysztof Kuzmicki fixed most of the issue, shall be ready now. MR created to populate chartmuseum to be reviewed and merged
- next planned cmpv2
- dcaemod more complex...
- 5G Slicing: policy code merged in demo
- kafka HC => Morgan Richomme created a page to explain how the issue coudl be solved: Refactoring of tests due to kafka refactoring. Krzysztof Kuzmicki indicated that it woudl also be possible to call kubernetes command from the robot test
- Discussion on AAI stress tests with ATT, ATT gatling scripts shall be available and reusable for tests
- DCAE related use cases
- Lab status
- Istanbul
- results not bad. Remaining test errors due to DCAE changes but a patch is in gate - no results due to error below.
- The Makupsafe issue (jinja2 upstream dependency) is also responsible for the istanbul k8s installation issue
- Master
- deployment OK but all smoke tests failing due to Makupsafe
- pb Markupsafe lib:
- ugly test merged...https://gerrit.onap.org/r/c/integration/xtesting/+/127250 => tests shall be run on Master tomorrow (does not fix the istanbulinstallation issue)
- I tried a cleaner way (https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk/-/merge_requests/334) but all tests are failing...wait for Michał Jagiełło comeback to check the issue and be sure to do thing properly on SDK versioning
- ugly test merged...https://gerrit.onap.org/r/c/integration/xtesting/+/127250 => tests shall be run on Master tomorrow (does not fix the istanbulinstallation issue)
- Istanbul
- jakarta tests
- 5G Slicing automation follow up
- All patches of Maciej Lisowski merged
- Question from Ahila P on simulators sent by mail. For the moment 2 of the 3 simulators are in the wiki. Action morgan push simu to the repo (the repo already exists)
- kubescape => demo next week
- Load test on AAI (using gatling) => action Morgan Richomme resync Michał Jagiełło /William Reehil / Christophe Auzizeau
- kafka (testing refactoring) => Refactoring of tests due to kafka refactoring, answer sent by mail to Fiachra Corcoran
- 5G Slicing automation follow up
- Misc
- Morgan Richomme and Sylvain Desbureaux are stepping down
- end of contributions: end of March 2022
- end of Orange platform: end of Jakarta release (with minimal support)
- Azure / NH won't be touched (gating will work, staging, ...) but resources to be identified to manage them...
- Morgan Richomme and Sylvain Desbureaux are stepping down
- Lab status
- onap-k8s issue - there is an another issue after the dependency library new version release
- Orange lab - some issues with connection during random tests
still blocks master gatingJira Legacy server System Jira columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key AAI-3450
- Former user (Deleted) http://scancode.onap.eu/
- logging currently using GitHub account (consider using LF ID)
- needs to be discussed with seccom where the scan policies should be stored
- prepare a demo with two Integration projects:
- integration/docker/onap-java11 · Gerrit Code Review
- integration/docker/onap-python · Gerrit Code ReviewAdd images into scancode during
docker-verifyJenkis job and run pipeline for them. We could analyse results and present that during one of the incoming PTL meetings
INT-2021 - Automate 5G Slicing use case in CI OTWARTY update
- Create a demo · Gerrit Code Review (onap.org) repo directory to keep there scripts and migrate Manual Configurations - Developer Wiki - Confluence (onap.org) into markdown files there to be able to keep history and review changes
- Lab status
- Orange lab down
- DT still down
- Gating is OK, but since today some A&AI issues observed
- basic_cnf_macro test
- E2E network slicing use case
INT-2064 - Automate 5G Slicing use case in CI - Manual configuration step 1 - SDC OTWARTY
...
- at the beginning Integration team needs to take a look on the first phase: Manual Configurations - Developer Wiki - Confluence (onap.org) - preparing an ONAP for instantiation
- several steps - for sure some missing gaps in ONAP SDK
- There is a plan to create a scenario in pythonsdk-tests repo for that use case
- [INT-2021] Automate 5G Slicing use case in CI - ONAP JIRA ticket which is going to be used to plan a work
- Automate use case 5G Slicing in CI - Developer Wiki - Confluence (onap.org) ← links shared by Ahila P
...
Today morning Trivy was ran on ONAP daily master
| Code Block | ||
|---|---|---|
| ||
Pods
onap-cli-757c76f546-zfd4m in vulnerable!
onap-message-router-0 in vulnerable!
onap-sdnrdb-coordinating-only-6f6d6b887b-kzr9x in vulnerable!
onap-sdnrdb-master-0 in vulnerable!
onap-sdnrdb-master-1 in vulnerable!
onap-sdnrdb-master-2 in vulnerable!
onap-vfc-huawei-vnfm-driver-57fddd8bc7-nqn7b in vulnerable!
onap-vnfsdk-56d9fb886f-sk4s4 in vulnerable!
Images:
nexus3.onap.org:10001/onap/cli:6.0.1 -> onap-cli-757c76f546-zfd4m
nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.3.0 -> onap-message-router-0
docker.nexus.azure.onap.eu/bitnami/elasticsearch:7.9.3 -> onap-sdnrdb-coordinating-only-6f6d6b887b-kzr9x, onap-sdnrdb-master-X
nexus3.onap.org:10001/onap/vfc/nfvo/svnfm/huawei:1.3.9 -> onap-vfc-huawei-vnfm-driver-57fddd8bc7-nqn7b
nexus3.onap.org:10001/onap/vnfsdk/refrepo:1.6.3 -> onap-vnfsdk-56d9fb886f-sk4s4 |
...
- Disable portal tests Krzysztof Kuzmicki
- Former user (Deleted) starts http://scancode.onap.eu/ . Thanks!
- Lab status
- Better results, but due to Nexus restart and no images all tests failed
- cds-resource-resolution is still failing... looks the instance is created before tests is executed
- Integration risks for Jakarta Jakarta Risks - Developer Wiki - Confluence (onap.org)
- Main risks are resources: labs and humans
- Nexus restart https://lists.onap.org/g/onap-discuss/message/23758 should we restart the images creation?
- Yes, after Jenkins and Nexus are back again jobs below need to be rerun
- https://jenkins.onap.org/view/integration/job/integration-xtesting-healthcheck-docker-merge-master/
- https://jenkins.onap.org/view/integration/job/integration-xtesting-security-docker-merge-master/
- https://jenkins.onap.org/view/integration/job/integration-xtesting-infra-healthcheck-docker-merge-master/
- https://jenkins.onap.org/view/integration/job/integration-xtesting-smoke-usecases-pythonsdk-docker-merge-master/
- https://jenkins.onap.org/view/integration/job/integration-xtesting-smoke-usecases-robot-docker-merge-master/
- https://jenkins.onap.org/view/integration/job/integration-xtesting-smoke-usecases-robot-py3-docker-merge-master/
- Yes, after Jenkins and Nexus are back again jobs below need to be rerun
- Nexus doesn't work - tests doesn't work....
- LF ticket created - Theme Extension - Linux Foundation JIRA
- gatig tests works due to Azure cache - images are not downloaded from nexus during each execution
- When it worked we had several tests issues
- needed to check if we don't use same kubeconfig on many tests pipelines
- that can create conflicts because one test tries to create an instance of existing resource
- needed to check if we don't use same kubeconfig on many tests pipelines
- Kubescape tests
- could be useful but won't replace already existing tests
- root pods check based just on the kubernetes resource definition (pod, deployment, statefulset etc.) and not check if the root user is disabled on the image
- could be possible to add them into pipeline soon
- could be useful but won't replace already existing tests
...
- there are currently 3 Azure labs
- 2 gatings and 1 integration lab with ONAP
- needs to ask TSC if it's possible to add one more to run daily tests - outside of any company infrastructure (currently Orange, Nokia, DT)
- we could run daily on the 1 integration lab machine, but there is a risk we won't have a working lab if during daily execution ONAP creation fails
...
- it's not required and will take a lot of time and resources, leave it as it is
...