Integration Meeting Minutes

01 Jun 2022

PTL
- Integration committers cleanup after release:
- Open nominations for Integration Project PTL - Michał Jagiełło is going to step down from that role (but still being committer)
Lab status
- A&AI fails on both Orange and DT daily
  - Maciej Wereski Probably race condition on onap-aai-graphadmin-create-db-schema job
  - Radoslaw Chmiel found a workaround?
- Gating 4 was failing on apt update
  - Disk was full
  - Remove docker artifacts with docker system prune -a
- Gating failing on another step: deploy_full (#2532722215) · Jobs · Orange-OpenSource / lfn / onap / onap_oom_automatic_installation · GitLab
- Gating is going to be down due to Changes to GitLab.com public project CI/CD minute quotas | GitLab - gating pipelines are running on Orange (as we have still no GitLab CI on ONAP group) so we are going to reach the limit very soon
Tests on the UNH
- LF created project ONAP / integration / chained-ci · GitLab
- still no Integration project committers as members
- University New Hampshire lab - Developer Wiki - Confluence (onap.org) ← wiki to follow the status of running tests on UNH lab
  - GitLab runner and jumphost servers up and running
  - GitLab runner not registered - there is no place to register it
Release of base Python/Java(?) images is needed

11 May 2022

Lab status
- still failing holmes-healthcheck on Orange lab
  - [HOLMES] Fixed the Healthcheck Problem (I64d5f326) · Gerrit Code Review (onap.org) OOM change merged
  - k8s version on gating (where we don't have that issue) is different than on Orange daily - should be checked if change version on Orange fix that issue
    - done: Use the same k8s version as on the gating (4113928f) · Commits · Orange-OpenSource / lfn / ci_cd / chained-ci · GitLab
- a lot of failures on security tests
  - Fiachra Corcoran - waivers directory gerrit.onap Code Review - integration/seccom.git/tree - waivers/
- good results on stability: both instantiation and onboarding
- gating s3 error on tests results upload - no space left on the device
Tests on the UNH
- still no LF agree on https://gitlab.com/onap/integration creation
- we have an integration tenant on UNH lab
- other projects as well and some of them already created their own ONAP instances (Policy for sure)
- plan is to create a GitLab runner on UNH Infrastructure tenant and run Orange pipelines there
- all integration commiters should have an access to UNH lab following that guide: ONAP IOL-UNH Lab - Developer Wiki - Confluence

20 Apr 2022

Lab status
- weekly tests Xtesting ONAP Results
- versions and stability tests ran manually
MariaDB issues Lukasz Rajewski
- several changes:
  - [MARIADB] Fix wsrep software error (Ib7d8be50) · Gerrit Code Review (onap.org) - downgrade to still acceptable version by SECCOM
  - [MARIADB] Back to istanbul version of mariaDB image (I23b31db0) · Gerrit Code Review (onap.org) - back to the Istanbul version of MariaDB
  - [SO][MARIADB] Improve Mariadb connection (I3dc20f27) · Gerrit Code Review (onap.org) - WIP: Use sequential jbdc connection on SO
- Wait for tests results to check if there is any improvement
E2E network slicing automation
- Maciej Lisowski provide a change to automate A&AI
- marcin krasowski works on SDC part automation
- Ahila P works on the CPS TBDMT part

06 Apr 2022

Lab status
Rearrange tests execution order:
- Shoulnd't security tests be executed on the beginning? Some tests on failure don't cleanup properly
- onap-k8s and onap-k8s-teardown false-positives
  - Orange-OpenSource / lfn / tools / kubernetes-status · GitLab [INT-2103] Fix onap-k8s and onap-k8s-teardown tests - ONAP JIRA
Jakarta Integration Blocking points - Developer Wiki - Confluence (onap.org)
Maciej Wereski mentioned about ETCD issue on version recommended by SECCOM
- Issue: Inconsistent revision and data occurs · Issue #13766 · etcd-io/etcd (github.com)
- SECCOM recommendation: gerrit.onap Code Review - integration/seccom.git/blob - recommended_versions.yaml
- Amy Zwarico Paweł Pawlak take a look please

30 Mar 2022

Lab status
- Lack of stability mostly due to SDC onboarding issues
- There is an idea to stop executing tests if previous stage failed. We have currently 4 stages:
  - Infra healthcheck
  - Healthcheck (Splitted into 2 on Orange)
  - Smoke tests (Splitted into 2 on Orange)
  - Onap security
- So if stage 1 failed (Infra healthcheck) we don't execute any other tests
- There is also an idea to reduce the number of tests to execute only tests which are not a part of another, e.g. basic_onboard tests is only onboarding service into SDC, but for many other tests we do the same.
Morgan Richomme Q&A
Volunteers needed to manage gating and testing
- Needs to ask TSC
Failing CSIT tests:
Jakarta Integration Blocking points - Developer Wiki - Confluence (onap.org)

16 Mar 2022

Lab status
- Regression candidate; Tests results - Summary (onap.org)
- Krzysztof Kuzmicki is working on:
  - cmpv2 - daily result cmpv2 (#2214007103) · Jobs · Orange-OpenSource / lfn / onap / xtesting-onap · GitLab Krzysztof Kuzmicki
  - hv-ves
Feedback on New Q&A review & Demo weekly stability tests Morgan Richomme & Demo kubescape
- how to run stability tests:
  - link to the script:
  - link to the onap_bench library:
- chaos tests:
  - link to the python package used in resiliency/chaos testing:
- kubescape tests:
  - almost ready
  - link to the python package used in kubescape tests:
  - if we want to get more details we should create an ONAP account on kubescape to push results there (or create own kubescape instance)
  - exchange with seccom Michał Jagiełło
Use case automation status:
- Maciej Lisowski marcin krasowski any update?
What's next with the robot:
- testsuite/python-testing-utils · Gerrit Code Review (onap.org) is Python 3.X compatible (no Python 3.10, but it's more with testing framework there and no code)
- xtesting-smoke-usecases-robot docker image has no Python 2.7 installed
- nexus3.onap.org:10001/onap/testsuite:1.8.0 image has Ubuntu installed and Python 2.7, bash etc installed there
Do not trust license scanning tools: [INT-2086] onap/integration-python:9.1.0 contains libuuid which is GPL-3.0 - ONAP JIRA
- tern marked libuuid as GPLv3
- ScanCode.io marked libuuid as GPLv3
- libuuid has BSD license COPYING « libuuid - util-linux/util-linux.git - The util-linux code repository. (kernel.org)
Thomas Kulik reported we have an empty doc directory in pythonsdk-tests, We should:
- provide tests documentation, or
- remove that directory
- Michał Jagiełło is going to create a ticket for Morgan Richomme to provide some doc there

09 Mar 2022

Lab status
- Two new-old pipelines
  - ONAP istanbul daily Index of /onap-integration/daily/onap-daily-dt-oom-istanbul
  - ONAP master daily Index of /onap-integration/daily/onap-daily-dt-oom-master
  - Executed on DT lab - thanks Andreas Geißler !!!
- basic_vm regression - probably that change [AAI] Updates for AAI Jakarta to release latest versions (I9a068590) · Gerrit Code Review (onap.org) provides it - needs to be discussed with OOM team
- good resutls on Istanbul daily Xtesting ONAP Results
- Krzysztof Kuzmicki works on the CMPv2 tests
New Q&A review Morgan Richomme
- Few Q&A wiki pages from Morgan about ONAP integration related daily tasks:
- Till the end of the March it will be a fixed point on our agends: Morgan Q&A
- One more wiki page prepared by Sylvain Desbureaux : Daily Deployments and gating
Use case automation status:
- No update
Run tests on UNH lab
- Lab is ready to use
- I created an epic for that [INT-2085] Run ONAP tests on UNH lab - ONAP JIRA

02 Mar 2022

New committers: marcin krasowski Maciej Lisowski congratulations!
Lab status
- Istanbul
  - daily istanbul is back (problem due python dependency in k8s kubespray installer, increase of mino version 1.15→1.16 foxed the issue)
  - results are very good (as for the release validation): https://logs.onap.org/onap-integration/daily/onap_daily_pod4_istanbul/2022-03/02_04-01/
  - note it includes the log4j changes (istanbul = istanbul oom branch)
  - problem solved also on DT side, daily shall be back with new name soon
  - basic_cnf_macro not integrated in istanbul - not in istanbul xtesting dockers. But test done on istanbul with master xtesting docker shows that basic_cnf_macro is PASS (no regression) Lukasz Rajewski
- master
  - daily master error seem reproducible
    - regressions on basic_vm (so cleanup of volume) => needs to be investigated AP: Michał Jagiełło
    - errors on *_macro due to bad cleanup => Michał Jagiełło patch merged today
  - weekly restarted thsi week end
    - tern results: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2022-02/25_20-26/legal/tern/index.html. Former user (Deleted) asks if tern shall be removed (JIRA for that). unitl scancode is not fullyu integrate, no problem to keep tern in weekly (not run so frequently..)
    - versions results: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2022-02/25_20-26/security/versions/versions.html + Amy Zwarico Paweł Wieczorek rouzaut
    - internal certificates: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2022-02/25_20-26/infrastructure-healthcheck/internal_check_certs/internal_check_certs/certificates.html
      - in addition of robot, issue with ejbca and multicloud-fcaps
    - stability tests not run due to basic_vm regression
Use case automation status:
- DCAE related use cases
  - Update on 5Gbulkpm: Krzysztof Kuzmicki fixed most of the issue, shall be ready now. MR created to populate chartmuseum to be reviewed and merged
  - next planned cmpv2
  - dcaemod more complex...
- 5G Slicing: policy code merged in demo
- kafka HC => Morgan Richomme created a page to explain how the issue coudl be solved: Refactoring of tests due to kafka refactoring. Krzysztof Kuzmicki indicated that it woudl also be possible to call kubernetes command from the robot test
- Discussion on AAI stress tests with ATT, ATT gatling scripts shall be available and reusable for tests

23 Feb 2022

Lab status
- Istanbul
  - results not bad. Remaining test errors due to DCAE changes but a patch is in gate - no results due to error below.
  - The Makupsafe issue (jinja2 upstream dependency) is also responsible for the istanbul k8s installation issue
- Master
  - deployment OK but all smoke tests failing due to Makupsafe
- pb Markupsafe lib:
  - ugly test merged...https://gerrit.onap.org/r/c/integration/xtesting/+/127250 => tests shall be run on Master tomorrow (does not fix the istanbulinstallation issue)
  - I tried a cleaner way (https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk/-/merge_requests/334) but all tests are failing...wait for Michał Jagiełło comeback to check the issue and be sure to do thing properly on SDK versioning
jakarta tests
- 5G Slicing automation follow up
  - All patches of Maciej Lisowski merged
  - Question from Ahila P on simulators sent by mail. For the moment 2 of the 3 simulators are in the wiki. Action morgan push simu to the repo (the repo already exists)
- kubescape => demo next week
- Load test on AAI (using gatling) => action Morgan Richomme resync Michał Jagiełło /William Reehil / Christophe Auzizeau
- kafka (testing refactoring) => Refactoring of tests due to kafka refactoring, answer sent by mail to Fiachra Corcoran
Misc
- Morgan Richomme and Sylvain Desbureaux are stepping down
  - end of contributions: end of March 2022
  - end of Orange platform: end of Jakarta release (with minimal support)
  - Azure / NH won't be touched (gating will work, staging, ...) but resources to be identified to manage them...

16 Feb 2022

Lab status
- onap-k8s issue - there is an another issue after the dependency library new version release
- Orange lab - some issues with connection during random tests
- AAI-3450 - Getting issue details... STATUS still blocks master gating
Former user (Deleted) http://scancode.onap.eu/
- logging currently using GitHub account (consider using LF ID)
- needs to be discussed with seccom where the scan policies should be stored
- prepare a demo with two Integration projects:
  - integration/docker/onap-java11 · Gerrit Code Review
  - integration/docker/onap-python · Gerrit Code ReviewAdd images into scancode during docker-verify Jenkis job and run pipeline for them. We could analyse results and present that during one of the incoming PTL meetings

INT-2021 - Automate 5G Slicing use case in CI OTWARTY update
- Create a demo · Gerrit Code Review (onap.org) repo directory to keep there scripts and migrate Manual Configurations - Developer Wiki - Confluence (onap.org) into markdown files there to be able to keep history and review changes

09 Feb 2022

Lab status
- Orange lab down
- DT still down
- Gating is OK, but since today some A&AI issues observed
  - https://logs.onap.org/onap-integration/gating/127003-4/k8s-teardown/k8s-teardown/kubernetes-status/pod-onap-aai-schema-service-55f58c6b95-llqwr-aai-schema-service.log
basic_cnf_macro test
- Tests results - Summary (onap.org)
E2E network slicing use case
- INT-2064 - Automate 5G Slicing use case in CI - Manual configuration step 1 - SDC OTWARTY
- INT-2065 - Automate 5G Slicing use case in CI - Manual configuration step 2 - UUI Configuration OTWARTY
- INT-2066 - Automate 5G Slicing use case in CI - Manual configuration step 3 - MSB Configuration OTWARTY
- INT-2067 - Automate 5G Slicing use case in CI - Manual configuration step 4 - SO OTWARTY
- INT-2068 - Automate 5G Slicing use case in CI - Manual configuration step 5 - OOF Configuration OTWARTY
- INT-2069 - Automate 5G Slicing use case in CI - Manual configuration step 6 - Policy Creation Steps OTWARTY
- INT-2070 - Automate 5G Slicing use case in CI - Manual configuration step 7 - AAI Configuration OTWARTY
- INT-2071 - Automate 5G Slicing use case in CI - Manual configuration step 8 - ConfigDB OTWARTY
- INT-2072 - Automate 5G Slicing use case in CI - Manual configuration step 9 - SDNC OTWARTY
- INT-2073 - Automate 5G Slicing use case in CI - prepare a demo repo section OTWARTY

02 Feb 2022

Ahila P Network slicing automation
- at the beginning Integration team needs to take a look on the first phase: Manual Configurations - Developer Wiki - Confluence (onap.org) - preparing an ONAP for instantiation
  - several steps - for sure some missing gaps in ONAP SDK
- There is a plan to create a scenario in pythonsdk-tests repo for that use case
- [INT-2021] Automate 5G Slicing use case in CI - ONAP JIRA ticket which is going to be used to plan a work
- Automate use case 5G Slicing in CI - Developer Wiki - Confluence (onap.org) ← links shared by Ahila P

List of log4j vulnerability impacted pods in ONAP

Today morning Trivy was ran on ONAP daily master

Pods
onap-cli-757c76f546-zfd4m in vulnerable!
onap-message-router-0 in vulnerable!
onap-sdnrdb-coordinating-only-6f6d6b887b-kzr9x in vulnerable!
onap-sdnrdb-master-0 in vulnerable!
onap-sdnrdb-master-1 in vulnerable!
onap-sdnrdb-master-2 in vulnerable!
onap-vfc-huawei-vnfm-driver-57fddd8bc7-nqn7b in vulnerable!
onap-vnfsdk-56d9fb886f-sk4s4 in vulnerable!

Images:
nexus3.onap.org:10001/onap/cli:6.0.1 -> onap-cli-757c76f546-zfd4m
nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.3.0 -> onap-message-router-0
docker.nexus.azure.onap.eu/bitnami/elasticsearch:7.9.3 -> onap-sdnrdb-coordinating-only-6f6d6b887b-kzr9x, onap-sdnrdb-master-X
nexus3.onap.org:10001/onap/vfc/nfvo/svnfm/huawei:1.3.9 -> onap-vfc-huawei-vnfm-driver-57fddd8bc7-nqn7b
nexus3.onap.org:10001/onap/vnfsdk/refrepo:1.6.3 -> onap-vnfsdk-56d9fb886f-sk4s4

There are more and more critical issue found, but these are just for cve-2021-44228

Disable portal tests Krzysztof Kuzmicki
Former user (Deleted) starts http://scancode.onap.eu/ . Thanks!

26 Jan 2022

Lab status
- Better results, but due to Nexus restart and no images all tests failed
- cds-resource-resolution is still failing... looks the instance is created before tests is executed
Integration risks for Jakarta Jakarta Risks - Developer Wiki - Confluence (onap.org)
- Main risks are resources: labs and humans
Nexus restart https://lists.onap.org/g/onap-discuss/message/23758 should we restart the images creation?
- Yes, after Jenkins and Nexus are back again jobs below need to be rerun

19 Jan 2022

Nexus doesn't work - tests doesn't work....
- LF ticket created - Theme Extension - Linux Foundation JIRA
- gatig tests works due to Azure cache - images are not downloaded from nexus during each execution
When it worked we had several tests issues
- needed to check if we don't use same kubeconfig on many tests pipelines
  - that can create conflicts because one test tries to create an instance of existing resource
Kubescape tests
- could be useful but won't replace already existing tests
  - root pods check based just on the kubernetes resource definition (pod, deployment, statefulset etc.) and not check if the root user is disabled on the image
- could be possible to add them into pipeline soon

Azure lab - should we migrate tests there? - discussion
- there are currently 3 Azure labs
  - 2 gatings and 1 integration lab with ONAP
- needs to ask TSC if it's possible to add one more to run daily tests - outside of any company infrastructure (currently Orange, Nokia, DT)
- we could run daily on the 1 integration lab machine, but there is a risk we won't have a working lab if during daily execution ONAP creation fails
Migration Integration repos to GitLab?
- it's not required and will take a lot of time and resources, leave it as it is
scancode.io - Former user (Deleted) back and would work on that

Integration weekly meetings 2021 archives

Integration weekly meeting 2020 archives

Integration weekly meeting 2019 archives

Integration weekly meeting 2018 archives

Integration weekly meeting 2017 archives