Integration weekly meeting 2022 archives
Jun 1, 2022
PTL
Integration committers cleanup after release:
@Morgan Richomme
@Bartek Grzybowski
@Maciej Lisowski
@marcin krasowski
@Illia Halych
Open nominations for Integration Project PTL - @Michał Jagiełło is going to step down from that role (but still being committer)
Lab status
A&AI fails on both Orange and DT daily
@Maciej Wereski Probably race condition on onap-aai-graphadmin-create-db-schema job
@Radoslaw Chmiel found a workaround?
Gating 4 was failing on
apt updateDisk was full
Remove docker artifacts with
docker system prune -a
Gating failing on another step: deploy_full (#2532722215) · Jobs · Orange-OpenSource / lfn / onap / onap_oom_automatic_installation · GitLab
Gating is going to be down due to Changes to GitLab.com public project CI/CD minute quotas | GitLab - gating pipelines are running on Orange (as we have still no GitLab CI on ONAP group) so we are going to reach the limit very soon
Tests on the UNH
LF created project ONAP / integration / chained-ci · GitLab
still no Integration project committers as members
University New Hampshire lab - Developer Wiki - Confluence (onap.org) ← wiki to follow the status of running tests on UNH lab
GitLab runner and jumphost servers up and running
GitLab runner not registered - there is no place to register it
Release of base Python/Java(?) images is needed
May 11, 2022
Lab status
still failing holmes-healthcheck on Orange lab
[HOLMES] Fixed the Healthcheck Problem (I64d5f326) · Gerrit Code Review (onap.org) OOM change merged
k8s version on gating (where we don't have that issue) is different than on Orange daily - should be checked if change version on Orange fix that issue
a lot of failures on security tests
@Fiachra Corcoran - waivers directory gerrit.onap Code Review - integration/seccom.git/tree - waivers/
good results on stability: both instantiation and onboarding
gating s3 error on tests results upload - no space left on the device
Tests on the UNH
still no LF agree on https://gitlab.com/onap/integration creation
we have an integration tenant on UNH lab
other projects as well and some of them already created their own ONAP instances (Policy for sure)
plan is to create a GitLab runner on UNH Infrastructure tenant and run Orange pipelines there
all integration commiters should have an access to UNH lab following that guide: ONAP IOL-UNH Lab - Developer Wiki - Confluence
Apr 20, 2022
Lab status
weekly tests Xtesting ONAP Results
versions and stability tests ran manually
MariaDB issues @Lukasz Rajewski
several changes:
[MARIADB] Fix wsrep software error (Ib7d8be50) · Gerrit Code Review (onap.org) - downgrade to still acceptable version by SECCOM
[MARIADB] Back to istanbul version of mariaDB image (I23b31db0) · Gerrit Code Review (onap.org) - back to the Istanbul version of MariaDB
[SO][MARIADB] Improve Mariadb connection (I3dc20f27) · Gerrit Code Review (onap.org) - WIP: Use sequential jbdc connection on SO
Wait for tests results to check if there is any improvement
E2E network slicing automation
@Maciej Lisowski provide a change to automate A&AI
@marcin krasowski works on SDC part automation
@Ahila P works on the CPS TBDMT part
Apr 6, 2022
Lab status
Rearrange tests execution order:
Shoulnd't security tests be executed on the beginning? Some tests on failure don't cleanup properly
onap-k8s and onap-k8s-teardown false-positives
Jakarta Integration Blocking points - Developer Wiki - Confluence (onap.org)
@Maciej Wereski mentioned about ETCD issue on version recommended by SECCOM
Issue: Inconsistent revision and data occurs · Issue #13766 · etcd-io/etcd (github.com)
SECCOM recommendation: gerrit.onap Code Review - integration/seccom.git/blob - recommended_versions.yaml
@Amy Zwarico @Paweł Pawlak take a look please
Mar 30, 2022
Lab status
Lack of stability mostly due to SDC onboarding issues
There is an idea to stop executing tests if previous stage failed. We have currently 4 stages:
Infra healthcheck
Healthcheck (Splitted into 2 on Orange)
Smoke tests (Splitted into 2 on Orange)
Onap security
So if stage 1 failed (Infra healthcheck) we don't execute any other tests
There is also an idea to reduce the number of tests to execute only tests which are not a part of another, e.g. basic_onboard tests is only onboarding service into SDC, but for many other tests we do the same.
@Morgan Richomme Q&A
Volunteers needed to manage gating and testing
Needs to ask TSC
Failing CSIT tests:
vfc-gvnfm-vnflcm-master-csit-sanity-check #3345 [Jenkins] (onap.org)
vfc-gvnfm-vnflcm-master-csit-sanity-check [Jenkins] (onap.org)
vfc-gvnfm-vnfres-master-csit-sanity-check [Jenkins] (onap.org)
vfc-gvnfm-vnfres-master-csit-sanity-check [Jenkins] (onap.org)
vfc-nfvo-driver-vnfm-gvnfm-master-csit-sanity-check [Jenkins] (onap.org)
vfc-nfvo-driver-vnfm-gvnfm-master-csit-sanity-check [Jenkins] (onap.org)
NOT OUR ISSUE
Jakarta Integration Blocking points - Developer Wiki - Confluence (onap.org)
Mar 16, 2022
Lab status
Regression candidate; Tests results - Summary (onap.org)
@Krzysztof Kuzmicki is working on:
cmpv2 - daily result cmpv2 (#2214007103) · Jobs · Orange-OpenSource / lfn / onap / xtesting-onap · GitLab @Krzysztof Kuzmicki
hv-ves
Feedback on New Q&A review & Demo weekly stability tests @Morgan Richomme & Demo kubescape
how to run stability tests:
link to the script:
link to the onap_bench library:
chaos tests:
link to the python package used in resiliency/chaos testing:
kubescape tests:
almost ready
link to the python package used in kubescape tests:
if we want to get more details we should create an ONAP account on kubescape to push results there (or create own kubescape instance)
exchange with seccom @Michał Jagiełło
Use case automation status:
@Maciej Lisowski @marcin krasowski any update?
What's next with the robot:
testsuite/python-testing-utils · Gerrit Code Review (onap.org) is Python 3.X compatible (no Python 3.10, but it's more with testing framework there and no code)
xtesting-smoke-usecases-robot docker image has no Python 2.7 installed
nexus3.onap.org:10001/onap/testsuite:1.8.0 image has Ubuntu installed and Python 2.7, bash etc installed there
Do not trust license scanning tools: [INT-2086] onap/integration-python:9.1.0 contains libuuid which is GPL-3.0 - ONAP JIRA
tern marked libuuid as GPLv3
ScanCode.io marked libuuid as GPLv3
libuuid has BSD license COPYING « libuuid - util-linux/util-linux.git - The util-linux code repository. (kernel.org)
@Thomas Kulik reported we have an empty doc directory in pythonsdk-tests, We should:
provide tests documentation, or
remove that directory
@Michał Jagiełło is going to create a ticket for @Morgan Richomme to provide some doc there
Mar 9, 2022
Lab status
Two new-old pipelines
ONAP istanbul daily Index of /onap-integration/daily/onap-daily-dt-oom-istanbul
ONAP master daily Index of /onap-integration/daily/onap-daily-dt-oom-master
Executed on DT lab - thanks @Andreas Geißler !!!
basic_vm regression - probably that change [AAI] Updates for AAI Jakarta to release latest versions (I9a068590) · Gerrit Code Review (onap.org) provides it - needs to be discussed with OOM team
good resutls on Istanbul daily Xtesting ONAP Results
@Krzysztof Kuzmicki works on the CMPv2 tests
New Q&A review @Morgan Richomme
Few Q&A wiki pages from Morgan about ONAP integration related daily tasks:
Till the end of the March it will be a fixed point on our agends: Morgan Q&A
One more wiki page prepared by @Sylvain Desbureaux : Daily Deployments and gating
Use case automation status:
No update
Run tests on UNH lab
Lab is ready to use
I created an epic for that [INT-2085] Run ONAP tests on UNH lab - ONAP JIRA
Mar 2, 2022
New committers: @marcin krasowski @Maciej Lisowski congratulations!
Lab status
Istanbul
daily istanbul is back (problem due python dependency in k8s kubespray installer, increase of mino version 1.15→1.16 foxed the issue)
results are very good (as for the release validation): https://logs.onap.org/onap-integration/daily/onap_daily_pod4_istanbul/2022-03/02_04-01/
note it includes the log4j changes (istanbul = istanbul oom branch)
problem solved also on DT side, daily shall be back with new name soon
basic_cnf_macro not integrated in istanbul - not in istanbul xtesting dockers. But test done on istanbul with master xtesting docker shows that basic_cnf_macro is PASS (no regression) @Lukasz Rajewski
master
daily master error seem reproducible
regressions on basic_vm (so cleanup of volume) => needs to be investigated AP: @Michał Jagiełło
errors on *_macro due to bad cleanup => @Michał Jagiełło patch merged today
weekly restarted thsi week end
tern results: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2022-02/25_20-26/legal/tern/index.html. @Former user (Deleted) asks if tern shall be removed (JIRA for that). unitl scancode is not fullyu integrate, no problem to keep tern in weekly (not run so frequently..)
versions results: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2022-02/25_20-26/security/versions/versions.html + @Amy Zwarico @Paweł Wieczorek @rouzaut
internal certificates: https://logs.onap.org/onap-integration/weekly/onap_weekly_pod4_master/2022-02/25_20-26/infrastructure-healthcheck/internal_check_certs/internal_check_certs/certificates.html
in addition of robot, issue with ejbca and multicloud-fcaps
stability tests not run due to basic_vm regression
Use case automation status:
DCAE related use cases
Update on 5Gbulkpm: @Krzysztof Kuzmicki fixed most of the issue, shall be ready now. MR created to populate chartmuseum to be reviewed and merged
next planned cmpv2
dcaemod more complex...
5G Slicing: policy code merged in demo
kafka HC => @Morgan Richomme created a page to explain how the issue coudl be solved: Refactoring of tests due to kafka refactoring. @Krzysztof Kuzmicki indicated that it woudl also be possible to call kubernetes command from the robot test
Discussion on AAI stress tests with ATT, ATT gatling scripts shall be available and reusable for tests
Feb 23, 2022
Lab status
Istanbul
results not bad. Remaining test errors due to DCAE changes but a patch is in gate - no results due to error below.
The Makupsafe issue (jinja2 upstream dependency) is also responsible for the istanbul k8s installation issue
Master
deployment OK but all smoke tests failing due to Makupsafe
pb Markupsafe lib:
ugly test merged...https://gerrit.onap.org/r/c/integration/xtesting/+/127250 => tests shall be run on Master tomorrow (does not fix the istanbulinstallation issue)
I tried a cleaner way (https://gitlab.com/Orange-OpenSource/lfn/onap/python-onapsdk/-/merge_requests/334) but all tests are failing...wait for @Michał Jagiełło comeback to check the issue and be sure to do thing properly on SDK versioning
jakarta tests
5G Slicing automation follow up
All patches of @Maciej Lisowski merged
Question from @Ahila P on simulators sent by mail. For the moment 2 of the 3 simulators are in the wiki. Action morgan push simu to the repo (the repo already exists)
kubescape => demo next week
Load test on AAI (using gatling) => action @Morgan Richomme resync @Michał Jagiełło /@William Reehil / @Christophe Auzizeau
kafka (testing refactoring) => Refactoring of tests due to kafka refactoring, answer sent by mail to @Fiachra Corcoran
Misc
Morgan Richomme and Sylvain Desbureaux are stepping down
end of contributions: end of March 2022
end of Orange platform: end of Jakarta release (with minimal support)
Azure / NH won't be touched (gating will work, staging, ...) but resources to be identified to manage them...
Feb 16, 2022
Lab status
onap-k8s issue - there is an another issue after the dependency library new version release
Orange lab - some issues with connection during random tests
https://lf-onap.atlassian.net/browse/AAI-3450 still blocks master gating
@Former user (Deleted) http://scancode.onap.eu/
logging currently using GitHub account (consider using LF ID)
needs to be discussed with seccom where the scan policies should be stored
prepare a demo with two Integration projects:
integration/docker/onap-python · Gerrit Code ReviewAdd images into scancode during
docker-verifyJenkis job and run pipeline for them. We could analyse results and present that during one of the incoming PTL meetings
- INT-2021 - Automate 5G Slicing use case in CI OTWARTY update
Create a demo · Gerrit Code Review (onap.org) repo directory to keep there scripts and migrate Manual Configurations - Developer Wiki - Confluence (onap.org) into markdown files there to be able to keep history and review changes
Feb 9, 2022
Lab status
Orange lab down
DT still down
Gating is OK, but since today some A&AI issues observed
basic_cnf_macro test
E2E network slicing use case
- INT-2064 - Automate 5G Slicing use case in CI - Manual configuration step 1 - SDC OTWARTY
- INT-2065 - Automate 5G Slicing use case in CI - Manual configuration step 2 - UUI Configuration OTWARTY
- INT-2066 - Automate 5G Slicing use case in CI - Manual configuration step 3 - MSB Configuration OTWARTY
- INT-2067 - Automate 5G Slicing use case in CI - Manual configuration step 4 - SO OTWARTY
- INT-2068 - Automate 5G Slicing use case in CI - Manual configuration step 5 - OOF Configuration OTWARTY
- INT-2069 - Automate 5G Slicing use case in CI - Manual configuration step 6 - Policy Creation Steps OTWARTY
- INT-2070 - Automate 5G Slicing use case in CI - Manual configuration step 7 - AAI Configuration OTWARTY
- INT-2071 - Automate 5G Slicing use case in CI - Manual configuration step 8 - ConfigDB OTWARTY
- INT-2072 - Automate 5G Slicing use case in CI - Manual configuration step 9 - SDNC OTWARTY