...
The following diagram illustrates the proposed new review verification and merge verification flows
...
- Developer pushes a commit to Gerrit
- Gerrit triggers review verification job in Jenkins
- The templates for review verification job are in ci-management/jjb/integration/integration-templates-csit.yaml, named as "{project-name}-review-verification-<artifact type>-{stream}"
- The only currently supported artifact type in review verification templates is maven
- The trigger is "gerrit-trigger-patch-submitted" and it is activated by any file change in the project
- the job starts with artifact build
- Artifacts should be created from the patch under review
- Currently the first step just compiles the artifacts locally (with unit tests), but additional operations - for example sonar analysis - should be considered here
- The templates for review verification job are in ci-management/jjb/integration/integration-templates-csit.yaml, named as "{project-name}-review-verification-<artifact type>-{stream}"
- Docker build is started
- Docker build step builds docker image(s) to local temporary docker repository
- The docker build scripts and the created artifacts are completely based on the submitted patch in Gerrit
- The actual build script is specific to the project and has to be specified in the specific docker build job definition
- The build should differentiate local image from any that could be found from Nexus3Missing Nexus
- For example, no repository prefix in the image name is would be an obvious hint
- CSIT job is started
- this step calls csit/run-project-csit.sh under the project's root directory via integration-run-project-test builder defined in ci-management/integration/integration-macros.yaml
- run-project-csit.sh is project-specific, but can take advantage of common scripts in integration/csit if considered useful
- run-project-csit.sh can and should run all the test plans of the project in one go and provide the results under csit/archives directory
- CSIT job pulls the docker images created in step 4. (by using project-specific setup scripts) and tests them
- The test cases to be executed are also from the patch under review
- the setup script responsible of pulling the required images should ensure that the images are local
- To be considered: should CSIT ever be able to pull the images under test from Nexus?
- Review verification job gives +1 or -1 verify vote to the review based on the overall success of the verification
- Once the commit is approved (+2'd) and submitted, Gerrit triggers merge verification job in Jenkins
- The template for merge verification job is in ci-management/jjb/integration/integration-templates-csit.yaml and currently named as "{project-name}-merge-verification-<artifact type>-{stream}
- The only currently supported artifact type in merge verification templates is maven
- The trigger is "gerrit-trigger-patch-merged" and it is activated by any file change in the project
- the job starts with artifact build
- Artifacts should be created from master
- Sonar analysis and other possible additional artifact-related steps should be considered here
- Docker build is started
- Docker build step builds docker image(s) to local temporary docker repository
- The docker build scripts and the created artifacts are completely based on master
- The actual build script is the same as in review verification and used in the same way
- CSIT is started in exactly the same way as review verification
- CSIT job pulls the docker images created in step 10. (by using project-specific setup scripts) and tests them
...
- The project must define a review verification job based on one of "{project-name}--review-verification-<artifact type>-{stream}" templates (which is defined in ci-management/jjb/integration/integration-templates-csit.yaml)
- the only currently supported artifact type is maven - new templates must be created to support different artifact types (such as golang)
- The existing maven template assumes the following JJB variables for maven by default:
- mvn-goals: clean deploy
- mvn-params: <empty>
- The purpose of these variables
- The existing maven template assumes the following JJB variables for maven by default:
- is to build both maven artifacts and then docker images containing the artifacts, but to not push either to Nexus
- if your project uses different maven goals or parameters for achieving these, the above variables must be overridden appropriately in the project-specific review verification job definition
- is to build both maven artifacts and then docker images containing the artifacts, but to not push either to Nexus
- the only currently supported artifact type is maven - new templates must be created to support different artifact types (such as golang)
- in addition, the following JJB variables must be defined in the project-specific review verification job definition:
- project-name (project identifier that can be freely chosen and is used in generated job names)
- project (should correspond project's name in Gerrit)
- stream (list of supported branches)
- javamvn-version
- mvn-version
- mvn-settings
- mvn-global-settingssettings
- should be named as '<project-name>-settings'
- note that you may have to ask to add support for your '<project-name>-settings' in Jenkins Sandbox separately from LF you intend to test the job there
- The project must support CSIT as follows:
- The project must have csit/run-project-csit.sh that is responsible of the entire CSIT execution
- run-project-csit.sh must return exit code 0 on success and non-zero code on any failure - this is how the success/failure of the test suite(s) and +1/-1 verification vote is communicated to gerrit
- currently run-project-csit.sh must provide robot test results or the entire review verification fails
- A more generic basic template that allows other test frameworks should be added
- The setup scripts should must be able to use locally built docker images
- preferably the images should be tagged and referred to unambiguously so there is no risk of them being pulled from Nexus3
- The project must have csit/run-project-csit.sh that is responsible of the entire CSIT execution
- Following the integration/csit subdirectory structure and using its run-csit.sh (by cloning the integration/csit and copying the common scripts to workspace) is possible but not mandatory
- the csit subdirectory should also contain README.md that explains how to build the relevant docker images locally and run the CSIT with them
- The project should must define a merge verification job based on one of "{project-name}-merge-verification-<artifact type>-{stream}" templates (which are defined in ci-management/jjb/integration/integration-templates-csit.yaml)
Example project
CCSDK/distribution
...
- Same requirements apply for merge verification job definition as for review verification job
Example projects
There are a handful of projects and repositories that are already applying this design:
- ccsdk/integration
- ccsdk/oran
- cps
- dmaap/buscontroller
- optf/has
- optf/odf
CPS
This is the first project to create their functional tests directly under their own repository instead of moving them from integration/csit
- All the related job and template changes definitions done to ci-management repository are available in master
- ci-management/jjb/integrationcps/
- integration-maven-jobs.yaml
- integration/integration-templates-cps-csit.yaml
- ci-management/jjb/ccsdkintegration/
- distributionintegration-templates-csit.yamlccsdk
- integration-csitmacros.yaml
- ci-management/jjb/integrationcps/
- The changes to ccsdk/distribution cps repository itself are available under https://gerrit.onap.org/r/admin/repos/ccsdk/distributioncps
- the entire cps/csit subdirectory
Open questions and remaining tasks
- Is there any need for SNAPSHOT docker images in Nexus3?
- Is there any need for staging artifacts in Nexus? Who needs them, where and how? How does the staging repository even work?
- (actual tests)
- cps-application/pom.xml (for docker image build)
- cps/docker-compose/docker-compose.yml (for docker image startup)
- CPS also has some further elaboration on their CSIT under the following links:
- https://lf-onap.atlassian.net/wiki/display/DW/CPS-188+CSIT+Testing+for+CPS
- https://lf-onap.atlassian.net/wiki/display/DW/CPS+Developers+Team+Meeting+Recordings
- Note though that the above links are already partially obsolete; for example, at the time of their creation the example project was still ccsdk/integration rather than cps
Open questions and remaining tasks
- Use of Robot framework is currently mandatory - this should be improved
- Should CSIT be able to test docker images from Nexus? If so, under which circumstances and for what purpose?
- Currently there are verification templates available only for maven artifact type - others (for example golang) may have to be added
- This proposal replaces plain Maven/Makefile/other artifact build jobs completely
- Sonar builds seem to be completely separate scheduled builds anyway, and unsupported by any global templates?
- Ideally all reviews should start with Sonar build and fail the review on violations