Introduction

CSIT ("Continuous System and Integration Testing", even though you could also read that as "Component-Specific Integration Testing") cases in ONAP exist typically for verifying functionalities of a single or few ONAP components in limited testing environments (suitable for lightweight installation in local development environment and Jenkins) and often rely on simulators. Currently these tests are executed once per day on images from master branch (as well as on maintenance branches of selected previous releases) or whenever the test cases themselves are committed for review. This means that by ONAP's current automated verification procedure any new bugs that could be caught by the existing CSIT cases are not discovered until the bugs are already merged to master. In addition, any failures in daily CSIT Jenkins are not sending any kind of notifications or raising tickets or alarms, so the teams are not even getting any automated feedback from the failures that can therefore go days or in worst cases weeks without notice (much less action).

...

• The most important thing needed to incorporate CSIT cases into code review verifications smoothly is to put them under the same repos so any code commit that changes some already verified functionality (or introduces some new functionality that requires new tests) also can (and must) bring the related CSIT changes. 
  
  • This would mean that current repository structures regarding CSIT would be heavily redesigned and only common, generic parts would remain under existing csit repo
  • If there are CSIT cases that are responsible for verifying functionalities in multiple different repositories at the same time, the redesign would become complicated (such projects should probably reconsider their repo structures, or alternatively the scope of the CSIT cases in question would have to be reconsidered)
• CSIT environment setup should be enhanced to be able to use images coming from the review branch 
  • If the images from each new review patch are uploaded to Nexus, we would be using multiple times more space there than we do now (and if we do not, test env deployment would need some more redesign)
• Execution times of the CSIT tests should be shortened to the absolute minimum 
  • A major part of this is closely related to the product maturity in general - image sizes (which  affects the download time from Nexus) and container startup time optimization could help a lot also in CSIT cases where significant amount of time is spent in setting up the test environment 
    • Building a new image and uploading it to Nexus for every new review patch would make this optimization even more crucial
  • Reducing all kinds of test-specific retries and waiting times to absolute minimum but no more than that
    • Note that the other side of this coin is test stability in different environments with varying resources, so special care also needs to taken to ensure that timeouts are not overoptimized
  • In the long run this requirement is also at direct odds with the requirement that every integration-level bug correction and new feature should be verified in CSIT (if you didn't have this requirement before, you have it now  ) - eventually the total mass of things that need to be covered in CSIT will (or at least should) grow too big to be executed in every review verification
    • Streamlined set of critical CSIT tests executed in review verification may have to be separated from full regression set that would be executed only on merged code
• CSIT tests should be reliable and working
  • non-problematic code changes would always pass on the first attempt without needing to retry them a couple of times just in case they failed for some unrelated random causes
  • people are less likely to just ignore and override negative CSIT votes if they know failing CSIT is an indication of a real problem in the commit 

Manual CSIT Verification

...

on Local Development Environment

• Developers should always verify their changes with CSIT (and usually also enhance the related CSIT suite) before committing regardless of if it's also done as part of the automated review verification or not
  
  • Many of the action items listed above apply also here, but will be repeated 
• Tools for executing the CSIT cases in local development environments should be enhanced to enable easy deployment of local test images as part of the test environment setup
  • Currently there is partial support for this at least in some docker_run scripts, but this support does not extend to run-csit.sh and is not sufficiently documented
• Documentation of CSIT execution in local development environments should be up-to-date, sufficient and working for every project at all times so developers can actually use it to their advantage
  • Part of the problem is that there are project-specific requirements that need to be known - integration team should try to consolidate this and lay down some common requirements and, on the other hand, provide common ways of doing things that teams have had to solve for themselves until now
    • Concrete first step to remedy this is to go through all existing projects, see and document what it actually takes to make them pass, and see if there are any common improvements to be made (this can be driven by integration but needs obviously support from project teams)

...