CPS Testing

CPS Team Testing Strategy

1. Test Strategy.pptx

2. Current Tests with Vertical Slice Example

CPS Test Overview

CPS testing strategy follows best in class quality principles and practices; based on the Testing Pyramid as recommended by Martin Fowler.

credit: https://martinfowler.com/articles/practical-test-pyramid.html

Over time we have developed an extensive set of tests:

Integration Test

CSIT Integration Test

• CPS Integration Test Plan

• CSIT Testing for CPS

• Local Setup Using CSIT

• CPS-188 CSIT Testing for CPS

• Project-specific CSIT structure (non-CPS page)

K6 Test Overview

Grafana k6 is an open-source and extensible load tool. Grafana k6 helps the CPS engineering team prevent breaches of SLI (Service Level Indicators) and SLO (Service Level Objectives).

The k6 tests have been applied for NCMP (Network Config Management Proxy) APIs in the CPS Project.

There are two test profiles for NCMP: kpi and endurance

kpi Profile

The kpi profile is used for load (performance) testing.

It defines various scenarios to test different APIs offered in NCMP, including passthrough read/write operations, cm handle id searches, and legacy batch operations.

Each scenario specifies the executor type, execution function, rate, duration, and other parameters.

Please have a look at the details of NCMP Stone Tablet KPIs from the link: The Stone Tablet KPIs of NCMP

endurance Profile

The endurance profile is used for stability testing over extended periods for the APIs offered in NCMP.

For more details like characteristics and solution design, please have a look at the link: The Characteristics of endurance tests of NCMP

Service / Semi-Integration Test

• CPS (semi-) Integration Tests

Persistence Testing

1. CPS Persistence layer (semi-integration) testing

2. Liquibase Change Log (sets) Testing (test DB Upgrade)

Contract Testing

1. Contract Testing.pptx

2. CPS has developed stubs for clients. 

   1. Link to Gerrit patch:  https://gerrit.onap.org/r/c/cps/+/129256

   2. Jira Link: CPS-919: PoC Contract Stubs NCMP Rest EndpointsClosed

Performance Testing

Automated Performance Test

Our K6 test now includes automated performance testing for some (NCMP) Test Cases (KPIs). See K6 Test Overview

Performance Test Stubs

1. NCMP Performance Test With Wiremock (not in use anymore)

2. TBC DMI Stub

3. TBC Policy Executor Stub 

Performance Test Results

This table tracks once-off performance test exercises/results, preferably on a child page here.

Testing Frameworks

Groovy & Spock

1. Groovy & Spock Test Code Conventions

2. OneSummit Groovy & Spock Workshop Nov 2022 (includes sample project & exercises!)

3. Additional resources:

   1. https://www.baeldung.com/groovy-spock

   2. http://spockframework.org/spock/docs

   3. https://github.com/spockframework/spock-example

   4. http://groovy-lang.org/documentation.html

Other Test / Verification Tools

Read the Docs (RTD)

Prerequisites

1. Linux Environment (WSL works too)

2. Python

3. Tox  (see https://tox.wiki/en/3.14.1/install.html)

Steps

1. (temporary) Modify docs/tox.ini (for example https://gerrit.onap.org/r/c/cps/+/135892/2/docs/tox.ini ) do NOT merge these changes into master!

   1. to use YOUR Python version

   2. If you have a lot of new documentation you can (temporarily) remove the -W flag from the first two  sphinx-build commands (sections; tox and docs-linkcheck) so it doesn't stop on the first error
      (note there are always a lot of spelling 'warnings' so the third build command should always be run without the -W flag (section; docs-spellcheck) 

2. cd into <your project location>docs

3. run tox

4. tip: First-time run can take a few minutes! use -V or -VV for more verbose output to see what it is doing

Alternate checks

1. Windows Subsystem (WSL):  doc8 *.rst --ignore D002 --ignore D004
   (need to ignore warnings about CR, trailing whitespace and line length because of different filesystems)

2. Note. If there are unexpected issues with Linelength ie. errors locally that are not causing problems in CI then these can be ignored using --ignore D001
   @Thomas Kulik suspects this might have to do with the doc8 tool version and he will report back to us on this issue

Reference Material

1. Ref. Martin Fowler Practical-test-pyramid

   1. conclusion-summary (PPT)

2. CSIT References

   1. Local Setup Using CSIT 

   2. Maximizing Benefits of CSIT in ONAP Development

   3. https://robotframework.org/