Versions Compared

Old Version 70

changes.mady.by.user Toine Siebelink

Saved on

compared with

New Version Current

changes.mady.by.user Toine Siebelink

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents









CPS Team Testing Strategy

  1. Test Strategy.pptx
  2. Current Tests with Vertical Slice Example

CPS Test Overview

CPS testing strategy follows best in class quality principles and practices; based on the Testing Pyramid as recommended by Martin Fowler.


credit: https://martinfowler.com/articles/practical-test-pyramid.html


type os ImplementationIntegration Unit

This Diagram shows an overview of the different

types of test and their scope related to the architecture layers

Color Coding

Drawio
bordertrue
diagramNameCPS Test Overview
simpleViewerfalse
width400600
linksauto
tbstyletop
lboxtrue
diagramWidth1621
revision2


Source Code


Unit Test


Semi Integration

Test
(Service Tests)


Integration Test


Over time we have developed an extensive set of tests:

Unit Tests. CPS uses Groovy & Spock for unit testing
1

(UI) Integration test (since we don't have a UI these start at the REST layer of our components)

    1. CSIT Component System Integration Test.  CPS utilizes Robot framework for CSIT tests..
    2. K6 Integration Test
    3. K6 Performance Tests
      1. Performance Graphs

Unit Tests. CPS uses Groovy & Spock for unit testing

2

Service Test (or Semi-Integration Test) using Spring Boot Test Containers for Postgress Postgres and Kafka and Mocked/Stubbed Services for external components like DMI and Policy Executor. These test are written in Groovy & Spock

    1. Functional
    2. Performance
      1. Performance Graphs
3

(UI) Integration test (since we don't have a UI these start at the REST layer of our components)

    1. CSIT Component System Integration Test.  CPS utilizes Robot framework for CSIT tests..
    2. K6 Integration Test
    3. K6 Performance Tests
      1. Performance Graphs
4

Contract Testing (not visible in diagram)
contact testing is implicit sas CPS is build using a contract-first approach. ie.  the Java Interface is generated from the Open-API )(swagger) contracts and therefor always adheres to it.
We also have MVC test on each controller method to verify the correct mapping of parameters etc.

Integration Test

CSIT Integration Test

K6 Test Overview

<TBD>

Service / Semi-Integration Test

Persistence Testing

  1. CPS Persistence layer (semi-integration) testing
  2. Liquibase Change Log (sets) Testing (test DB Upgrade)

Contract Testing

  1. Contract Testing.pptx
  2. CPS has developed stubs for clients. 
    1. Link to Gerrit patch:  https://gerrit.onap.org/r/c/cps/+/129256
    2. Jira Link:
      Jira Legacy
      serverSystem Jira
      serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
      keyCPS-919

Performance Testing

Automated Performance Test

Our K6 test now includes automated performance testing for some (NCMP) Test Cases (KPIs). See K6 Test Overview

Performance Test Stubs

  1. NCMP Performance Test With Wiremock (not in use amymore)
  2. TBC DMI Stub
  3. TBC Policy Executor Stub 

Performance Test Results

This table is to track once-off performance test exercises/results, preferrable on a child page here.

DateJiraReportNotes
TBD

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-2371

CM Data Notification Subscription Create Performance
TBD
Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-2329
CM Data Notification Forwarding Performance
Jul 2024
NCMP: Read data operation resource API Performance for multiple cm-handles
Mar 2024
Performance test for updating YANG schema set API
Feb 2024
TBC

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-2049

Alternate ID performance testing
August

Sep 2023

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-1777

Performance plots for CPS in Jenkins
Aug 2023
Montreal Read/Write Performance
Apr 2023
CPS-Core Read & Write (large load) Test ResultsUsed Data Schema and Samples provides by Wipro
Feb 2023
CPS Analyse Heap-Memory uselooks like some attachments got lost in ONAP Wiki Cleanup
Jan 2023
NCMP De-Register API PerformanceNCMP De-registration Performance test guide
Jan 2023
Performance: disable CPS-temporal notifications
Sep 2022
NCMP Performance Test With Wiremock
Nov 2021

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-307

CPS Basic Performance (Load) Test

Testing Frameworks

Groovy & Spock

  1. Groovy & Spock Test Code Conventions
  2. OneSummit Groovy & Spock Workshop Nov 2022 (includes sample project & exercises!)
  3. Additional resources:
    1. https://www.baeldung.com/groovy-spock
    2. http://spockframework.org/spock/docs
    3. https://github.com/spockframework/spock-example
    4. http://groovy-lang.org/documentation.html

Other Test / Verification Tools

Read the Docs (RTD)

Prerequisites

  1. Linux Environment (WSL works too)
  2. Python
  3. Tox  (see https://tox.wiki/en/3.14.1/install.html)

Steps

  1. (temporary) Modify docs/tox.ini (for example https://gerrit.onap.org/r/c/cps/+/135892/2/docs/tox.ini ) do NOT merge these changes into master!
    1. to use YOUR python version
    2. If you have a lot of new documentation you can (temporary) remove -W flag from the first two  sphinx-build commands (sections; tox and docs-linkcheck) so it doesn't stop on the first error
      (note there are always a lot of spelling 'warnings' so the third build command should always be run without the -W flag (section; docs-spellcheck) 
  2. cd into <your project location>docs
  3. run tox
  4. tip: First time run can take a few minutes! use -V or -VV for more verbose output to see what it is doing

Alternate checks

  1. Windows Subsystem (WSL):  doc8 *.rst --ignore D002 --ignore D004
    (need to ignore warnings about CR, trailing whitespace and line-length because of different filesystem)
  2. Note. If there are unexpected issues with Linelength ie. errors locally that are not causing problems in CI then these can be ignored using --ignore D001
    Thomas Kulik suspects this might have to do with doc8 tool version and he will report back to us on this issue

Reference Material

  1. Ref. Martin Fowler Practical-test-pyramid
    1. conclusion-summary (PPT)
  2. CSIT References
    1. Local Setup Using CSIT 
    2. Maximizing Benefits of CSIT in ONAP Development
    3. https://robotframework.org/