Versions Compared

Old Version 77

changes.mady.by.user Toine Siebelink

Saved on

compared with

New Version Current

changes.mady.by.user Toine Siebelink

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents









CPS Team Testing Strategy

  1. Test Strategy.pptx
  2. Current Tests with Vertical Slice Example

CPS Test Overview

CPS testing strategy follows best in class quality principles and practices; based on the Testing Pyramid as recommended by Martin Fowler.


credit: https://martinfowler.com/articles/practical-test-pyramid.html


Source CodeIntegration Unit

This Diagram shows an overview of the different types of test and their scope related to the architecture layers

Color Coding

Drawio
bordertrue
diagramNameCPS Test Overview
simpleViewerfalse
width600
linksauto
tbstyletop
lboxtrue
diagramWidth1621
revision2


Source Code


Unit Test


Semi Integration Test

(Service Tests)


Integration Test


Over time we have developed an extensive set of tests:

(UI) Integration test (since we don't have a UI these start at the REST layer of our components)

  • CSIT Component System Integration Test.  CPS utilizes Robot framework for CSIT tests..
  • K6 Integration Test
    • K6 Performance TestsPerformance GraphsUnit Tests. CPS uses Groovy & Spock for unit testing
    1

    Unit Tests. CPS uses Groovy & Spock for unit testing

    2

    Service Test (or Semi-Integration Test) using Spring Boot Test Containers for Postgress Postgres and Kafka and Mocked/Stubbed Services for external components like DMI and Policy Executor. These test are written in Groovy & Spock

      1. Functional
      2. Performance
        1. Performance Graphs
    3

    (UI) Integration test (since we don't have a UI these start at the REST layer of our components)

      1. CSIT Component System Integration Test.  CPS utilizes Robot framework for CSIT tests..
      2. K6 Integration Test
      3. K6 Performance Tests
        1. Performance Graphs
    4

    Contract Testing (not visible in diagram)
    contact testing is implicit sas CPS is build using a contract-first approach. ie.  the Java Interface is generated from the Open-API )(swagger) contracts and therefor always adheres to it.
    We also have MVC test on each controller method to verify the correct mapping of parameters etc.

    Integration Test

    CSIT Integration Test

    K6 Test Overview

    <TBD>

    Service / Semi-Integration Test

    Persistence Testing

    1. CPS Persistence layer (semi-integration) testing
    2. Liquibase Change Log (sets) Testing (test DB Upgrade)

    Contract Testing

    1. Contract Testing.pptx
    2. CPS has developed stubs for clients. 
      1. Link to Gerrit patch:  https://gerrit.onap.org/r/c/cps/+/129256
      2. Jira Link:
        Jira Legacy
        serverSystem Jira
        serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
        keyCPS-919

    Performance Testing

    Automated Performance Test

    Our K6 test now includes automated performance testing for some (NCMP) Test Cases (KPIs). See K6 Test Overview

    Performance Test Stubs

    1. NCMP Performance Test With Wiremock (not in use amymore)
    2. TBC DMI Stub
    3. TBC Policy Executor Stub 

    Performance Test Results

    This table is to track once-off performance test exercises/results, preferrable on a child page here.

    DateJiraReportNotes
    TBD

    Jira Legacy
    serverSystem Jira
    serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
    keyCPS-2371

    		CM Data Notification Subscription Create Performance
    TBD
    Jira Legacy
    serverSystem Jira
    serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
    keyCPS-2329
    		CM Data Notification Forwarding Performance
    Jul 2024
    		NCMP: Read data operation resource API Performance for multiple cm-handles
    Mar 2024
    		Performance test for updating YANG schema set API
    Feb 2024

    Jira Legacy
    serverSystem Jira
    serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
    keyCPS-2049

    		Alternate ID performance testing
    Sep 2023

    Jira Legacy
    serverSystem Jira
    serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
    keyCPS-1777

    		Performance plots for CPS in Jenkins
    Aug 2023
    		Montreal Read/Write Performance
    Apr 2023
    		CPS-Core Read & Write (large load) Test ResultsUsed Data Schema and Samples provides by Wipro
    Feb 2023
    		CPS Analyse Heap-Memory uselooks like some attachments got lost in ONAP Wiki Cleanup
    Jan 2023
    		NCMP De-Register API PerformanceNCMP De-registration Performance test guide
    Jan 2023
    		Performance: disable CPS-temporal notifications
    Sep 2022
    		NCMP Performance Test With Wiremock
    Nov 2021

    Jira Legacy
    serverSystem Jira
    serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
    keyCPS-307

    		CPS Basic Performance (Load) Test

    Testing Frameworks

    Groovy & Spock

    1. Groovy & Spock Test Code Conventions
    2. OneSummit Groovy & Spock Workshop Nov 2022 (includes sample project & exercises!)
    3. Additional resources:
      1. https://www.baeldung.com/groovy-spock
      2. http://spockframework.org/spock/docs
      3. https://github.com/spockframework/spock-example
      4. http://groovy-lang.org/documentation.html

    Other Test / Verification Tools

    Read the Docs (RTD)

    Prerequisites

    1. Linux Environment (WSL works too)
    2. Python
    3. Tox  (see https://tox.wiki/en/3.14.1/install.html)

    Steps

    1. (temporary) Modify docs/tox.ini (for example https://gerrit.onap.org/r/c/cps/+/135892/2/docs/tox.ini ) do NOT merge these changes into master!
      1. to use YOUR python version
      2. If you have a lot of new documentation you can (temporary) remove -W flag from the first two  sphinx-build commands (sections; tox and docs-linkcheck) so it doesn't stop on the first error
        (note there are always a lot of spelling 'warnings' so the third build command should always be run without the -W flag (section; docs-spellcheck) 
    2. cd into <your project location>docs
    3. run tox
    4. tip: First time run can take a few minutes! use -V or -VV for more verbose output to see what it is doing

    Alternate checks

    1. Windows Subsystem (WSL):  doc8 *.rst --ignore D002 --ignore D004
      (need to ignore warnings about CR, trailing whitespace and line-length because of different filesystem)
    2. Note. If there are unexpected issues with Linelength ie. errors locally that are not causing problems in CI then these can be ignored using --ignore D001
      Thomas Kulik suspects this might have to do with doc8 tool version and he will report back to us on this issue

    Reference Material

    1. Ref. Martin Fowler Practical-test-pyramid
      1. conclusion-summary (PPT)
    2. CSIT References
      1. Local Setup Using CSIT 
      2. Maximizing Benefits of CSIT in ONAP Development
      3. https://robotframework.org/