Table of Contents

Addresses 

Test Flow Description

According to requirements the test flow included following operation:

• Create new anchor with unique name in given dataspace
• Create data node - full data tree upload for given anchor
• Update data node - node fragment replacement 
• Remove anchor (and associated data)

The dataspace and schema set were predefined.

The service functionality was tested in two modes:

• Embedded – the CPS was accessed directly via Java API
• Remote – The CPS was accessed using REST API

Load:

• The flow was repeated in a loop multiple times (defined with circles parameter);
  no delay between loops
• The flow loops were executed in a multiple concurrent threads (threads parameter);
  all threads were started subsequently with no delay
• each thread used same CPS service (embedded case) and REST client (remote case) instances
• All the data used (except schema set preparation) was taken from memory (loaded on preparation stage)

Full test application flow is shown on diagram below

Test Data

Yang model (schema set) based on:

• iana-if-type@2017-01-19.yang
• ieee802-dot1q-types.yang
• ietf-inet-types@2013-07-15.yang
• ietf-interfaces@2018-02-20.yang
• ietf-l2-topology@2020-11-15.yang
• ietf-l2-topology-state@2020-11-15.yang
• ietf-network@2018-02-26.yang
• ietf-network-state@2018-02-26.yang
• ietf-network-topology@2018-02-26.yang
• ietf-network-topology-state@2018-02-26.yang
• ietf-yang-types@2013-07-15.yang

Full data JSON is (taken as is from from RFC-8944 Appendix-B):

• ietf-network-sample-rfc8944.json (52 data nodes total)

Update data:

// node parent xpath: /networks/network[@network-id='l2-topo-example']/node[@node-id='D3']
// replacement fragment:
{ 
    "ietf-l2-topology:l2-node-attributes":
        {
	    	"management-address": [ "192.0.2.3", "2001:db8:0:3::" ]
		} 
}

Environment

The load test was performed using dedicated Spring Boot Application. The application was executed directly from IDE.

Remote services were deployed as docker containers. Docker was running on same machine.

Hardware details

Laptop DELL Latitude 5500

• Processors: 8 × Intel® Core™ i7-8665U CPU @ 1.90GHz
• Memory: 15,5 GiB of RAM
• lshw > lshw.txt

Software details

OS

• Operating System: Kubuntu 20.04
• KDE Plasma Version: 5.18.5
• KDE Frameworks Version: 5.68.0
• Qt Version: 5.12.8
• Kernel Version: 5.4.0-70-generic
• OS Type: 64-bit

Docker

• Docker version 20.10.5, build 55c4c88

Test application runtime

• OpenJDK version 13.0.4, 2020-07-14
• OpenJDK Runtime Environment (build 13.0.4+8-Ubuntu-120.04)
• OpenJDK 64-Bit Server VM (build 13.0.4+8-Ubuntu-120.04, mixed mode)
• java -XX:+PrintFlagsFinal -version > java-print-flags-final.txt

Test Results

CPS revision

Tested CPS from a master branch at state on 12 Apr 2021 
representing the Honolulu 1.0.1 + DELETE ANCHOR functionality targeted for Istanbul release

Operation execution time

The operation per seconds was calculated as 1000/(exec time in millis).

Resource usage

Below are resource usage diagrams (VisualVM monitor screenshots) for used Threads/Circles combinations.

Embedded then remote load tests were performed via single application execution (see test flow diagram above)

Threads: 2 Circles: 500	Threads: 10 Circles: 100
Threads: 20 Circles: 50	Threads: 50 Circles: 20

Following (calculated by default) heap settings were used:

ErgoHeapSizeLimit                        = 0
HeapSizePerGCThread                      = 43620760
InitialHeapSize                          = 260046848
LargePageHeapSizeThreshold               = 134217728
MaxHeapSize                              = 4148166656
NonNMethodCodeHeapSize                   = 5836300
NonProfiledCodeHeapSize                  = 122910970
ProfiledCodeHeapSize                     = 122910970

Full dump of JVM flags is listed in attachment java-print-flags-final.txt

Test Application

Source

The source of test application is attached: cps-load-test-application.zip

Setting up

Unzip into cps folder.

Update root pom.xml file to include cps-load-test-application

...
    <modules>
        <module>cps-dependencies</module>
        <module>cps-bom</module>
        <module>cps-parent</module>
        <module>cps-service</module>
        <module>cps-rest</module>
        <module>cps-ncmp-service</module>
        <module>cps-ncmp-rest</module>
        <module>cps-ri</module>
        <module>checkstyle</module>
        <module>spotbugs</module>
        <module>cps-application</module>
        <!-- add following-line -->
        <module>cps-load-test-application</module>
    </modules>
...

Refresh the maven project. New module will appear as on screenshot below

Create run configuration using same environment variables as for core CPS Application

Tests related configuration is allocated within cps-load-test-application/src/main/resources/application.yml in load-test section 
like below

load-test:
    preset:
        dataspace: test-dataspace
        schema-set: test-schema-set
        anchor-prefix: test-anchor-
    embedded:
        enabled: true
    remote:
        enabled: true
        # resources zip file is corrupted on build, so referencing original from resources
        resources-zip: src/main/resources/yang/network-topology.zip
        base-url: http://localhost:8883/cps/api/
        auth:
            username: ***
            password: ***
    # number of threads
    threads: 50
    # number of times the flow repeated within a thread
    circles: 20
    # thread termination timeout
    termination-timeout-seconds: 1200

spring:
...