Summary

This is an assessment of current performance of CPS operations, as of August 2023.

...

• NCMP CM-handle de-registration moved from average quadratic to linear time (constant rate). De-registering 20,000 CM-handles is thousands of times faster now.
• CPS Path Queries have moved from worst-case quadratic to linear time complexity, and best case linear to constant. Queries that took days previously take minutes or seconds now.
• Memory usage during read, query, and update operations has been decreased by more than 90% (10 times less memory).
• Not only has performance increased, but reliability and robustness. It is now possible for a single batch operation to handle tens of thousands of elements, where it would have failed before.

CPS Core Performance

Test Environment

TO DO Laptop specs

The performance tests are written in Groovy (a JVM language). As all CPS Core operations are synchronous, the results here are to be considered as single-threaded operation only.

Test data

Test data used complies with Open ROADM YANG model - a real-world model for optical devices. Specifically, openroadm-device nodes, each consisting of 86 fragments, are created. For example, a test that creates 1000 device nodes will result in 86,000 fragments in the database. Some tests use up to 3000 device nodes (258,000 fragments - equivalent to around 20,000 CM-handles in NCMP), with four anchors replicating the data, meaning that the system has been tested up to 1 million fragments.

Storing data nodes

A varying number of Open ROADM device nodes will be stored using CpsDataService::saveData.

...

Time (seconds)

...

Image Removed

Observations

• Storing data nodes has linear time complexity (as expected).
• Raw performance is roughly 3000 fragments per second for the given test setup.
• Performance can be improved by enabling write batching (CPS-1795)
• There are edge cases with exponential complexity (adding books to the bookstore model).

Commentary

The current database schema does not allow for Hibernate to use JDBC write batching. There is work in progress to address this.

JPA/Hibernate is not well-suited to tree-structured data. As an example, writing data nodes / fragments happens in two low-level steps:

• SQL INSERT to create the fragments
• SQL UPDATE to set the parent IDs of those created fragments
• meaning each created fragment requires two DB operations

There is work in progress to address this behavior using a custom algorithm using JDBC. The use of a graph database would implicitly fix such issues.

Updating data nodes

In this scenario, 1000 Open ROADM device nodes are already defined. A number of these existing data nodes will be updated using CpsDataService::updateDataNodeAndDescendants.

...

Time (seconds)

...

69.46

...

Image Removed

Observations

• Updating data nodes has linear time complexity (as expected).
• Raw performance is roughly 600 fragments per second for the given model and test setup.
• Updating data nodes is 5 times slower than storing data nodes.

Commentary

This is the weak spot of all write operations. It could be as fast to simply delete and recreate the data.

A custom algorithm comparing existing data against updated data (delta algorithm) may be required to improve performance here. This could also indicate that Hibernate is not being used effectively.

Updating data leaves

In this scenario, 1000 Open ROADM device nodes are already defined. The data leaves of a number of these existing data nodes will be updated using CpsDataService::updateNodeLeaves.

Example JSON payload for updating data leaves for one device:

...

...

Hardware	Specifications
CPU	13th Gen Intel© Core™ i9-13900H (24 MB cache, 14 cores, up to 5.40 GHz turbo mode)
Memory	32 GB RAM (16 GB x 2, DDR5, 4800 MHz)
Storage	1 TB M.2 PCIe NVMe SSD
Operating System	Linux Mint 21.2 Cinnamon (based on Ubuntu 22.04)

Test setup

The performance tests are written in Groovy (a JVM language). The tests were executed from Intelli-J using the Amazon Corretto 17 JDK.

As all CPS Core operations are synchronous, the results here are to be considered as single-threaded performance only.

Test data

Test data uses the Open ROADM YANG model - a real-world model for optical devices. Specifically, openroadm-device nodes, each consisting of 86 fragments, are created. For example, a test that creates 1,000 device nodes will result in 86,000 fragments in the database. Some tests use up to 3,000 device nodes (258,000 fragments - equivalent to around 20,000 CM-handles in NCMP), with four anchors replicating the data, meaning that the system has been tested up to 1 million fragments.

Storing data nodes

A varying number of Open ROADM device nodes will be stored using CpsDataService::saveData.

Graph of time taken to store device nodes

Image Added

Observations

• Storing data nodes has linear time complexity (as expected).
• Raw performance is roughly 3,500 fragments per second for the given test setup.
• Performance can be improved by enabling write batching (CPS-1795)
• There are edge cases with exponential complexity (adding books to the bookstore model).

Commentary

The current database schema does not allow for Hibernate to use JDBC write batching. There is worke in progress to address this.

JPA/Hibernate is not well-suited to tree-structured data. As an example, writing data nodes / fragments happens in two low-level steps:

• SQL INSERT to create the fragments
• SQL UPDATE to set the parent IDs of those created fragments
• meaning each created fragment requires two DB operations

There is work in progress to address this behavior using a custom algorithm using JDBC. The use of a graph database would implicitly fix such issues.

See here: https://vladmihalcea.com/the-best-way-to-map-a-onetomany-association-with-jpa-and-hibernate/ for explanation on why this occurs. Basically, unidirectional mapping OneToMany suffers from poor performance due to the order in which Hibernate writes out the entities. Switching to either unidirectional ManyToOne, or bidirectional OneToMany and ManyToOne could solve the issue using pure Hibernate, but they come with drawbacks. (To Do: More details on this.)

Updating data nodes

NOTE: I have removed the previous test results here, as the test analysis was flawed. Analysis is ongoing as per CPS-1674.

Performance tests to support new analysis are provisionally available at: https://gerrit.nordix.org/c/onap/cps/+/19086

Updating data leaves

In this scenario, 1,000 OpenROADM device nodes are already defined. The data leaves of a number of these existing data nodes will be updated using CpsDataService::updateNodeLeaves.

Example JSON payload for updating data leaves for one device:

{
    'openroadm-device': [
        {'device-id':'C201-7-1A-1', 'status':'fail', 'ne-state':'jeopardy'}
    ]
}

Test Results

...

Observations

• Updating data leaves has linear time complexity.
• Raw performance is about 3000 fragments per second.
• This is very fast compared to updating whole data nodes, and should be preferred where possible.

Commentary

The performance of this function is excellent, and yet it may be improved by write batching.

 I recommend that NCMP use this API for updating CM-handle state.

Deleting data nodes

In this scenario, 300 Open ROADM device nodes are already defined. A number of these data nodes will be deleted using CpsDataService::deleteDataNodes. The types of nodes will be varied, for example, deleting container nodes, list elements, or whole lists.

Test results

Graph of updating data leaves of device nodes

Image Added

Observations

• Updating data leaves has linear time complexity.
• Raw performance is around 1,500 fragments per second.
• This is very fast compared to updating whole data nodes, and should be preferred where possible.

Commentary

The performance of this function is excellent, and yet it may be improved by write batching.

 I recommend that NCMP use this API for updating CM-handle state.

Deleting data nodes

In this scenario, 300 OpenROADM device nodes are already defined. A number of these data nodes will be deleted using CpsDataService::deleteDataNodes. The types of nodes will be varied, for example, deleting container nodes, list elements, or whole lists.

Test results

...

Observations

• Delete performance is linear on the amount of data being deleted (as expected).
• Raw performance of deleting container nodes is around 40,000 fragments per second. (So we can delete data nodes more than 10x faster than creating them.)
• Deleting lists is much slower than deleting the parent container of the list (this can be improved).
• Of note, attempting to delete non-existing data nodes takes longer than actually deleting the equivalent amount of nodes with descendants - it is a slow operation.

Observations

• Delete performance is linear on the amount of data being deleted (as expected).
• Raw performance of deleting container nodes is around 35,000 fragments per second. (So we can delete data nodes around 10x faster than creating them.)
• Deleting lists is much slower than deleting the parent container of the list (this can be improved).
• Of note, attempting to delete non-existing data nodes takes longer than actually deleting the equivalent amount of nodes with descendants - it is a slow operation.

Suggested improvement: For whole list deletion, add a condition to the WHERE clause in the SQL for deleting lists, to narrow the search space to children of the parent. For example:

SELECTDELETE * FROM fragment WHERE (existing conditions)

  AND parent_id = (SELECT id FROM fragment WHERE xpath = '/parent-xpath')

...

Reading the top-level container node with no descendants:

The above data clearly indicates constant time.

Reading the top-level container node with all descendants:

Image Removed

Observations

...

e

Graph of time taken to read top-level container node with all descendants

Image Added

Observations

• Reading a single top-level container node with no descendants has constant time (as expected).
• Reading a single top-level container node with all descendants has linear time (as expected).
• Raw performance of reading with all descendants is roughly 100,000 fragments per second.

...

Test results

Special case: attempting to read multiple non-existing data nodes

In this case, we attempt to read many non-existing data nodes:

...

The above case appears to be constant time, but theoretically must be linear - we'll call it negligible time.

Observations

• Reading many data nodes with all descendants has linear time (as expected).
• Attempting to read many non-existing data nodes takes negligible time.
• Raw performance of reading many with all descendants is roughly 80,000 fragments per second.

Additional test cases: Reading container node versus list

Recently, functionality was added to enable reading whole lists (CPS-1696). Here we compare performance of reading a container node containing a list, versus reading the list (with all descendants).

...

As can be seen, it is slower reading the list than reading the parent node containing the list.

Suggested improvement: Add a condition to the WHERE clause in the SQL for reading lists, to narrow the search space to children of the parent. For example:

SELECT * FROM fragment WHERE (existing conditions)

  AND parent_id = (SELECT id FROM fragment WHERE xpath = '/parent-xpath')

This should narrow the performance gap in this case.

Cps Path Queries

There have been massive orders-of-magnitude improvements to query performance. Formerly, queries had quadratic complexity, which is now linear. In the last month alone, performance has improved by 100x.

TO DO Fill this section. See here for a previous PoC analysis: CPS-1646 CpsPath queries using path-component lookup

Observations

• Reading many data nodes with all descendants has linear time (as expected).
• Raw performance of reading many with all descendants is roughly 80,000 fragments per second.

Additional test cases: Reading container node versus list

Recently, functionality was added to enable reading whole lists (CPS-1696). Here we compare performance of reading a container node containing a list, versus reading the list (with all descendants).

As can be seen, it is slower reading the list than reading the parent node containing the list.

Suggested improvement: Add a condition to the WHERE clause in the SQL for reading lists, to narrow the search space to children of the parent. For example:

SELECT * FROM fragment WHERE (existing conditions)

  AND parent_id = (SELECT id FROM fragment WHERE xpath = '/parent-xpath')

This should narrow the performance gap in this case.

Cps Path Queries

There have been massive orders-of-magnitude improvements to query performance. Formerly, queries had quadratic complexity, which is now linear. In the last month alone, performance has improved by 100x.

With a simple improvement, some cases can be made constant time instead of linear.

In the graphs below, time is measured in milliseconds.

Query one device out of many using absolute cps path

In this case, a query that matches a single device node using an absolute cps path is executed.

Query one device out of many using descendant cps path

In this case, a query that matches a single device node using a descendant cps path is executed.

Query all devices using absolute cps path

In this case, a query that matches all device nodes using an absolute cps path is executed.

Query all devices using descendant cps path

In this case, a query that matches all device nodes using a descendant cps path is executed.

Commentary

Suggested improvement: For absolute cps path queries, the complexity can be reduced to best-case constant time (so it is no longer linear on the total number of fragments in the database).

Add a condition to the WHERE clause in the SQL to narrow the search space to children of the parent. For example:

SELECT * FROM fragment WHERE (existing conditions)

  AND parent_id IN (SELECT id FROM fragment WHERE xpath = '/parent-xpath')

(Note we used "parent_id IN" as there may be multiple parents with the same xpath during query across anchors.

Comparison of NCMP Performance across versions

CM-handle registration

...

CM-handle de-registration

Current release has exactly linear performance for CM-handle de-registration (on a single thread):

Commentary

CM-handle de-registration is a synchronous operation. Since the above tests send a single large batch, the operation is performed in a single thread. The linear performance reflects this. If the operation was broken into many smaller requests, the expectation is that it would complete many times faster - but this has not been tested.