CPS Core Performance

Test Environment

TODO: Laptop specs

The performance tests are written in Groovy (a JVM language). As all CPS Core operations are synchronous, the results here are to be considered as single-threaded operation only.

Test data

Test data used complies with Open ROADM YANG model. Specifically, openroadm-device nodes consisting of 86 fragments are created. For example, a test that creates 1000 device nodes will result in 86,000 fragments in the database. Some tests use up to 3000 device nodes (258,000 fragments - equivalent to around 20,000 CM-handles in NCMP), with four anchors replicating the data, meaning that the system has been tested up to 1 million fragments. Not all results are displayed on this page, but are included in the attached spreadsheet.

Storing data nodes

A varying number of Open ROADM device nodes will be stored using CpsDataService::saveData.

Observations

• Storing data nodes has linear time complexity (as expected).
• Raw performance is roughly 3000 fragments per second for the given test setup.
• Performance can be improved by enabling write batching (CPS-1795)
• There are edge cases with exponential complexity.

Updating data nodes

In this scenario, 1000 Open ROADM device nodes are already defined. A number of these existing data nodes will be updated using CpsDataService::updateDataNodeAndDescendants.

Observations

• Updating data nodes has linear time complexity (as expected).
• Raw performance is roughly 600 fragments per second for the given model and test setup.
• Updating data nodes is 5 times slower than storing data nodes.

Updating data leaves

In this scenario, 1000 Open ROADM device nodes are already defined. The data leaves of a number of these existing data nodes will be updated using CpsDataService::updateNodeLeaves.

Example JSON payload for updating data leaves for one device:

{
    'openroadm-device': [
        {'device-id':'C201-7-1A-1', 'status':'fail', 'ne-state':'jeopardy'}
    ]
}

Test Results

Observations

• Updating data leaves has linear time complexity.
• Raw performance is about 3000 fragments per second.
• This is very fast compared to updating whole data nodes. I recommend that NCMP use this API for updating CM-handle state.

Deleting data nodes

In this scenario, 300 Open ROADM device nodes are already defined. A number of these data nodes will be deleted using CpsDataService::deleteDataNodes. The types of nodes will be varied, for example, deleting container nodes, list elements, or whole lists.

Test results

Observations

• Delete performance is linear on the amount of data being deleted (as expected).
• Raw performance of deleting containers of list elements is around 40,000 fragments per second. (So we can delete data nodes around 10x faster than creating them.)
• Deleting lists is much slower than deleting the parent container of the list (which can be easily improved).
• Of note, attempting to delete non-existing data nodes takes longer than actually deleting the equivalent amount of nodes with descendants - it is a slow operation.

Suggested improvement: For whole list deletion, add a condition to the WHERE clause in the SQL for deleting lists, to narrow the search space to children of the parent. For example:

SELECT * FROM fragment WHERE (existing conditions)

  AND parent_id = (SELECT id FROM fragment WHERE xpath = '/parent-xpath')

This should narrow the performance gap in this case.

Reading data nodes

In these tests, a varying number of Open ROADM devices are created and retrieved.

Reading top-level container node

In this test, CpsDataService::getDataNodes is used to retrieve the top-level container node.

Test results

Reading the top-level container node with no descendants:

The above data clearly indicates constant time.

Reading the top-level container node with all descendants:

Observations

• Reading a single top-level container node with no descendants has constant time (as expected).
• Reading a single top-level container node with all descendants has linear time (as expected).
• Raw performance of reading with all descendants is roughly 100,000 fragments per second.

Reading data nodes for multiple xpaths

This test uses CpsDataService::getDataNodesForMultipleXpaths with all descendants to retrieve a varying number of Open ROADM device nodes.

Test results

Special case: attempting to read multiple non-existing data nodes

In this case, we attempt to read many non-existing data nodes:

The above case appears to be constant time, but theoretically must be linear - we'll call it negligible time.

Observations

• Reading many data nodes with all descendants has linear time (as expected).
• Attempting to read many non-existing data nodes takes negligible time.
• Raw performance of reading many with all descendants is roughly 80,000 fragments per second.

Additional test cases: Reading container node versus list

Recently, functionality was added to enable reading whole lists (CPS-1696). Here we compare performance of reading a container node containing a list, versus reading the list (with all descendants).

As can be seen, it is slower reading the list than reading the parent node containing the list.

Suggested improvement: Add a condition to the WHERE clause in the SQL for reading lists, to narrow the search space to children of the parent. For example:

SELECT * FROM fragment WHERE (existing conditions)

  AND parent_id = (SELECT id FROM fragment WHERE xpath = '/parent-xpath')

This should narrow the performance gap in this case.

Cps Path Queries

TODO

Comparison of NCMP Performance across versions

CM-handle registration

CM-handle registration is multi-threaded, so performance may appear to scale better than linear, until the CPU cores are maxed out.

As can be seen below, CPU usage never reached 100% during the tests of the current version.

CM-handle deregistration

Current release has exactly linear performance for CM-handle de-registration (on a single thread):