Summary

This is an assessment of current performance of CPS operations, as of August 2023.

...

Image Removed

Observations

Graph of time taken to store device nodes

Image Added

Observations

• Storing data nodes has linear time complexity (as expected).
• Raw performance is roughly 3000 3,500 fragments per second for the given test setup.
• Performance can be improved by enabling write batching (CPS-1795)
• There are edge cases with exponential complexity (adding books to the bookstore model).

...

In this scenario, 1,000 Open ROADM device nodes are already defined. A number of these existing data nodes will be updated using CpsDataService::updateDataNodeAndDescendants.

...

Observations

...

Graph of time taken to update device nodes and descendants

Image Added

Observations

• Updating data nodes has linear time complexity (as expected).
• Raw performance is roughly 600 fragments per second for the given model and test setup.
• Updating data nodes is 5 times slower than storing data nodes.

...

This is the weak spot of all write operations. A custom algorithm comparing existing data against updated data (delta algorithm) may be required to improve performance here. This could also indicate that Hibernate is not being used effectively.

TODO mention json alphabetical sorting

Updating data leaves

In this scenario, 1,000 Open ROADM device nodes are already defined. The data leaves of a number of these There are a number of issues possibly affecting performance here. For example, after DataNodes are converted to FragmentEntity objects, they are being persisted using the Hibernate persist method, which will always write the changes, regardless of whether the data changes. There is an alternative method called merge which compares existing data to updated data. Even if this were implemented, there are additional issues, such as JSON encoding giving inconsistent ordering. For example, data leaves could be encoded as "{'x':'1', 'y':'2'}" or "{'y':'2','x':'1'}", depending on exact object type for storing leaves (HashMap, LinkedHashMap, ImmutableMap, etc.). There is an option for JsonObjectMapper to order the keys alphabetically during encoding.

Updating data leaves

In this scenario, 1,000 Open ROADM device nodes are already defined. The data leaves of a number of these existing data nodes will be updated using CpsDataService::updateNodeLeaves.

...

{
    'openroadm-device': [
        {'device-id':'C201-7-1A-1', 'status':'fail', 'ne-state':'jeopardy'}
    ]
}

Test Results

...

}

Test Results

Graph of updating data leaves of device nodes

Observations

• Updating data leaves has linear time complexity.
• Raw performance is about 3000 around 1,500 fragments per second.
• This is very fast compared to updating whole data nodes, and should be preferred where possible.

...

In this scenario, 300 Open ROADM device nodes are already defined. A number of these data nodes will be deleted using CpsDataService::deleteDataNodes. The types of nodes will be varied, for example, deleting container nodes, list elements, or whole lists.

Test results

...

• Delete performance is linear on the amount of data being deleted (as expected).
• Raw performance of deleting container nodes is around 4035,000 fragments per second. (So we can delete data nodes more than 10x faster than creating them.)
• Deleting lists is much slower than deleting the parent container of the list (this can be improved).
• Of note, attempting to delete non-existing data nodes takes longer than actually deleting the equivalent amount of nodes with descendants - it is a slow operation.

...

Reading the top-level container node with no descendants:

...

node with no descendants:

The above data clearly indicates constant time.

Reading the top-level container node with all descendants:

Graph of time taken to read top-level container node with all descendants

Observations

• Reading a single top-level container node with no descendants has constant time (as expected).
• Reading a single top-level container node with all descendants has linear time (as expected).
• Raw performance of reading with all descendants is roughly 100,000 fragments per second.

...

Test results

Special case: attempting to read multiple non-existing data nodes

In this case, we attempt to read many non-existing data nodes:

The above case appears to be constant time, but theoretically must be linear - we'll call it negligible time.

...

Recently, functionality was added to enable reading whole lists (CPS-1696). Here we compare performance of reading a container node containing a list, versus reading the list (with all descendants).

...

CM-handle registration

...

CM-handle de-registration

Current release has exactly linear performance for CM-handle de-registration (on a single thread):

Commentary

CM-handle de-registration is a synchronous operation. Since the above tests send a single large batch, the operation is performed in a single thread. The linear performance reflects this. If the operation was broken into many smaller requests, the expectation is that it would complete many times faster - but this has not been tested.