Page Comparison

...

The current database schema does not allow for Hibernate to use JDBC write batching. There is work worke in progress to address this.

...

There is work in progress to address this behavior using a custom algorithm using JDBC. The use of a graph database would implicitly fix such issues.

Updating data nodes

(NOTE: I have removed the previous test results here, as the testing was flawed).

Analysis is still ongoing, but here is an overview of some of the findings:

• For making CM-handle state updates, NCMP uses CpsDataService::updateDataNodesAndDescendants (plural - taking a Map<xpath, json>). The performance of this function is poor. Because it takes a map of JSON, it needs to instantiate new YangParser for piece of JSON. Better performance would be achieved by using CpsDataService::updateDataNodeAndDescendants (singular - taking a single piece of JSON). Note this API could be removed after, since it is only used by NCMP, and was never exposed as part of the public REST API.
• Related: CpsDataService::saveListElementsBatch likewise has very poor performance compared with CpsDataService::saveListsElements, which can achieve the same effect by correctly preparing the input JSON. The poor performance stems from needing a new YangParser instance for each chunk of JSON instead of a single YangParser for one JSON string. This API is not publicly exposed via the REST API and is only used in one place in NCMP, so it could be removed.
• The main public API for updating data nodes is CpsDataService::updateDataNodeAndDescendants (singular). This API has very high variations in performance, depending on how the JSON is prepared, for example, given an update that effects exactly the same data nodes:
  • updating a top-level node with parent path of '/' is much slower than updating the descendants. See table below.
  • As such, it is now advised to update data trees at the deepest possible level.

Comparison of updating using different approaches

There are multiple was of updating data nodes, each with different performance.

In these scenario, 1,000 Open ROADM device nodes are already defined. A number of these existing data nodes will be updated using CpsDataService::updateDataNodeAndDescendants (singular).

...

Updating data leaves

In this scenario, 1,000 Open ROADM device nodes are already defined. The data leaves of a number of these existing data nodes will be updated using CpsDataService::updateNodeLeaves.

Example JSON payload for updating data leaves for one device:

{
    'openroadm-device': [
        {'device-id':'C201-7-1A-1', 'status':'fail', 'ne-state':'jeopardy'}
    ]
}

Test Results

...

Time (seconds)

...

...

Graph of updating data leaves of device nodes

Image Removed

Observations

• Updating data leaves has linear time complexity.
• Raw performance is around 1,500 fragments per second.
• This is very fast compared to updating whole data nodes, and should be preferred where possible.

Commentary

The performance of this function is excellent, and yet it may be improved by write batching.

 I recommend that NCMP use this API for updating CM-handle state.

Deleting data nodes

In this scenario, 300 Open ROADM device nodes are already defined. A number of these data nodes will be deleted using CpsDataService::deleteDataNodes. The types of nodes will be varied, for example, deleting container nodes, list elements, or whole lists.

Test results

...

See here: https://vladmihalcea.com/the-best-way-to-map-a-onetomany-association-with-jpa-and-hibernate/ for explanation on why this occurs. Basically, unidirectional mapping OneToMany suffers from poor performance due to the order in which Hibernate writes out the entities. Switching to either unidirectional ManyToOne, or bidirectional OneToMany and ManyToOne could solve the issue using pure Hibernate, but they come with drawbacks. (To Do: More details on this.)

Updating data nodes

NOTE: I have removed the previous test results here, as the test analysis was flawed. Analysis is ongoing as per CPS-1674.

Performance tests to support new analysis are provisionally available at: https://gerrit.nordix.org/c/onap/cps/+/19086

Updating data leaves

In this scenario, 1,000 OpenROADM device nodes are already defined. The data leaves of a number of these existing data nodes will be updated using CpsDataService::updateNodeLeaves.

Example JSON payload for updating data leaves for one device:

{
    'openroadm-device': [
        {'device-id':'C201-7-1A-1', 'status':'fail', 'ne-state':'jeopardy'}
    ]
}

Test Results

Graph of updating data leaves of device nodes

Image Added

Observations

• Updating data leaves has linear time complexity.
• Raw performance is around 1,500 fragments per second.
• This is very fast compared to updating whole data nodes, and should be preferred where possible.

Commentary

The performance of this function is excellent, and yet it may be improved by write batching.

 I recommend that NCMP use this API for updating CM-handle state.

Deleting data nodes

In this scenario, 300 OpenROADM device nodes are already defined. A number of these data nodes will be deleted using CpsDataService::deleteDataNodes. The types of nodes will be varied, for example, deleting container nodes, list elements, or whole lists.

Test results

Observations

• Delete performance is linear on the amount of data being deleted (as expected).
• Raw performance of deleting container nodes is around 35,000 fragments per second. (So we can delete data nodes more than 10x faster than creating them.)
• Deleting lists is much slower than deleting the parent container of the list (this can be improved).
• Of note, attempting to delete non-existing data nodes takes longer than actually deleting the equivalent amount of nodes with descendants - it is a slow operation.

Suggested improvement: For whole list deletion, add a condition to the WHERE clause in the SQL for deleting lists, to narrow the search space to children of the parent. For example:

DELETE FROM fragment WHERE (existing conditions)

  AND parent_id = (SELECT id FROM fragment WHERE xpath = '/parent-xpath')

This should narrow the performance gap in this case.

Reading data nodes

In these tests, a varying number of Open ROADM devices are created and retrieved.

Reading top-level container node

In this test, CpsDataService::getDataNodes is used to retrieve the top-level container node.

Test results

Reading the top-level container node with no descendants:

...

Observations

• Delete performance is linear on the amount of data being deleted (as expected).
• Raw performance of deleting container nodes is around 35,000 fragments per second. (So we can delete data nodes around 10x faster than creating them.)
• Deleting lists is much slower than deleting the parent container of the list (this can be improved).
• Of note, attempting to delete non-existing data nodes takes longer than actually deleting the equivalent amount of nodes with descendants - it is a slow operation.

Suggested improvement: For whole list deletion, add a condition to the WHERE clause in the SQL for deleting lists, to narrow the search space to children of the parent. For example:

DELETE FROM fragment WHERE (existing conditions)

  AND parent_id = (SELECT id FROM fragment WHERE xpath = '/parent-xpath')

This should narrow the performance gap in this case.

Reading data nodes

In these tests, a varying number of Open ROADM devices are created and retrieved.

Reading top-level container node

In this test, CpsDataService::getDataNodes is used to retrieve the top-level container node.

Test results

Reading the top-level container node with all no descendants:

Graph of time taken to read top-level container node with all descendants

Image Removed

Observations

• Reading a single top-level container node with no descendants has constant time (as expected).
• Reading a single top-level container node with all descendants has linear time (as expected).
• Raw performance of reading with all descendants is roughly 100,000 fragments per second.

Commentary

This is the fastest operation in CPS, in terms of fragments per second. This is not surprising, as a simple read should be the fastest operation.

Reading data nodes for multiple xpaths

This test uses CpsDataService::getDataNodesForMultipleXpaths with all descendants to retrieve a varying number of Open ROADM device nodes.

Test results

...

Image Removed

Special case: attempting to read multiple non-existing data nodes

In this case, we attempt to read many non-existing data nodes:

The above data clearly indicates constant time.

Reading the top-level container node with all descendants:

e

Graph of time taken to read top-level container node with all descendants

Image Added

Observations

• Reading a single top-level container node with no descendants has constant time (as expected).
• Reading a single top-level container node with all descendants has linear time (as expected).
• Raw performance of reading with all descendants is roughly 100,000 fragments per second.

Commentary

This is the fastest operation in CPS, in terms of fragments per second. This is not surprising, as a simple read should be the fastest operation.

Reading data nodes for multiple xpaths

This test uses CpsDataService::getDataNodesForMultipleXpaths with all descendants to retrieve a varying number of Open ROADM device nodes.

Test results

...

Image Added

Observations

• Reading many data nodes with all descendants has linear time (as expected).Attempting to read many non-existing data nodes takes negligible time.
• Raw performance of reading many with all descendants is roughly 80,000 fragments per second.

...