1 References
2 Test Steps & Test Data
3 Test Environment
4 Test Results for only Registration & Deregistration APIs
5 Test Results for all NCMP APIs
- 5.1 Observations
- 5.2 The Best Performing Parameters

References

CPS-2626: Optimize registration parametersIn Progress

Test Steps & Test Data

Create CM Handles
Running these tests for 15 min:
1. Passthrough Read & Write with CMHandle and AlternateID
2. CM Handle & CM Handle ID Search with filters
  1. Module
  2. Property
  3. Cps Path
  4. Trust Level
3. CM Handle & CM Handle ID Search without filters
4. Legacy Batch Read Operation
Delete CM Handles

Test Environment

Test Environment

Test Environment
Nordix Server	xerces-cps-hw [Jenkins]
CPUs	Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
Operating System	Ubuntu 22.04.5 LTS (GNU/Linux 5.15.0-126-generic x86_64)
Memory	251.0 GiB
Commit	140173: Fix for intermittent failures of module sync tests
Job	onap-cps-performance-test-k6 [Jenkins]
Docker Compose Deployment	docker-compose version `v2.29.2` application (CPS) instance - 2 or 3 depending test postgres instance - 1 #WorkerThread - 1 ( The AsyncTaskExecutor and the thread pool removed for ModuleSyncTask. Thus, there is only one thread running at a time )

Test Results for only Registration & Deregistration APIs

	input batch size	watchdog timer	#CPS instances CPU and Memory	avg cm-handles/second : registration	avg cm-handles/second : deregistration	comments

	input batch size	watchdog timer	#CPS instances CPU and Memory	avg cm-handles/second : registration	avg cm-handles/second : deregistration	comments
1	100	5 sec	2 instances 3 CPU 3G memory	88	120	Average of build numbers: 5929, 5930, 5931
2	500	5 sec		86	191	Average of build numbers: 5932, 5933
3	2,000	5 sec		90	211	Average of build numbers randomly picked from the range: 5934-5951
4	3,000	5 sec		87	200	Build numbers: 5955, 5956, 5957 All CM-handle and CM-handle ID searches failed
5	5,000	5 sec		0	0	Build numbers: 5952, 5953, 5954 All the builds stop working after 20 min. Reminder: Setup timeout is 20 min.
6	100	60 sec		65	130	The build number: 5958, 5959
7	100	10 sec		84	124	Average of build numbers randomly picked from the range: 5960-5976
8	100	1 sec		86	120	Average of build numbers: 5977, 5978, 5979
9	100	5 sec	3 CPS instances with 3 CPU 3G memory	80	136	Build: 5983 failed because of an intermittent infra issue. Average of build numbers: 5984, 5985
10	1,000	5 sec		95	191	Average of build numbers: 5980, 5981, 582
11	2,000	5 sec		94	199	Average of build numbers: 6002, 6003, 6005 6004 failed.
12	1,000	5 sec	3 CPS instances 2 CPU 2G memory	86	188	Average of build numbers: 6026, 6027, 6028
13	2,000	5 sec	3 CPS instances 2 CPU 2G memory	102	190	Average of build numbers: 6006, 6007, 6008

Test Results for all NCMP APIs

The commit used in this test:

Commit: 140281: retry before stopping in docker-compose | https://gerrit.onap.org/r/c/cps/+/140281

I attached the spreadsheet with details including particular test cases with results in this jira:

CPS-2626: Optimize registration parametersIn Progress

	Master Branch 2 CPS instances 3 CPU 3G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 1000 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 2000 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 3000 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 2000 Internal Batch Size 300

	Master Branch 2 CPS instances 3 CPU 3G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 1000 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 2000 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 3000 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 2000 Internal Batch Size 300
Registration of CM-handles (CM-handles/second)	87	91	90	104	106	114
De-registration of CM-handles (CM-handles/second)	120	126	206	197	208	212
CM-handle ID search with No filter (milliseconds)	969	1129	1130	1169	0	1160
CM-handle ID search with Module filter (milliseconds)	236	248	251	249	0	253
CM-handle ID search with Property filter (milliseconds)	2115	2519	2496	2470	0	2560
CM-handle ID search with Cps Path filter (milliseconds)	2171	2550	2552	2513	0	2569
CM-handle ID search with Trust Level filter (milliseconds)	10824	24555	25289	24570	0	24830
CM-handle search with No filter (milliseconds)	7993	10123	10506	10191	0	10101
CM-handle search with Module filter (milliseconds)	10011	12621	12509	12764	0	12598
CM-handle search with Property filter (milliseconds)	11686	15156	14970	15207	0	15311
CM-handle search with Cps Path filter (milliseconds)	12327	15240	15327	15524	0	15613
CM-handle search with Trust Level filter (milliseconds)	21181	36201	36951	36145	0	37137
NCMP overhead for Synchronous single CM-handle pass-through read (milliseconds)	34	37	39	37	40	39
NCMP overhead for Synchronous single CM-handle pass-through read with alternate id	88	100	100	102	102	99
NCMP overhead for Synchronous single CM-handle pass-through write (milliseconds)	40	49	49	49	48	50
NCMP overhead for Synchronous single CM-handle pass-through write with alternate id	89	99	104	103	105	101
Legacy batch read operation (events/second)	120	118	120	117	115	116

Observations

Two test scenarios have better test results than others:

3 CPS instances
- 2 CPU
- 2G memory
Input Batch Size
- 2000

and

3 CPS instances
- 2 CPU
- 2G memory
Input Batch Size
- 2000
Internal Batch Size
- 300

Please find them compared to the default case (master branch) on the tables below:

	Master Branch 2 CPS instances 3 CPU 3G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 2000 Internal Batch Size 100	Gain / Loss in Percentage

	Master Branch 2 CPS instances 3 CPU 3G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 2000 Internal Batch Size 100	Gain / Loss in Percentage
Registration of CM-handles (CM-handles/second)	87	104	20
De-registration of CM-handles (CM-handles/second)	120	197	64
CM-handle ID search with No filter (milliseconds)	969	1169	-17
CM-handle ID search with Module filter (milliseconds)	236	249	-5
CM-handle ID search with Property filter (milliseconds)	2115	2470	-14
CM-handle ID search with Cps Path filter (milliseconds)	2171	2513	-14
CM-handle ID search with Trust Level filter (milliseconds)	10824	24570	-56
CM-handle search with No filter (milliseconds)	7993	10191	-22
CM-handle search with Module filter (milliseconds)	10011	12764	-22
CM-handle search with Property filter (milliseconds)	11686	15207	-23
CM-handle search with Cps Path filter (milliseconds)	12327	15524	-21
CM-handle search with Trust Level filter (milliseconds)	21181	36145	-41
NCMP overhead for Synchronous single CM-handle pass-through read (milliseconds)	34	37	-8
NCMP overhead for Synchronous single CM-handle pass-through read with alternate id	88	102	-14
NCMP overhead for Synchronous single CM-handle pass-through write (milliseconds)	40	49	-18
NCMP overhead for Synchronous single CM-handle pass-through write with alternate id	89	103	-14
Legacy batch read operation (events/second)	120	117	-3

There is a loss of more than %40 in CM-handle searches with the Trust Level filter
- The reason for this is the reduction in memory (1GB). Since the Trust Level feature is being handled in embedded Hazelcast caches.

	Master Branch 2 CPS instances 3 CPU 3G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 2000 Internal Batch Size 300	Gain / Loss in Percentage

	Master Branch 2 CPS instances 3 CPU 3G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 2 CPU 2G memory Input Batch Size 2000 Internal Batch Size 300	Gain / Loss in Percentage
Registration of CM-handles (CM-handles/second)	87	114	31
De-registration of CM-handles (CM-handles/second)	120	212	77
CM-handle ID search with No filter (milliseconds)	969	1160	-16
CM-handle ID search with Module filter (milliseconds)	236	253	-7
CM-handle ID search with Property filter (milliseconds)	2115	2560	-17
CM-handle ID search with Cps Path filter (milliseconds)	2171	2569	-15
CM-handle ID search with Trust Level filter (milliseconds)	10824	24830	-56
CM-handle search with No filter (milliseconds)	7993	10101	-21
CM-handle search with Module filter (milliseconds)	10011	12598	-21
CM-handle search with Property filter (milliseconds)	11686	15311	-24
CM-handle search with Cps Path filter (milliseconds)	12327	15613	-21
CM-handle search with Trust Level filter (milliseconds)	21181	37137	-43
NCMP overhead for Synchronous single CM-handle pass-through read (milliseconds)	34	39	-13
NCMP overhead for Synchronous single CM-handle pass-through read with alternate id	88	99	-11
NCMP overhead for Synchronous single CM-handle pass-through write (milliseconds)	40	50	-20
NCMP overhead for Synchronous single CM-handle pass-through write with alternate id	89	101	-12
Legacy batch read operation (events/second)	120	116	-3

The change made for internal batch size, has increased the registration by %31, and deregistration by %77. However, a degradation was observed in CM-handle searches with the Trust Level filter
with more than -%40.

The Best Performing Parameters

	Master Branch 2 CPS instances 3 CPU 3G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 3 CPU 3G memory Input Batch Size 2000 Internal Batch Size 300	Gain / Loss in Percentage

	Master Branch 2 CPS instances 3 CPU 3G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 3 CPU 3G memory Input Batch Size 2000 Internal Batch Size 300	Gain / Loss in Percentage
Registration of CM-handles (CM-handles/second)	87	118	36
De-registration of CM-handles (CM-handles/second)	120	215	79
CM-handle ID search with No filter (milliseconds)	969	838	16
CM-handle ID search with Module filter (milliseconds)	236	220	7
CM-handle ID search with Property filter (milliseconds)	2115	1875	13
CM-handle ID search with Cps Path filter (milliseconds)	2171	1917	13
CM-handle ID search with Trust Level filter (milliseconds)	10824	11185	-3
CM-handle search with No filter (milliseconds)	7993	7025	14
CM-handle search with Module filter (milliseconds)	10011	8928	12
CM-handle search with Property filter (milliseconds)	11686	10600	10
CM-handle search with Cps Path filter (milliseconds)	12327	10628	16
CM-handle search with Trust Level filter (milliseconds)	21181	20412	4
NCMP overhead for Synchronous single CM-handle pass-through read (milliseconds)	34	21	62
NCMP overhead for Synchronous single CM-handle pass-through read with alternate id	88	70	26
NCMP overhead for Synchronous single CM-handle pass-through write (milliseconds)	40	25	60
NCMP overhead for Synchronous single CM-handle pass-through write with alternate id	89	69	29
Legacy batch read operation (events/second)	120	115	-4

On one hand, reducing the internal batch size to 100 from 300, lowered the registration performance by %16 and lowered the de-registration performance by %11

On the other hand, it proved that the internal batch size does not affect the performance of CM Handle Search APIs.

	Master Branch 2 CPS instances 3 CPU 3G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 3 CPU 3G memory Input Batch Size 2000 Internal Batch Size 100	Gain / Loss in Percentage

	Master Branch 2 CPS instances 3 CPU 3G memory Input Batch Size 100 Internal Batch Size 100	3 CPS instances 3 CPU 3G memory Input Batch Size 2000 Internal Batch Size 100	Gain / Loss in Percentage
Registration of CM-handles (CM-handles/second)	87	104	20
De-registration of CM-handles (CM-handles/second)	120	202	68
CM-handle ID search with No filter (milliseconds)	969	840	15
CM-handle ID search with Module filter (milliseconds)	236	223	6
CM-handle ID search with Property filter (milliseconds)	2115	1857	14
CM-handle ID search with Cps Path filter (milliseconds)	2171	1897	14
CM-handle ID search with Trust Level filter (milliseconds)	10824	11077	-2
CM-handle search with No filter (milliseconds)	7993	6994	14
CM-handle search with Module filter (milliseconds)	10011	8907	12
CM-handle search with Property filter (milliseconds)	11686	10565	11
CM-handle search with Cps Path filter (milliseconds)	12327	10636	16
CM-handle search with Trust Level filter (milliseconds)	21181	20275	4
NCMP overhead for Synchronous single CM-handle pass-through read (milliseconds)	34	21	62
NCMP overhead for Synchronous single CM-handle pass-through read with alternate id	88	70	26
NCMP overhead for Synchronous single CM-handle pass-through write (milliseconds)	40	27	48
NCMP overhead for Synchronous single CM-handle pass-through write with alternate id	89	69	29
Legacy batch read operation (events/second)	120	115	-4

ONAP Wiki

NCMP CM Handle Create & Delete Test Results

Analytics