CPS-2169: NCMP to support 200k Cells (50KCM-handles)

Review the use case returning all 50k cmHandle

Validity of the use case where ALL 50k CMhandles are expected to be returned every time ?

AP @Csaba Kocsis & @kieran mccarthy 

Jan 30, 2025

1. Returning all 50k cmhandles will impact the speed and overall performance. Focus on Read & Write

2. Review usecases that don’t need to return all 50k cmhandles. @kieran mccarthy

Proposal to remove hazelcast for NCMP Module sync

The use of Hazelcast during NCMP's CM-handle Module Sync is leading to:

1. High memory usage during CM-handle registration

2. Consistency problem

3. Poor load balancing between NCMP instances for module sync

Proposal to remove hazelcast from NCMP Module Sync requires an LCM State Machine Change. Need stakeholder input.

Proposal:  CPS-2161: Remove Hazelcast from NCMP Module Sync

Oct 30, 2024

• CPS optimised locking mechanism on the bug CPS-2403: CmHandle discovery degradationClosed

• Still open pending CPS completion of Hazelcast study - CPS-2161: Remove Hazelcast from NCMP Module SyncOpen

DDP Metrics Collection

CPS to investigate how DDP metrics can be collected to investigate client usecase of AP @Daniel Hanrahan

Jan 30, 2025

Registration & De-registration (Discovery) In Batch size of 100 per request.

• With moduleSetTag &

• With alternateID

Parallel

1. How long should it take cmhandle to reach in ready state - 11 Cm Handles/second per NCMP instance.

2. Overall budget; @Peter Turcsanyi
   a. Advise → locked → Ready: ?
   b. Advise → Ready: ?

3. Initial delay ; Currently 2 min to move to → 3min

Jan 30, 2025 @kieran mccarthy

CM-handle ID Search

5 Parallel

1. No frequency increase expected

2. Search shall return all 50k searches ?

3. Searches shall return all 50k searches within 1 minute max TBD

Aug 28, 2024 

1. 5 parallel request of ID search and search, = combined total of 10 parallel search requests.

Jan 30, 2025

@kieran mccarthy

CM-handle search

5 Parallel

1. No frequency increase expected

2. Search shall return all 50k searches ?  

3. Searches shall return all 50k searches within 1 minute max TBD

Aug 28, 2024 

1. 5 parallel request of ID search and search, Total of 10 parallel search requests.

Jan 30, 2025

No change TBD AP; @Peter Turcsanyi @kieran mccarthy

Synchronous single CM-handle pass-through Read (CUD)

4 parallel Operations

Review; the current FS numbers

1. What will be frequencies for write req. ? AP: TBD

2. Frequencies will increase (2.5); 10 req/sec= 25 req/sec

3. Response time should stay the same as current - NCMP Characteristics

Nov 19, 2024

1. Stick with 10 req/sec= 25 req/sec (Should CPS PoC it?) to measure the current no of req.

2. CPS measure current capacity

3. CPS will engage in Setting the frequency (Based on previous result)

4. No changes to Delay is expected

Jan 30, 2025

Policy executor have different test suit and won’t be impacted by this

@kieran mccarthy

Synchronous single CM-handle pass-through Write

4 (Parallel operations)

1. Frequencies will increase (2.5) currently 5 request/second for 80k cells = 12.5 req/sec for 200k cells

2. Response time should stay the same as current NCMP Characteristics (delay, size)

Jan 30, 2025 @kieran mccarthy

1. CPS measure current capacity

2. CPS will engage in Setting the frequency (Based on previous result)

3. Share with stakeholder

4. No changes to Delay is expected

5. No volume increase expected

Batch Read Operation/Legacy 

1. Frequencies will increase (2.5)

2. Same load 200 per request (Same response size)

3. Same duration of test, No changes to volume expected

4. Responses are expected back in 80 sec.

Investigation needed because target is not met at this point

Jan 30, 2025

CM change Notification Event

1. NCMP shall support a CM notification load of 150 million CM change notifications per day with an average of 870 notification

2. NCMP shall support a peak CM change notification load of 7k/s for a duration 5 minutes

1. 2,300  / sec (base/Average load)

2. 8.75 k/sec (peak load) TBC @kieran mccarthy

3. Daily Average

4. Base:

5. Average:

6. Peak: 

7. Storm:

8. Load distribution discussion still pending

Nov 19, 2024

Current numbers on FS as of today should be by 2.5 (non-negotiable).

Jan 30, 2025

Use number on FS and * 2.5 as the source of truth

Base Performance & Performance

Average Performance & Duration

Peak Performance & Duration

Number of Instances

TBC - Each NCMP instance takes 11 CM-Handles/second.