References:
...
Expectation/ goal: cm-handle registration + module sync → 25 cm handles / sec (see CPS -NCMP-I-01 Specification: Interface Performance Requirements Overview)
Improvements Overview
| Test Environment | Secs Taken for ModuleSyncWatchdog (Cm Handle moves ADVISED->READY) | Secs taken for Registration sub-process | Commit id | Parrealel Request to DMI | Main improvement | Comments | ||
|---|---|---|---|---|---|---|---|---|
Ericsson Testing cluster (with Model adapter that is implementation of dmi-plugin) Total 1458 cm Handles | 6,998 → 0.20/sec | 2,278 → 0.64/sec | 1410509e33142c0c79ff0e111c63abc47f5936d3 https://gerrit.onap.org/r/gitweb?p=cps.git;a=commit;h=1410509e33142c0c79ff0e111c63abc47f5936d3 | N/A (1) | N/A | Baseline as per the bug | ||
| 600 → 2.43/sec | 70 → 20.82/sec | c3596fe3a85381668598c75a1ffee4bf973a8518 | N/A (1) |
Note : Root cause : We have 3 watchdogs (2 @Scheduled into module sync and 1 into data sync) that was blocking each other as all @Scheduled methods share a single thread by default. ) | ||||
| 540 → 2.70 /sec | 88(?!) → 16.57/sec | 6e97bc6ee9f1cb2722aafc42325918e98da5cff2 | N/A (1) |
| ||||
10,000 → 4.17 / sec 10K in 40 mins | N/A | 0e7970a1ec12eac0f45fa016f98148da7a2266a8 | 10 | 1. Introduced timeout for connect and reading for RestTemplate while connecting dmi-plugin. 2. Modified data structure of moduleSyncSemaphore to hold YangModelCmHandle referance (cm handle id) as key of map 3. Working of Modulewatchdog is modified with parallelism and broken into async tasks. 4. ModuleSyncSemaphoreMap is now populated with advised cm handles and processed by Modulewatchdog by batch size (Dynamic batch size at cm registration). 5. Introduled parallelism level for async tasks that are suppiled. 6. Includes cm handle registration preformance code change. 7. Performance Improvement: Postgres Pool Configuration 8. Performance Improvement: Async Notification Pool Configuration. 9. Performance Improvement: Save/update batch of CM-Handle States in one operation. 10. Performance Improvement: Updating (many) child datanodes/datanode tree into one SQL operation 11. Performance Improvement: Native query to more efficiently get all yangResourceIds. 12. Performance Improvement: Detaching fragment entities early. 13. Performance Improvement: Decouple Temporal DB Notification from main thread. Note : It includes bottleneck of in my recursive method (Joe K is working on it to improve. org.onap.cps.spi.impl.CpsDataPersistenceServiceImpl#toDataNode) | ||||
Local Development Machine with CSIT docker deployment (with dmi-plugin that is implemented by CPS team) | As we are storing cm handle id into hazelcast distributed map, need aditional time to get YangModelCmHandle object from database : Get ADVISED cm handles +
| ADVISED to READY 97+296 = 393 | 30 → 48.6 / sec | 5 | Removed YangModelCmHandle object as key of map and stores only the reference that is cm handle id. | Note :
(org.onap.cps.spi.impl.CpsDataPersistenceServiceImpl#toDataNode)Joe K is working on it to improve. 2. dmi-plugin is able to handle only 5-8 cm handle per seconds concurrently. | ||
| 125+378 = 503 → 2.98 / sec | 33 → 45.45 / sec | 10 | ||||||
246 +356 = 602 | 31 → 48.39 / sec | 30 | ||||||
| Local Development Machine wit h dmi-plugin-perf-stub | 10, | 000→ 000 10K in 35 mins | Not measured | 0e7970a1ec12eac0f45fa016f98148da7a2266a8 | 10 |
|
|
Test Configuration
Use Case
...
| # | Parameter | Value |
|---|---|---|
| 1 | timers: advised-modules-sync: sleep-time-ms: | 30000 |
| 2 | timers: locked-modules-sync: sleep-time-ms: | 300000 |
| 3 | timers: cm-handle-data-sync: sleep-time-ms: | 30000 |
| 4 | data-sync: cache: enabled: | false |
Improvement Details
Performance matrix of improved methods that are called by ModuleSyncWatchdog or Cm handle registration API
...