/
CPS-2587 : Add CM Data Notification background Load to K6

CPS-2587 : Add CM Data Notification background Load to K6

Overview

This page documents the results of the CM data performance test conducted to evaluate the system's response time under different loads. The tests measured the impact of processing 500 and 1,000 AVC events per second on various CM-handle search and NCMP overhead operations.

Test Details

  • Test Environment: [Nordix-onap-cps-performance-test-k6 ]

  • Load Conditions:

    • No Load: Baseline measurement without additional event processing.

    • 500 AVC events/sec: Moderate load scenario.

    • 1,000 AVC events/sec: High load scenario.

  • Metric: Response time in milliseconds (ms) for search and pass-through operations.

  • Objective: Measure performance degradation (percentage loss/gain) under different load conditions.

  • The raw data is in the attached spreadsheet.

Results Summary

CM-Handle ID Search Performance

This test measures the time required to search for a CM-handle ID with different filters.

Test Name

No Load (ms)

500 AVC events/sec (ms)

% Loss

1000 AVC events/sec (ms)

% Loss

Test Name

No Load (ms)

500 AVC events/sec (ms)

% Loss

1000 AVC events/sec (ms)

% Loss

No Filter

1,003

1,067

-6.4%

1,122

-12.0%

Module Filter

243

259

-6.4%

264

-8.7%

Property Filter

2,233

2,404

-7.6%

2,541

-13.8%

Cps Path Filter

2,276

2,449

-7.6%

2,589

-13.7%

Trust Level Filter

11,353

12,283

-8.2%

13,367

-17.7%

Analysis:

  • All ID search operations show performance degradation as load increases.

  • The Trust Level Filter search has the highest latency, with a 17.7% increase in response time at 1,000 AVC events/sec.

CM-Handle Search Performance

These tests measure search time for CM-handles with various filters.

Test Name

No Load (ms)

500 AVC events/sec (ms)

% Loss

1000 AVC events/sec (ms)

% Loss

Test Name

No Load (ms)

500 AVC events/sec (ms)

% Loss

1000 AVC events/sec (ms)

% Loss

No Filter

8,393

9,175

-9.3%

9,809

-16.9%

Module Filter

10,578

11,420

-8.0%

12,000

-13.4%

Property Filter

12,499

13,383

-7.1%

14,519

-16.2%

Cps Path Filter

13,017

14,145

-8.7%

14,925

-14.7%

Trust Level Filter

22,222

23,840

-7.3%

25,452

-14.5%

Analysis:

  • Performance degradation is noticeable, especially at 1,000 AVC events/sec.

  • CM-handle search with No Filter shows a 16.9% degradation, indicating it is heavily impacted by load.

  • Trust Level Filter search remains the slowest operation across tests.

NCMP Overhead Analysis

This test measures the additional processing time for synchronous pass-through read/write operations.

Test Name

No Load (ms)

500 AVC events/sec (ms)

% Loss

1000 AVC events/sec (ms)

% Loss

Test Name

No Load (ms)

500 AVC events/sec (ms)

% Loss

1000 AVC events/sec (ms)

% Loss

Read

34

35

-2.5%

37

-7.5%

Read (Alternate ID)

92

95

-2.9%

98

-6.1%

Write

40

42

-5.6%

46

-14.6%

Write (Alternate ID)

90

97

-7.0%

100

-10.7%

Analysis:

  • Read operations are less affected by load compared to write operations.

  • Write operations, especially at 1,000 AVC events/sec, show a 14.6% increase in processing time.

Legacy Batch Read Performance

This test measures batch read operations per second.

Test Name

No Load (events/sec)

500 AVC events/sec

% Loss

1000 AVC events/sec

% Loss

Test Name

No Load (events/sec)

500 AVC events/sec

% Loss

1000 AVC events/sec

% Loss

Legacy Batch Read

116

110

-5.5%

107

-8.1%

Analysis:

  • Legacy batch read performance declines under load, with a 5.5% decrease at 500 AVC events/sec and an 8.1% decrease at 1,000 AVC events/sec. This indicates that as the system experiences more AVC events, the efficiency of batch read operations is negatively impacted. The increasing loss suggests that higher event rates introduce processing overhead, potentially due to resource contention, increased latency, or queuing delays. These factors may contribute to slower response times and reduced throughput for batch read operations under heavy system load.

Conclusion & Recommendations

  • Performance Degradation Observed: Across all test cases, response times increase under load, with Trust Level Filter searches being the most affected.

  • Read vs. Write Operations: NCMP synchronous write operations degrade more than read operations under high load.

  • Batch Read Stability: While performance shows a slight gain under load, batch reads remain consistent and improve efficiency as the event rate increases.

Overall Performance Impact:

  • The average performance loss at 500 AVC events/sec is -5.9%.

  • The average performance loss at 1,000 AVC events/sec is -11.5%.

This indicates a gradual performance degradation as the event rate increases, emphasizing the need for optimization in high-load scenarios.

Possible Bottlenecks

  • Kafka partitioning

    • using a single Kafka partition can significantly affect event processing, especially under high load. Here’s how it impacts performance:

Impact of Single Kafka Partition on Event Processing:

  1. Throughput Bottleneck:

    • Kafka distributes messages across partitions, allowing multiple consumers to process data in parallel.

    • With only one partition, all events are handled by a single consumer, limiting the processing speed.

  2. Increased Latency:

    • Since there is no parallelism, events are processed sequentially rather than concurrently.

    • As load increases (e.g., 500 or 1000 AVC events/sec), processing delays may accumulate, causing performance degradation.

  3. Consumer Scaling Limitation:

    • Kafka allows multiple consumers within a consumer group to read from different partitions.

    • With a single partition, adding more consumers will not improve performance since only one consumer can read from it at a time.

Recommendations to Improve Performance:

Increase the Number of Partitions

  • Use multiple partitions to enable parallel processing and higher throughput.

  • A good rule of thumb: Number of partitions = Number of consumers × Desired parallelism level.

Tune Consumer Configuration

  • Increase fetch.max.bytes, max.poll.records, and optimize commit.interval.ms for better performance.

Conclusion:

Yes, using a single Kafka partition is likely affecting event processing, especially under high event rates (500 or 1,000 AVC events/sec). Scaling partitions and optimizing consumer settings can help mitigate performance issues. 🚀

Notes: How to check kafka configuration of CPS ?

Checking in Kafka Configuration (For Default Settings)

If you haven't explicitly set the number of partitions while creating the topic, Kafka may use the default partition count from the broker settings.

To check Kafka's default number of partitions, run:

cat /etc/kafka/server.properties | grep num.partitions

Example Output:

num.partitions=1

num.partitions=1

  • If it's 1, all newly created topics will have a single partition by default unless overridden.

  • This image confirms that Kafka is configured with a single partition (num.partitions=1) into CPS.

image-20250219-112342.png

Related content

CPS-2169: NCMP to support 200k Cells (50KCM-handles)
CPS-2169: NCMP to support 200k Cells (50KCM-handles)
More like this
Comparison of 20K and 50K CH-handles
Comparison of 20K and 50K CH-handles
More like this
CPS R13 Release Planning
CPS R13 Release Planning
More like this
CPS Performance Testing
CPS Performance Testing
More like this
CPS R15 Release Planning
CPS R15 Release Planning
More like this
CPS R14 Release Planning
CPS R14 Release Planning
More like this