Table of Contents

• Test Flow Description
  • Test Data
• Environment
  • Hardware details
  • Software details
• Test Results
  • Operation execution time
  • Resource usage
• Test Application
  • Source
  • Setting up

This detailed workload setup provides a comprehensive overview of the Kafka consumer environment on Linux, detailing how cloud events are handled, system resource configurations, and message publishing mechanisms.

Test Flow Description

According to requirements the test flow included following steps:

• Use docker compose (C:\CPS\master\cps\docker-compose\docker-compose.yml) to make all the required containers running

  docker-compose --profile dmi-stub --profile monitoring up -d 

• Stop "cps-and-ncmp" container.
• Use K6 script (k6-tests\once-off-test\kafka\produce-avc-event.js) to publish cm avc cloud events to  topic "dmi-cm-events"
• Verify number of messages published to topic "dmi-cm-events"  using kafka UI "http://localhost:8089/"  
• Start "cps-and-ncmp" container again
• Verify number of messages consumed by topic "cm-events"  using kafka UI "http://localhost:8089/"  
• Login  grafana http://localhost:3000/ to capture CPU, Memory and Threads.

Load:

• The flow was repeated in a loop multiple times (defined with circles parameter);

  K6 Scenarios Expand source

   produce_cm_avc_event: {
              executor: 'shared-iterations',
              exec: 'produce_cm_avc_event',
              vus: 1000, // You can adjust VUs to meet performance requirements
              iterations: $TOTAL_MESSAGES, // Total messages to publish  (for example: 100 K , 200 K)
              maxDuration: '15m', // Adjust depending on the expected completion time
          }

Test Environment

dmi-cm-events

{
  "data": {
    "push-change-update": {
      "datastore-changes": {
        "ietf-yang-patch:yang-patch": {
          "patch-id": "34534ffd98",
          "edit": [
            {
              "edit-id": "ded43434-1",
              "operation": "replace",
              "target": "ran-network:ran-network/NearRTRIC[@id='22']/GNBCUCPFunction[@id='cucpserver2']/NRCellCU[@id='15549']/NRCellRelation[@id='14427']",
              "value": {
                "attributes": []
              }
            },
            {
              "edit-id": "ded43434-2",
              "operation": "create",
              "target": "ran-network:ran-network/NearRTRIC[@id='22']/GNBCUCPFunction[@id='cucpserver1']/NRCellCU[@id='15548']/NRCellRelation[@id='14426']",
              "value": {
                "attributes": [
                  {
                    "isHoAllowed": false
                  }
                ]
              }
            },
            {
              "edit-id": "ded43434-3",
              "operation": "delete",
              "target": "ran-network:ran-network/NearRTRIC[@id='22']/GNBCUCPFunction[@id='cucpserver1']/NRCellCU[@id='15548']/NRCellRelation[@id='14426']"
            }
          ]
        }
      }
    }
  }
}

Performance Comparison Between Linux and Windows Kafka Consumers:

Kafka Consumer Performance on Linux:

1. CPU Efficiency:

   • Linux consumes 113.98% CPU, which is much lower compared to Windows (268.01%). For a Kafka consumer, this indicates that Linux efficiently handles Kafka message consumption with a lower CPU overhead. This is crucial in production environments where maximizing resource efficiency is key.

2. Memory Usage:

   • 522 MB of memory is used by the Kafka consumer on Linux. While Linux consumes slightly more memory than Windows, this difference is not significant. The memory footprint appears stable and manageable, showing that Linux maintains Kafka consumption processes effectively.

3. Network Data:

   • The Kafka consumer on Linux sends 59.6 MB of data and receives 113 MB of data, indicating a balanced network workload. Lower network data usage could imply efficient message batching or optimized data handling strategies, which are often critical in Kafka consumers for reducing latency and improving throughput.

4. Thread Management:

   • The 360-361 threads on Linux suggest a stable and scalable multi-threading architecture for consuming Kafka messages. Since Kafka consumers can handle parallel processing of messages, this thread count shows Linux is performing within a controlled range without overloading the system with too many threads.

5. Message Consumption Rate:

   • 33,333 messages per second on Linux is an exceptionally high consumption rate compared to Windows (16,667). This suggests that the Linux environment is highly optimized for Kafka consumers, allowing it to process a large volume of messages in real-time scenarios. This high throughput is vital in scenarios with large-scale data streams, where message backlogs need to be minimized.

6. Message Latency:

   • The first Kafka message is forwarded at 13:49:25 and the last at 13:49:28, showing a minimal delay between the start and completion of message forwarding. This low latency is essential in Kafka-based systems that rely on quick, real-time message processing, and it demonstrates that Linux provides fast message handling.

Conclusion for Kafka Consumers on Linux:

• Efficient Resource Usage: Linux is shown to handle Kafka consumer tasks with efficient CPU usage and a stable memory footprint.
• Superior Throughput: The 33,333 messages/second throughput shows Linux can handle very high message ingestion rates, making it suitable for data-intensive Kafka streams.
• Low Latency: The minimal lag in forwarding Kafka messages indicates that Linux performs well under real-time processing demands.
• Optimized for Network and Threads: Balanced network data handling and stable thread management further highlight Linux’s ability to maintain performance under high workloads.

Overall, Linux is an excellent environment for running Kafka consumers, especially in scenarios that demand high throughput, low latency, and efficient resource management.

Key Observations:

1. CPU Usage:
   • Linux shows significantly lower CPU usage (113.98%) compared to Windows (268.01%), indicating that the process is more CPU-efficient on Linux.
2. Memory Usage:
   • Linux uses slightly more memory (522 MB) than Windows (497.3 MB), but the difference is minimal.
3. Network Data:
   • Linux sends less data (59.6 MB) than Windows (120 MB) but also receives less data (113 MB vs 168 MB). This could indicate different network handling efficiencies or workload patterns.
4. Threads:
   • The number of threads is almost identical between the two environments, so thread management is consistent.
5. Kafka Message Timing:
   • Linux processed Kafka messages earlier (13:49:25) than Windows (14:56:33). The processing duration is almost the same, but Linux starts earlier.
6. Consumption Rate:
   • Linux has a much higher message consumption rate (33,333 messages/second) than Windows (16,667 messages/second). This suggests that Linux is significantly more efficient in handling Kafka messages.

Conclusion:

• Linux demonstrates better performance overall in terms of CPU usage and message consumption rate. Despite similar memory usage, Linux handles Kafka message forwarding faster and consumes more messages per second.
• Windows consumes significantly more CPU resources and handles Kafka messages at a slower rate, making it less efficient for high-performance use cases.

If high throughput and CPU efficiency are critical, Linux would be the better environment for this application based on these metrics.