...

...

...

...

...

...

...

...

...

...

...

...

...

Kafka Consumer Performance on Linux:

1. CPU Efficiency:

   • Linux consumes 113.98% CPU, which is much lower compared to Windows (268.01%). For a Kafka consumer, this indicates that Linux efficiently handles Kafka message consumption with a lower CPU overhead. This is crucial in production environments where maximizing resource efficiency is key.

2. Memory Usage:

   • 522 MB of memory is used by the Kafka consumer on Linux. While Linux consumes slightly more memory than Windows, this difference is not significant. The memory footprint appears stable and manageable, showing that Linux maintains Kafka consumption processes effectively.

3. Network Data:

   • The Kafka consumer on Linux sends 59.6 MB of data and receives 113 MB of data, indicating a balanced network workload. Lower network data usage could imply efficient message batching or optimized data handling strategies, which are often critical in Kafka consumers for reducing latency and improving throughput.

4. Thread Management:

   • The 360-361 threads on Linux suggest a stable and scalable multi-threading architecture for consuming Kafka messages. Since Kafka consumers can handle parallel processing of messages, this thread count shows Linux is performing within a controlled range without overloading the system with too many threads.

5. Message Consumption Rate:

   • 33,333 messages per second on Linux is an exceptionally high consumption rate compared to Windows (16,667). This suggests that the Linux environment is highly optimized for Kafka consumers, allowing it to process a large volume of messages in real-time scenarios. This high throughput is vital in scenarios with large-scale data streams, where message backlogs need to be minimized.

6. Message Latency:

   • The first Kafka message is forwarded at 13:49:25 and the last at 13:49:28, showing a minimal delay between the start and completion of message forwarding. This low latency is essential in Kafka-based systems that rely on quick, real-time message processing, and it demonstrates that Linux provides fast message handling.

Conclusion for Kafka Consumers on Linux:

• Efficient Resource Usage: Linux is shown to handle Kafka consumer tasks with efficient CPU usage and a stable memory footprint.
• Superior Throughput: The 33,333 messages/second throughput shows Linux can handle very high message ingestion rates, making it suitable for data-intensive Kafka streams.
• Low Latency: The minimal lag in forwarding Kafka messages indicates that Linux performs well under real-time processing demands.
• Optimized for Network and Threads: Balanced network data handling and stable thread management further highlight Linux’s ability to maintain performance under high workloads.

Overall, Linux is an excellent environment for running Kafka consumers, especially in scenarios that demand high throughput, low latency, and efficient resource management.

Key Observations:

1. CPU Usage:
   • Linux shows significantly lower CPU usage (113.98%) compared to Windows (268.01%), indicating that the process is more CPU-efficient on Linux.
2. Memory Usage:
   • Linux uses slightly more memory (522 MB) than Windows (497.3 MB), but the difference is minimal.
3. Network Data:
   • Linux sends less data (59.6 MB) than Windows (120 MB) but also receives less data (113 MB vs 168 MB). This could indicate different network handling efficiencies or workload patterns.
4. Threads:
   • The number of threads is almost identical between the two environments, so thread management is consistent.
5. Kafka Message Timing:
   • Linux processed Kafka messages earlier (13:49:25) than Windows (14:56:33). The processing duration is almost the same, but Linux starts earlier.
6. Consumption Rate:
   • Linux has a much higher message consumption rate (33,333 messages/second) than Windows (16,667 messages/second). This suggests that Linux is significantly more efficient in handling Kafka messages.

Conclusion:

• Linux demonstrates better performance overall in terms of CPU usage and message consumption rate. Despite similar memory usage, Linux handles Kafka message forwarding faster and consumes more messages per second.
• Windows consumes significantly more CPU resources and handles Kafka messages at a slower rate, making it less efficient for high-performance use cases.

If high throughput and CPU efficiency are critical, Linux would be the better environment for this application based on these metrics.

Table of Contents

• Test Flow Description
  • Test Data
• Environment
  • Hardware details
  • Software details
• Test Results
  • Operation execution time
  • Resource usage
• Test Application
  • Source
  • Setting up

This detailed workload setup provides a comprehensive overview of the Kafka consumer environment on Linux, detailing how cloud events are handled, system resource configurations, and message publishing mechanisms.

Test Flow Description

According to requirements the test flow included following operation:

• Use K6 script (k6-tests\once-off-test\kafka\produce-avc-event.js) to publish 100,000 cm avc cloud events to  topic ""

Load:

• The flow was repeated in a loop multiple times (defined with circles parameter);
  no delay between loops
• The flow loops were executed in a multiple concurrent threads (threads parameter);
  all threads were started subsequently with no delay
• each thread used same CPS service (embedded case) and REST client (remote case) instances
• All the data used (except schema set preparation) was taken from memory (loaded on preparation stage)

Test Environment

This YAML snippet is a CPS resource configuration for deploying a service or containerized application. It defines the resource allocations and limits for the deployment, possibly for a container orchestrator like Kubernetes or Docker Swarm.

Here's a breakdown of the configuration:

replicas: 1: Specifies that there should be 1 replica of this deployment (only one instance will run).

resources: Defines the resource requests and limits for CPU and memory.

reservations: The minimum amount of resources guaranteed for the container.
cpus: '2': The container requests 2 CPUs.
memory: 2G: The container requests 2GB of memory.
limits: The maximum amount of resources the container can use.
cpus: '3': The container can use up to 3 CPUs.
memory: 3G: The container can use up to 3GB of memory.
This configuration ensures that the container will always have at least 2 CPUs and 2GB of memory but can scale up to 3 CPUs and 3GB of memory when needed. If it tries to exceed those limits, it might be throttled or killed by the orchestrator, depending on the setup.

dmi-cm-events

{
  "data": {
    "push-change-update": {
      "datastore-changes": {
        "ietf-yang-patch:yang-patch": {
          "patch-id": "34534ffd98",
          "edit": [
            {
              "edit-id": "ded43434-1",
              "operation": "replace",
              "target": "ran-network:ran-network/NearRTRIC[@id='22']/GNBCUCPFunction[@id='cucpserver2']/NRCellCU[@id='15549']/NRCellRelation[@id='14427']",
              "value": {
                "attributes": []
              }
            },
            {
              "edit-id": "ded43434-2",
              "operation": "create",
              "target": "ran-network:ran-network/NearRTRIC[@id='22']/GNBCUCPFunction[@id='cucpserver1']/NRCellCU[@id='15548']/NRCellRelation[@id='14426']",
              "value": {
                "attributes": [
                  {
                    "isHoAllowed": false
                  }
                ]
              }
            },
            {
              "edit-id": "ded43434-3",
              "operation": "delete",
              "target": "ran-network:ran-network/NearRTRIC[@id='22']/GNBCUCPFunction[@id='cucpserver1']/NRCellCU[@id='15548']/NRCellRelation[@id='14426']"
            }
          ]
        }
      }
    }
  }
}

Performance Comparison Between Linux and Windows Kafka Consumers:

Kafka Consumer Performance on Linux:

1. CPU Efficiency:

   • Linux consumes 113.98% CPU, which is much lower compared to Windows (268.01%). For a Kafka consumer, this indicates that Linux efficiently handles Kafka message consumption with a lower CPU overhead. This is crucial in production environments where maximizing resource efficiency is key.

2. Memory Usage:

   • 522 MB of memory is used by the Kafka consumer on Linux. While Linux consumes slightly more memory than Windows, this difference is not significant. The memory footprint appears stable and manageable, showing that Linux maintains Kafka consumption processes effectively.

3. Network Data:

   • The Kafka consumer on Linux sends 59.6 MB of data and receives 113 MB of data, indicating a balanced network workload. Lower network data usage could imply efficient message batching or optimized data handling strategies, which are often critical in Kafka consumers for reducing latency and improving throughput.

4. Thread Management:

   • The 360-361 threads on Linux suggest a stable and scalable multi-threading architecture for consuming Kafka messages. Since Kafka consumers can handle parallel processing of messages, this thread count shows Linux is performing within a controlled range without overloading the system with too many threads.

5. Message Consumption Rate:

   • 33,333 messages per second on Linux is an exceptionally high consumption rate compared to Windows (16,667). This suggests that the Linux environment is highly optimized for Kafka consumers, allowing it to process a large volume of messages in real-time scenarios. This high throughput is vital in scenarios with large-scale data streams, where message backlogs need to be minimized.

6. Message Latency:

   • The first Kafka message is forwarded at 13:49:25 and the last at 13:49:28, showing a minimal delay between the start and completion of message forwarding. This low latency is essential in Kafka-based systems that rely on quick, real-time message processing, and it demonstrates that Linux provides fast message handling.

Conclusion for Kafka Consumers on Linux:

• Efficient Resource Usage: Linux is shown to handle Kafka consumer tasks with efficient CPU usage and a stable memory footprint.
• Superior Throughput: The 33,333 messages/second throughput shows Linux can handle very high message ingestion rates, making it suitable for data-intensive Kafka streams.
• Low Latency: The minimal lag in forwarding Kafka messages indicates that Linux performs well under real-time processing demands.
• Optimized for Network and Threads: Balanced network data handling and stable thread management further highlight Linux’s ability to maintain performance under high workloads.

Overall, Linux is an excellent environment for running Kafka consumers, especially in scenarios that demand high throughput, low latency, and efficient resource management.

Key Observations:

1. CPU Usage:
   • Linux shows significantly lower CPU usage (113.98%) compared to Windows (268.01%), indicating that the process is more CPU-efficient on Linux.
2. Memory Usage:
   • Linux uses slightly more memory (522 MB) than Windows (497.3 MB), but the difference is minimal.
3. Network Data:
   • Linux sends less data (59.6 MB) than Windows (120 MB) but also receives less data (113 MB vs 168 MB). This could indicate different network handling efficiencies or workload patterns.
4. Threads:
   • The number of threads is almost identical between the two environments, so thread management is consistent.
5. Kafka Message Timing:
   • Linux processed Kafka messages earlier (13:49:25) than Windows (14:56:33). The processing duration is almost the same, but Linux starts earlier.
6. Consumption Rate:
   • Linux has a much higher message consumption rate (33,333 messages/second) than Windows (16,667 messages/second). This suggests that Linux is significantly more efficient in handling Kafka messages.

Conclusion:

• Linux demonstrates better performance overall in terms of CPU usage and message consumption rate. Despite similar memory usage, Linux handles Kafka message forwarding faster and consumes more messages per second.
• Windows consumes significantly more CPU resources and handles Kafka messages at a slower rate, making it less efficient for high-performance use cases.

If high throughput and CPU efficiency are critical, Linux would be the better environment for this application based on these metrics.