Test architecture

In order to conduct client tests this will be conducted in following architecture:

HV-VES Client - produces high amount of events for processing.
Processing Consumer - consumes events from Kafka topics and creates performance metrics.
Offset Consumer - reads Kafka offsets.
Prometheus - sends requests for performance metrics to HV-VES, Processing Consumer and Offset Consumer, provides data to Grafana.
Grafana - delivers analytics and its visualization.

Note: In the Without DMaaP Kafka tests the DMaaP/Kafka service was substituted with wurstmeister kafka

Environment and Resources

Kubernetes cluster with 4 worker nodes, sharing hardware configuration shown in a table below, is deployed in OpenStack cloud operating system. The test components in docker containers are further deployed on the Kubernetes cluster.

Configuration
CPU	Model	Intel(R) Xeon(R) CPU E5-2680 v4
	No. of cores	24
	CPU clock speed [GHz]	2.40
Total RAM [GB]		62.9

Network Performance

Pod measurement method

In order to check cluster network performance tests with usage of Iperf3 have been applied. Iperf is a tool for measurement of the maximum bandwidth on IP networks, it runs on two mode: server and client. We used a docker image: networkstatic/iperf3.

Following deployment creates a pod with iperf (server mode) on one worker, and one pod with iperf client for each worker.

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iperf3-server
  namespace: onap
  labels:
    app: iperf3-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: iperf3-server
  template:
    metadata:
      labels:
        app: iperf3-server
    spec:
      containers:
        - name: iperf3-server
          image: networkstatic/iperf3
          args: ['-s']
          ports:
            - containerPort: 5201
              name: server


---

apiVersion: v1
kind: Service
metadata:
  name: iperf3-server
  namespace: onap
spec:
  selector:
    app: iperf3-server
  ports:
    - protocol: TCP
      port: 5201
      targetPort: server



---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: iperf3-clients
  namespace: onap
  labels:
    app: iperf3-client
spec:
  selector:
    matchLabels:
      app: iperf3-client
  template:
    metadata:
      labels:
        app: iperf3-client
    spec:
      containers:
        - name: iperf3-client
          image: networkstatic/iperf3
          command: ['/bin/sh', '-c', 'sleep infinity']

To create deployment, execute following command:

kubectl create -f deployment.yaml

To find all iperf pods, execute:

kubectl -n onap get pods -o wide | grep iperf

To measure connection between pods, run iperf on iperf-client pod, using following command:

kubectl -n onap exec -it <iperf-client-pod> -- iperf3 -c iperf3-server

To change output format from MBits/sec to MBytes/sec:

kubectl -n onap exec -it <iperf-client-pod> -- iperf3 -c iperf3-server -f MBytes

To change measure time:

kubectl -n onap exec -it <iperf-client-pod> -- iperf3 -c iperf3-server -t <time-in-second>

To gather results, the command was executed:

kubectl -n onap exec -it <iperf-client-pod> -- iperf3 -c iperf3-server -f MBytes

Results of performed tests

worker1 (136 MBytes/sec)

results

Connecting to host iperf3-server, port 5201
[  4] local 10.42.5.127 port 39752 connected to 10.43.25.161 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   141 MBytes   141 MBytes/sec   32    673 KBytes       
[  4]   1.00-2.00   sec   139 MBytes   139 MBytes/sec    0    817 KBytes       
[  4]   2.00-3.00   sec   139 MBytes   139 MBytes/sec    0    936 KBytes       
[  4]   3.00-4.00   sec   138 MBytes   137 MBytes/sec    0   1.02 MBytes       
[  4]   4.00-5.00   sec   138 MBytes   137 MBytes/sec    0   1.12 MBytes       
[  4]   5.00-6.00   sec   129 MBytes   129 MBytes/sec    0   1.20 MBytes       
[  4]   6.00-7.00   sec   129 MBytes   129 MBytes/sec    0   1.27 MBytes       
[  4]   7.00-8.00   sec   134 MBytes   134 MBytes/sec    0   1.35 MBytes       
[  4]   8.00-9.00   sec   135 MBytes   135 MBytes/sec    0   1.42 MBytes       
[  4]   9.00-10.00  sec   135 MBytes   135 MBytes/sec   45   1.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.32 GBytes   136 MBytes/sec   77             sender
[  4]   0.00-10.00  sec  1.32 GBytes   135 MBytes/sec                  receiver

worker2 (87 MBytes/sec)

results

Connecting to host iperf3-server, port 5201
[  4] local 10.42.3.188 port 35472 connected to 10.43.25.161 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  88.3 MBytes  88.3 MBytes/sec  121    697 KBytes       
[  4]   1.00-2.00   sec  96.2 MBytes  96.3 MBytes/sec    0    796 KBytes       
[  4]   2.00-3.00   sec  92.5 MBytes  92.5 MBytes/sec    0    881 KBytes       
[  4]   3.00-4.00   sec  90.0 MBytes  90.0 MBytes/sec    0    957 KBytes       
[  4]   4.00-5.00   sec  87.5 MBytes  87.5 MBytes/sec    0   1.00 MBytes       
[  4]   5.00-6.00   sec  88.8 MBytes  88.7 MBytes/sec    0   1.06 MBytes       
[  4]   6.00-7.00   sec  80.0 MBytes  80.0 MBytes/sec    0   1.12 MBytes       
[  4]   7.00-8.00   sec  81.2 MBytes  81.3 MBytes/sec   25    895 KBytes       
[  4]   8.00-9.00   sec  85.0 MBytes  85.0 MBytes/sec    0    983 KBytes       
[  4]   9.00-10.00  sec  83.8 MBytes  83.7 MBytes/sec    0   1.03 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   873 MBytes  87.3 MBytes/sec  146             sender
[  4]   0.00-10.00  sec   870 MBytes  87.0 MBytes/sec                  receiver

worker3 (135 MBytes/sec)

results

Connecting to host iperf3-server, port 5201
[  4] local 10.42.4.182 port 35288 connected to 10.43.25.161 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   129 MBytes   129 MBytes/sec   45   1.17 MBytes       
[  4]   1.00-2.00   sec   134 MBytes   134 MBytes/sec   32   1.25 MBytes       
[  4]   2.00-3.00   sec   135 MBytes   135 MBytes/sec    0   1.32 MBytes       
[  4]   3.00-4.00   sec   139 MBytes   139 MBytes/sec    0   1.40 MBytes       
[  4]   4.00-5.00   sec   144 MBytes   144 MBytes/sec    0   1.47 MBytes       
[  4]   5.00-6.00   sec   131 MBytes   131 MBytes/sec   45   1.14 MBytes       
[  4]   6.00-7.00   sec   129 MBytes   129 MBytes/sec    0   1.25 MBytes       
[  4]   7.00-8.00   sec   134 MBytes   134 MBytes/sec    0   1.33 MBytes       
[  4]   8.00-9.00   sec   138 MBytes   138 MBytes/sec    0   1.39 MBytes       
[  4]   9.00-10.00  sec   135 MBytes   135 MBytes/sec    0   1.44 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.31 GBytes   135 MBytes/sec  122             sender
[  4]   0.00-10.00  sec  1.31 GBytes   134 MBytes/sec                  receiver

worker0 (2282 MBytes/sec) (iperf client and server exist on same worker )

results

Connecting to host iperf3-server, port 5201
[  4] local 10.42.6.132 port 51156 connected to 10.43.25.161 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  2.13 GBytes  2185 MBytes/sec    0    536 KBytes       
[  4]   1.00-2.00   sec  1.66 GBytes  1702 MBytes/sec    0    621 KBytes       
[  4]   2.00-3.00   sec  2.10 GBytes  2154 MBytes/sec    0    766 KBytes       
[  4]   3.00-4.00   sec  1.89 GBytes  1937 MBytes/sec    0   1.01 MBytes       
[  4]   4.00-5.00   sec  1.87 GBytes  1914 MBytes/sec    0   1.39 MBytes       
[  4]   5.00-6.00   sec  2.76 GBytes  2826 MBytes/sec    0   1.39 MBytes       
[  4]   6.00-7.00   sec  1.81 GBytes  1853 MBytes/sec  792   1.09 MBytes       
[  4]   7.00-8.00   sec  2.54 GBytes  2600 MBytes/sec    0   1.21 MBytes       
[  4]   8.00-9.00   sec  2.70 GBytes  2763 MBytes/sec    0   1.34 MBytes       
[  4]   9.00-10.00  sec  2.82 GBytes  2889 MBytes/sec    0   1.34 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  22.3 GBytes  2282 MBytes/sec  792             sender
[  4]   0.00-10.00  sec  22.3 GBytes  2282 MBytes/sec                  receiver

Average speed (without worker 0 ) : 119 MBytes/sec

Test Setup

Preconditions

Installed ONAP
Plain TCP connection between HV-VES and clients (default configuration)
Metric port exposed on HV-VES service

In order to reach metrics endpoint in HV-VES there is a need to add the following lines in the ports section of HV-VES service configuration file:

Lines to add to ports section of HV-VES service configuration file

  - name: port-t-6060
	port: 6060
	protocol: TCP
	targetPort: 6060

Before start tests, download docker image of producer which is available here. To extract image locally use command:

docker load < hv-collector-go-client.tar.gz

To execute performance tests we have to run functions from a shell script cloud-based-performance-test.sh in HV-VES project directory: ~/tools/performance/cloud/

First we have to generate certificates in ~/tools/ssl folder by using gen_certs. This step only needs to be performed during the first test setup (or if the generated files have been deleted).
Generating certificates
```
./cloud-based-performance-test.sh gen_certs
```
Then we call setup in order to send certificates to HV-VES, and deploy Consumers, Prometheus, Grafana and create their ConfigMaps.
Setting up the test environment
```
./cloud-based-performance-test.sh setup
```
After that we have to change HV-VES configuration in Consul KEY/VALUE tab (typically we can access Consul at port 30270 of any Controller node, i.e. http://slave1:30270/ui/#/dc1/kv/dcae-hv-ves-collector/edit).
HV-VES Consul configuration Expand source
```
{"security.sslDisable": false,
"logLevel": "INFO",
"server.listenPort": 6061,
"server.idleTimeoutSec": 300,
"cbs.requestIntervalSec": 5,
"streams_publishes": {
	"perf3gpp": {
		"type": "kafka",
		"aaf_credentials": {
			"username": "admin",
			"password": "admin_secret"
			},
		"kafka_info": {
			"bootstrap_servers": "message-router-kafka:9092",
			"topic_name": "HV_VES_PERF3GPP"
			}
		}
	},
"security.keys.trustStoreFile": "/etc/ves-hv/ssl/custom/trust.p12",
"security.keys.keyStoreFile": "/etc/ves-hv/ssl/custom/server.p12",
"security.keys.trustStorePasswordFile":"/etc/ves-hv/ssl/custom/trust.pass",
"security.keys.keyStorePasswordFile": "/etc/ves-hv/ssl/custom/server.pass"}
```
Hint

How to access Consul UI?
Consul's address: http://<worker external IP>:<Consul External Port>

To check Consul External Port, execute:
ubuntu@onap-5422-rke-node:~$ kubectl -n onap get svc | grep consul
consul-server-ui                   NodePort       10.43.132.178   <none>                                 8500:31190/TCP                                           6d20h
----------------------------------------------------------------------------------------------------------------------------------------------
If service "consul-server-ui" is not exposed to external port (NodePort) and is configured as Cluster IP, please follow steps below.

ubuntu@onap-5422-rke-node:~$ kubectl -n onap get svc | grep consul
consul-server-ui                                       ClusterIP      10.43.132.178   <none>                                 8500/TCP                                                 25h

ubuntu@onap-5422-rke-node:~$ kubectl -n onap edit svc consul-server-ui
----
...
apiVersion: v1
kind: Service
metadata:
...
spec:
...
ports:
...
type: ClusterIP --> NodePort ### change value to NodePort
status:
...
---------------------------------------
service/consul-server-ui edited
ubuntu@onap-5422-rke-node:~$ kubectl -n onap get svc | grep consul
consul-server-ui                                       NodePort       10.43.132.178   <none>                                 8500:31190/TCP                                           25h

After completing previous steps we can run the test. The table below contains the parameters that can be passed to cloud-based-performance-test.sh script.

gen_certs	generate certs in ../../ssl directory
setup	set up ConfigMaps and consumers
setup_all	set up ConfigMaps consumers and producers
send_config	send producers configuration (message interval and payload), located in producers-config/producer-config.json to each producer
start_interval	start interval mode, config file is located in producers-config/interval-config.json Optional parameters: --producers : number of producers in deployment (10) --retention-time-minutes : messages retention time on kafka in minutes (60)
start_instant	start_instant : start instant mode, config file is located in producers-config/instant-config.json Optional parameters: --producers : number of producers in deployment (10) --retention-time-minutes : messages retention time on kafka in minutes (60)
scale_producers	scale producer deployment to number provide in argument
stop	stop all producers
reset_producers	reset all metrics on each producer
clean	remove ConfigMap, HV-VES consumers
clean_all	remove ConfigMap, HV-VES consumers and producers
help	print usage

Performing the test

./cloud-based-performance-test.sh start_interval --producers 10

The command above starts the test that creates 10 producers which send messages in the interval mode. The parameters can be changed in configuration files located in producers-config folder.

interval-config.json

{
  "intervalConfigs": [
    {
      "duration": 1,
      "interval": 100
    },
    {
      "duration": 2,
       "interval": 100
    }
  ]
}

Above request body queues a task to create new connections for 1 second with 100 milliseconds interval and then for 2 seconds with 100 milliseconds interval, meaning that overall of 30 connections should be setup over 3 seconds.

producer-config.json

{
  "payloadSize": 8192,
  "messageIntervalMs": 100
}

Above request updates configuration of producers to send messages with 8192 bytes with 100 milliseconds interval.

It is also possible to run the test in instant mode.

instant-config.json

{
  "connections": 500
}

Above request queues a task to create 500 connections without intervals between them.

The test.properties file contains Producers and Consumers configurations and it allows setting following properties:

Producer
hvVesAddress	HV-VES address (dcae-hv-ves-collector.onap:6061)
Certificates paths
client.cert.path	Path to cert file (/ssl/client.p12)
client.cert.pass.path	Path to cert's pass file (/ssl/client.pass)
Consumer
kafka.bootstrapServers	Adress of Kafka service to consume from (message-router-kafka:9092)
kafka.topics	Kafka topics to subscribe to (HV_VES_PERF3GPP)

Results can be accessed under following links:

To remove created ConfigMaps, Consumers, Producers, Grafana and Prometheus from Kubernetes cluster we call clean function. Note: clean doesn't remove certificates from HV-VES.
Cleaning the environment
```
./cloud-based-performance-test.sh clean
```

In order to restart the test environment, which means redeploying hv-ves pod, resetting kafka topic and performing setup, we use reboot-test-environment.sh.

Restarting the test environment

./reboot-test-environment.sh

Results can be accessed under following links:

HV-VES Performance test results

With dmaap Kafka

Conditions

Tests were performed with 5 repetitions for each configuration shown in the table below.

Number of producers	Messages per producer	Payload size [B]	Interval [ms]
2	90000	8192	10
4	90000	8192	10
6	60000	8192	10

Raw results data

Raw results data with screenshots can be found in following files:

Series 1 - results_series_1.zip
Series 2 - results_series_2.zip

Test Results - series 1

Click here to see results...

Below tables show the test results across a wide range of containers' number.

NUMBER OF PRODUCERS	TOTAL MESSAGES PROCESSED	AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]	AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]	PEAK INCOMING DATA RATE [MB/s]	PEAK PROCESSING MESSAGE QUEUE SIZE	PEAK CPU LOAD [%]	PEAK MEMORY USAGE [GB]
2	180000	0.03	54	1.6	175	2.9	0.38
2	180000	0.021	8.5	1.6	6	3.2	0.37
2	180000	0.022	14.8	1.6	66	2.4	0.38
2	180000	0.023	19.4	1.6	59	2.2	0.38
2	180000	0.021	7.6	1.6	58	2.6	0.36

NUMBER OF PRODUCERS	TOTAL MESSAGES PROCESSED	DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES	AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]	AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]	PEAK INCOMING DATA RATE [MB/s]	PEAK PROCESSING MESSAGE QUEUE SIZE	PEAK CPU LOAD [%]	PEAK MEMORY USAGE [GB]
4	315868	44824	82	260	3,2	2400	3	0,4
4	359193	807	3660	10408	3,2	3100	4,9	0,78
4	359401	591	1350	3486	3.7	3000	4,9	0,56
4	360000	0	1.5	130	3.3	1200	4.1	0.39
4	360000	0	0.02	57	3.3	180	3.5	0.38

NUMBER OF PRODUCERS	TOTAL MESSAGES PROCESSED	DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES	AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]	AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]	PEAK INCOMING DATA RATE [MB/s]	PEAK PROCESSING MESSAGE QUEUE SIZE	PEAK CPU LOAD [%]	PEAK MEMORY USAGE [GB]
6	358475	1525	6600	17875	4.9	4900	5.8	0.97
6	358494	1506	7080	20182	4.7	4500	4.5	0.97
6	359244	756	4900	17231	4.9	1900	3.9	0.93
6	358944	1056	6150	17200	4.9	4800	5	0.97
6	358689	1311	5410	17202	4.9	4100	5	0.96

Test Results - series 2

Click here to see results...

NUMBER OF PRODUCERS	TOTAL MESSAGES PROCESSED	DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES	AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]	AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]	PEAK INCOMING DATA RATE [MB/s]	PEAK PROCESSING MESSAGE QUEUE SIZE	PEAK CPU LOAD [%]	PEAK MEMORY USAGE [GB]	RESULTS PRESENTED IN GRAFANA
2	180000	0	0.026	8.9	1.6	11	3	0.35
2	180000	0	0.026	11.3	1.6	23	3.2	0.35

NUMBER OF PRODUCERS	TOTAL MESSAGES PROCESSED	DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES	AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [s]	AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]	PEAK INCOMING DATA RATE [MB/s]	PEAK PROCESSING MESSAGE QUEUE SIZE	PEAK CPU LOAD [%]	PEAK MEMORY USAGE [GB]	RESULTS PRESENTED IN GRAFANA
4	360000	0	0.022	41.3	3.2	130	4	0.35
4	360000	0	0.023	40.7	3.2	370	4.2	0.35

NUMBER OF PRODUCERS	TOTAL MESSAGES PROCESSED	DIFFERENCE BETWEEN ALL MESSAGES AND SENT TO HV-VES	AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]	AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]	PEAK INCOMING DATA RATE [MB/s]	PEAK PROCESSING MESSAGE QUEUE SIZE	PEAK CPU LOAD [%]	PEAK MEMORY USAGE [GB]	RESULTS PRESENTED IN GRAFANA
6	359317	683	6240	15965	4.8	4500	4.3	1
6	359217	783	6490	16834	4.8	4200	5.8	1

No DMaaP Kafka SetUp

Install Kafka Docker on Kubernetes

(based on: ultimate-guide-to-installing-kafka-docker-on-kuber)

Create config maps

Config maps are required by zookeeper and kafka-broker deployments.

Create kafka-config-map

kubectl -n onap create cm kafka-config-map --from-file=kafka_server_jaas.conf

kafka_server_jaas.conf

KafkaServer {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin_secret"
   user_admin="admin_secret";
};
Client {
   org.apache.zookeeper.server.auth.DigestLoginModule required
   username="kafka"
   password="kafka_secret";
};

Create zk-config-map

kubectl -n onap create cm zk-config-map --from-file=zk_server_jaas.conf

zk_server_jaas.conf

Server {
       org.apache.zookeeper.server.auth.DigestLoginModule required
       user_kafka="kafka_secret";
};

Create deployments

Create zookeeper deployment

kubectl -n onap create -f zookeeper.yml

zookeeper.yml

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-1
spec:
  template:
    metadata:
      labels:
        app: zookeeper-1
    spec:
      containers:
      - name: zoo1
        image: digitalwonderland/zookeeper
        volumeMounts:
        - name: config-volume
          mountPath: "/home/ubuntu/kafkadepl/myconfig/zk_server_jaas.conf"
          subPath: zk_server_jaas.conf
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1
        - name: KAFKA_OPTS
          value: "-Djava.security.auth.login.config=/home/ubuntu/kafkadepl/myconfig/zk_server_jaas.conf -Dzookeeper.kerberos.removeHostFromPrincipal=true -Dzookeeper.kerberos.removeRealmFromPrincipal=true -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -Dzookeeper.requireClientAuthScheme=sasl"
        - name: SERVER_JVMFLAGS
          value: "-Djava.security.auth.login.config=/home/ubuntu/kafkadepl/myconfig/zk_server_jaas.conf -Dzookeeper.skipACL=yes"
        - name: ZOOKEEPER_AUTHPROVIDER_1
          value: "org.apache.zookeeper.server.auth.SASLAuthenticationProvider"
        - name: ZOOKEEPER_REQUIRECLIENTAUTHSCHEME
          value: "SASL"
      volumes:
      - name: config-volume
        configMap:
          name: zk-config-map
---
apiVersion: v1
kind: Service
metadata:
  name: zoo1
  labels:
    app: zookeeper-1
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-1

Create kafka-service

kubectl -n onap create -f kafka-service.yml

kafka-service.yml

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP
  selector:
    app: kafka
    id: "0"
  type: LoadBalancer

Create kafka-broker

kubectl -n onap create -f kafka-broker.yml

kafka-broker.yml

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-broker0
spec:
  template:
    metadata:
      labels:
        app: kafka
        id: "0"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        volumeMounts:
        - name: config-volume
          mountPath: "/home/ubuntu/kafkadepl/myconfig/kafka_server_jaas.conf"
          subPath: kafka_server_jaas.conf
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "30718"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: 10.43.155.122
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: "zoo1:2181"
        - name: KAFKA_BROKER_ID
          value: "0"
        - name: KAFKA_DEFULT_REPLICATION_FACTOR
          value: "3"
        - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
          value: "true"
        - name: KAFKA_DELETE_TOPIC_ENABLE
          value: "true"
        - name: KAFKA_LISTENERS
          value: "INTERNAL_SASL_PLAINTEXT://0.0.0.0:9092"
        - name: KAFKA_ADVERTISED_LISTENERS
          value: "INTERNAL_SASL_PLAINTEXT://kafka-service:9092"
        - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
          value: "INTERNAL_SASL_PLAINTEXT:SASL_PLAINTEXT,EXTERNAL_SASL_PLAINTEXT:SASL_PLAINTEXT"
        - name: KAFKA_INTER_BROKER_LISTENER_NAME
          value: "INTERNAL_SASL_PLAINTEXT"
        - name: KAFKA_SASL_ENABLED_MECHANISMS
          value: "PLAIN"
        - name: KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL
          value: "PLAIN"
        - name: KAFKA_AUTHORIZER_CLASS_NAME
          value: "kafka.security.authorizer.AclAuthorizer"
        - name: KAFKA_ZOOKEEPER_SET_ACL
          value: "true"
        - name: KAFKA_OPTS
          value: "-Djava.security.auth.login.config=/home/ubuntu/kafkadepl/myconfig/kafka_server_jaas.conf"
        - name: KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND
          value: "true"
      volumes:
      - name: config-volume
        configMap:
          name: kafka-config-map

Verify that pods are up and running

Verify zookeeper and broker

kubectl -n onap get pods | grep 'zookeeper-deployment-1\|broker0'

Verify kafka service

kubectl -n onap get svc | grep kafka-service

If you need to change some variable or anything in a yml file, delete the current deployment, for example:

Delete broker deployment

kubectl -n onap delete deploy kafka-broker0

And after modifying the file create a new deployment as described above.

Run the test

Modify tools/performance/cloud scripts to match the names in your deployments, described in the previous step. Here is a diff file (you may need to adapt it to the current code situation):

tools.diff

Go to tools/performance/cloud and reboot the environment:

Run the script

./reboot-test-environment.sh -v

Now you are ready to run the test.

Without DMaaP Kafka

Conditions

Tests were performed with following configuration:

Messages per producer	Payload size [B]	Interval [ms]
90000	8192	10

Raw results data

Raw results data with screenshots can be found in following files:

Series 1 - results_series_1.zip
Series 2 - results_series_2.zip

To see custom Kafka metrics you may want to change kafka-and-producers.json (located in HV-VES project directory: tools/performance/cloud/grafana/dashboards) to

kafka-and-producers.json

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
 "datasource": "-- Grafana --",
 "enable": true,
 "hide": true,
 "iconColor": "rgba(0, 211, 255, 1)",
 "name": "Annotations & Alerts",
 "type": "dashboard"
 }
    ]
  },
 "editable": true,
 "gnetId": null,
 "graphTooltip": 0,
 "id": 3,
 "links": [],
 "panels": [
    {
      "aliasColors": {},
 "bars": false,
 "dashLength": 10,
 "dashes": false,
 "datasource": null,
 "fill": 0,
 "fillGradient": 0,
 "gridPos": {
        "h": 11,
 "w": 8,
 "x": 0,
 "y": 0
 },
 "hiddenSeries": false,
 "id": 12,
 "legend": {
        "avg": false,
 "current": false,
 "max": false,
 "min": false,
 "show": true,
 "total": false,
 "values": false
 },
 "lines": true,
 "linewidth": 1,
 "nullPointMode": "null",
 "options": {
        "dataLinks": []
      },
 "percentage": false,
 "pointradius": 2,
 "points": false,
 "renderer": "flot",
 "seriesOverrides": [],
 "spaceLength": 10,
 "stack": false,
 "steppedLine": false,
 "targets": [
        {
          "expr": "sum(irate(container_network_transmit_bytes_total{pod=~\".*producer.*\"}[1m]))",
 "interval": "",
 "legendFormat": "Total Transmit Producers",
 "refId": "A"
 },
 {
          "expr": "sum(irate(container_network_transmit_bytes_total{pod=~\".*kafka-broker.*\"}[1m]))",
 "interval": "",
 "legendFormat": "Total Transmit Kafka",
 "refId": "B"
 },
 {
          "expr": "irate(container_network_transmit_bytes_total{pod=~\".*producer.*\"}[1m])",
 "interval": "",
 "legendFormat": "{{pod}}",
 "refId": "C"
 },
 {
          "expr": "irate(container_network_transmit_bytes_total{pod=~\".*kafka-broker.*\"}[1m])",
 "interval": "",
 "legendFormat": "{{pod}}",
 "refId": "D"
 }
      ],
 "thresholds": [],
 "timeFrom": "20m",
 "timeRegions": [],
 "timeShift": null,
 "title": "Network Usage",
 "tooltip": {
        "shared": true,
 "sort": 0,
 "value_type": "individual"
 },
 "type": "graph",
 "xaxis": {
        "buckets": null,
 "mode": "time",
 "name": null,
 "show": true,
 "values": []
      },
 "yaxes": [
        {
          "format": "Bps",
 "label": null,
 "logBase": 1,
 "max": null,
 "min": null,
 "show": true
 },
 {
          "format": "short",
 "label": null,
 "logBase": 1,
 "max": null,
 "min": null,
 "show": true
 }
      ],
 "yaxis": {
        "align": false,
 "alignLevel": null
 }
    },
 {
      "aliasColors": {},
 "bars": false,
 "dashLength": 10,
 "dashes": false,
 "datasource": null,
 "fill": 0,
 "fillGradient": 0,
 "gridPos": {
        "h": 11,
 "w": 8,
 "x": 8,
 "y": 0
 },
 "hiddenSeries": false,
 "id": 22,
 "legend": {
        "avg": false,
 "current": false,
 "max": false,
 "min": false,
 "show": true,
 "total": false,
 "values": false
 },
 "lines": true,
 "linewidth": 1,
 "nullPointMode": "null",
 "options": {
        "dataLinks": []
      },
 "percentage": false,
 "pointradius": 2,
 "points": false,
 "renderer": "flot",
 "seriesOverrides": [],
 "spaceLength": 10,
 "stack": false,
 "steppedLine": false,
 "targets": [
        {
          "expr": "sum(rate(container_network_transmit_bytes_total{pod=~\".*kafka-broker.*|.*hv-ves-coll.*|.*producer.*\"}[1m]))",
 "hide": false,
 "interval": "",
 "legendFormat": "Total transmit",
 "refId": "B"
 }
      ],
 "thresholds": [],
 "timeFrom": "20m",
 "timeRegions": [],
 "timeShift": null,
 "title": "Total network usage (producers + hv-ves + kafka)",
 "tooltip": {
        "shared": true,
 "sort": 0,
 "value_type": "individual"
 },
 "type": "graph",
 "xaxis": {
        "buckets": null,
 "mode": "time",
 "name": null,
 "show": true,
 "values": []
      },
 "yaxes": [
        {
          "format": "Bps",
 "label": null,
 "logBase": 1,
 "max": null,
 "min": null,
 "show": true
 },
 {
          "format": "short",
 "label": null,
 "logBase": 1,
 "max": null,
 "min": null,
 "show": true
 }
      ],
 "yaxis": {
        "align": false,
 "alignLevel": null
 }
    },
 {
      "aliasColors": {},
 "bars": false,
 "dashLength": 10,
 "dashes": false,
 "datasource": null,
 "fill": 0,
 "fillGradient": 0,
 "gridPos": {
        "h": 11,
 "w": 8,
 "x": 16,
 "y": 0
 },
 "hiddenSeries": false,
 "id": 2,
 "legend": {
        "avg": false,
 "current": false,
 "max": false,
 "min": false,
 "show": true,
 "total": false,
 "values": false
 },
 "lines": true,
 "linewidth": 1,
 "nullPointMode": "null",
 "options": {
        "dataLinks": []
      },
 "percentage": false,
 "pointradius": 2,
 "points": false,
 "renderer": "flot",
 "seriesOverrides": [],
 "spaceLength": 10,
 "stack": false,
 "steppedLine": false,
 "targets": [
        {
          "expr": "irate(container_network_receive_bytes_total{pod=~\"dep-dcae-hv-ves-collector.*\"}[1m])",
 "hide": false,
 "interval": "",
 "legendFormat": "receive",
 "refId": "A"
 },
 {
          "expr": "irate(container_network_transmit_bytes_total{pod=~\"dep-dcae-hv-ves-collector.*\"}[1m])",
 "interval": "",
 "legendFormat": "transmit",
 "refId": "B"
 }
      ],
 "thresholds": [],
 "timeFrom": "20m",
 "timeRegions": [],
 "timeShift": null,
 "title": "HV-VES Network Usage",
 "tooltip": {
        "shared": true,
 "sort": 0,
 "value_type": "individual"
 },
 "type": "graph",
 "xaxis": {
        "buckets": null,
 "mode": "time",
 "name": null,
 "show": true,
 "values": []
      },
 "yaxes": [
        {
          "decimals": null,
 "format": "Bps",
 "label": "",
 "logBase": 1,
 "max": null,
 "min": null,
 "show": true
 },
 {
          "format": "short",
 "label": null,
 "logBase": 1,
 "max": null,
 "min": null,
 "show": true
 }
      ],
 "yaxis": {
        "align": false,
 "alignLevel": null
 }
    },
 {
      "aliasColors": {},
 "bars": false,
 "dashLength": 10,
 "dashes": false,
 "datasource": null,
 "fill": 0,
 "fillGradient": 0,
 "gridPos": {
        "h": 12,
 "w": 8,
 "x": 0,
 "y": 11
 },
 "hiddenSeries": false,
 "id": 14,
 "legend": {
        "avg": false,
 "current": false,
 "max": false,
 "min": false,
 "show": true,
 "total": false,
 "values": false
 },
 "lines": true,
 "linewidth": 1,
 "nullPointMode": "null",
 "options": {
        "dataLinks": []
      },
 "percentage": false,
 "pointradius": 2,
 "points": false,
 "renderer": "flot",
 "seriesOverrides": [],
 "spaceLength": 10,
 "stack": false,
 "steppedLine": false,
 "targets": [
        {
          "expr": "sum (rate (container_cpu_usage_seconds_total{pod=~\".*producer.*\"}[1m])) by (pod)",
 "interval": "",
 "legendFormat": "{{pod}}",
 "refId": "A"
 },
 {
          "expr": "sum (rate (container_cpu_usage_seconds_total{pod=~\".*kafka-broker.*\"}[1m])) by (pod)",
 "interval": "",
 "legendFormat": "{{pod}}",
 "refId": "B"
 }
      ],
 "thresholds": [],
 "timeFrom": "20m",
 "timeRegions": [],
 "timeShift": null,
 "title": "CPU usage [cores]",
 "tooltip": {
        "shared": true,
 "sort": 0,
 "value_type": "individual"
 },
 "type": "graph",
 "xaxis": {
        "buckets": null,
 "mode": "time",
 "name": null,
 "show": true,
 "values": []
      },
 "yaxes": [
        {
          "format": "none",
 "label": null,
 "logBase": 1,
 "max": null,
 "min": null,
 "show": true
 },
 {
          "format": "short",
 "label": null,
 "logBase": 1,
 "max": null,
 "min": null,
 "show": true
 }
      ],
 "yaxis": {
        "align": false,
 "alignLevel": null
 }
    },
 {
      "aliasColors": {},
 "bars": false,
 "dashLength": 10,
 "dashes": false,
 "datasource": null,
 "fill": 0,
 "fillGradient": 0,
 "gridPos": {
        "h": 12,
 "w": 8,
 "x": 8,
 "y": 11
 },
 "hiddenSeries": false,
 "id": 18,
 "legend": {
        "avg": false,
 "current": false,
 "max": false,
 "min": false,
 "show": true,
 "total": false,
 "values": false
 },
 "lines": true,
 "linewidth": 1,
 "nullPointMode": "null",
 "options": {
        "dataLinks": []
      },
 "percentage": false,
 "pointradius": 2,
 "points": false,
 "renderer": "flot",
 "seriesOverrides": [],
 "spaceLength": 10,
 "stack": false,
 "steppedLine": false,
 "targets": [
        {
          "expr": "sum(container_memory_working_set_bytes{image!=\"\", pod=~\".*kafka-broker.*\"}) by (pod)",
 "interval": "",
 "legendFormat": "{{pod}}",
 "refId": "A"
 },
 {
          "expr": "sum(container_memory_working_set_bytes{image!=\"\", pod=~\".*producer.*\"}) by (pod)",
 "hide": false,
 "interval": "",
 "legendFormat": "{{pod}}",
 "refId": "B"
 }
      ],
 "thresholds": [],
 "timeFrom": "20m",
 "timeRegions": [],
 "timeShift": null,
 "title": "Memory Usage",
 "tooltip": {
        "shared": true,
 "sort": 0,
 "value_type": "individual"
 },
 "type": "graph",
 "xaxis": {
        "buckets": null,
 "mode": "time",
 "name": null,
 "show": true,
 "values": []
      },
 "yaxes": [
        {
          "format": "decbytes",
 "label": null,
 "logBase": 1,
 "max": null,
 "min": null,
 "show": true
 },
 {
          "format": "short",
 "label": null,
 "logBase": 1,
 "max": null,
 "min": null,
 "show": true
 }
      ],
 "yaxis": {
        "align": false,
 "alignLevel": null
 }
    },
 {
      "datasource": null,
 "gridPos": {
        "h": 6,
 "w": 4,
 "x": 16,
 "y": 11
 },
 "id": 15,
 "options": {
        "colorMode": "value",
 "fieldOptions": {
          "calcs": [
            "lastNotNull"
 ],
 "defaults": {
            "mappings": [],
 "thresholds": {
              "mode": "absolute",
 "steps": [
                {
                  "color": "green",
 "value": null
 },
 {
                  "color": "red",
 "value": 80
 }
              ]
            }
          },
 "overrides": [],
 "values": false
 },
 "graphMode": "area",
 "justifyMode": "auto",
 "orientation": "auto"
 },
 "pluginVersion": "6.7.3",
 "targets": [
        {
          "expr": "sum(container_network_transmit_packets_dropped_total{pod=~\".*producer.*\"})",
 "interval": "",
 "legendFormat": "Packets dropped",
 "refId": "A"
 }
      ],
 "timeFrom": null,
 "timeShift": null,
 "title": "Packets Dropped (producers)",
 "type": "stat"
 },
 {
      "datasource": null,
 "gridPos": {
        "h": 6,
 "w": 4,
 "x": 20,
 "y": 11
 },
 "id": 4,
 "options": {
        "colorMode": "value",
 "fieldOptions": {
          "calcs": [
            "lastNotNull"
 ],
 "defaults": {
            "mappings": [],
 "thresholds": {
              "mode": "absolute",
 "steps": [
                {
                  "color": "green",
 "value": null
 },
 {
                  "color": "red",
 "value": 80
 }
              ]
            }
          },
 "overrides": [],
 "values": false
 },
 "graphMode": "area",
 "justifyMode": "auto",
 "orientation": "auto"
 },
 "pluginVersion": "6.7.3",
 "targets": [
        {
          "expr": "sum(container_network_transmit_packets_dropped_total{pod=~\".*hv-ves-collector.*\"})",
 "interval": "",
 "legendFormat": "Packets dropped",
 "refId": "A"
 }
      ],
 "timeFrom": null,
 "timeShift": null,
 "title": "Packets Dropped (hv-ves)",
 "type": "stat"
 },
 {
      "datasource": null,
 "gridPos": {
        "h": 6,
 "w": 4,
 "x": 16,
 "y": 17
 },
 "id": 17,
 "options": {
        "colorMode": "value",
 "fieldOptions": {
          "calcs": [
            "max"
 ],
 "defaults": {
            "mappings": [],
 "thresholds": {
              "mode": "absolute",
 "steps": [
                {
                  "color": "green",
 "value": null
 }
              ]
            },
 "unit": "bytes"
 },
 "overrides": [],
 "values": false
 },
 "graphMode": "none",
 "justifyMode": "auto",
 "orientation": "auto"
 },
 "pluginVersion": "6.7.3",
 "targets": [
        {
          "expr": "sum(container_network_transmit_bytes_total{pod=~\".*producer.*\"})",
 "interval": "",
 "legendFormat": "",
 "refId": "A"
 }
      ],
 "timeFrom": null,
 "timeShift": null,
 "title": "Sent (producers)",
 "type": "stat"
 },
 {
      "cacheTimeout": null,
 "datasource": null,
 "gridPos": {
        "h": 6,
 "w": 4,
 "x": 20,
 "y": 17
 },
 "id": 16,
 "links": [],
 "options": {
        "colorMode": "value",
 "fieldOptions": {
          "calcs": [
            "max"
 ],
 "defaults": {
            "mappings": [
              {
                "id": 0,
 "op": "=",
 "text": "N/A",
 "type": 1,
 "value": "null"
 }
            ],
 "nullValueMode": "connected",
 "thresholds": {
              "mode": "absolute",
 "steps": [
                {
                  "color": "green",
 "value": null
 }
              ]
            },
 "unit": "bytes"
 },
 "overrides": [],
 "values": false
 },
 "graphMode": "none",
 "justifyMode": "auto",
 "orientation": "horizontal"
 },
 "pluginVersion": "6.7.3",
 "targets": [
        {
          "expr": "container_network_receive_bytes_total{pod=~\".*hv-ves-collector.*\"}",
 "interval": "",
 "legendFormat": "Packets dropped",
 "refId": "A"
 }
      ],
 "timeFrom": null,
 "timeShift": null,
 "title": "Received (hv-ves)",
 "type": "stat"
 }
  ],
 "refresh": "5s",
 "schemaVersion": 22,
 "style": "dark",
 "tags": [],
 "templating": {
    "list": []
  },
 "time": {
    "from": "now-1h",
 "to": "now"
 },
 "timepicker": {
    "refresh_intervals": [
      "5s",
 "10s",
 "30s",
 "1m",
 "5m",
 "15m",
 "30m",
 "1h",
 "2h",
 "1d"
 ]
  },
 "timezone": "",
 "title": "Kafka and Producers Statistics",
 "uid": "-28uHcjZk",
 "variables": {
    "list": []
  },
 "version": 1
}

Test results - series 1

Click here to see results...

Below tables show the test results across a wide range of containers' number.

NUMBER OF PRODUCERS	TOTAL MESSAGES PROCESSED	AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]	AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]	PEAK INCOMING DATA RATE [MB/s]	PEAK PROCESSING MESSAGE QUEUE SIZE	PEAK CPU LOAD [%]	PEAK MEMORY USAGE [GB]
2	180000	0.024	1.5	1.6	3	3.5	0.37
4	360000	0.020	1.7	3.2	3	5.7	0.37
6	540000	0.02	2.6	4.8	24	6.0	0.37
8	720000	0.02	2.8	6.4	14	8.5	0.37
10	900000	0.02	7.3	8.1	552	8.5	0.38
12	10800000	0.02	78	9.7	1077	7.5	0.41
14	12600000	6140	19000	13.0	10630	9.8	0.99

Test results - series 2

Click here to see results...

Below tables show the test results across a wide range of containers' number.

NUMBER OF PRODUCERS	TOTAL MESSAGES PROCESSED	AVERAGE PROCESSING TIME IN HV-VES WITHOUT ROUTING [ms]	AVERAGE LATENCY TO HV-VES OUTPUT WITH ROUTING [ms]	PEAK INCOMING DATA RATE [MB/s]	PEAK PROCESSING MESSAGE QUEUE SIZE	PEAK CPU LOAD [%]	PEAK MEMORY USAGE [GB]
2	180000	0.025	1.7	1.6	3	3.4	0.37
4	360000	0.021	2.1	3.2	14	4.9	0.37
6	540000	0.02	2.5	4.8	60	6.2	0.38
8	720000	0.02	2.9	6.4	24	7.4	0.37
10	900000	0.018	5.5	8.0	201	8.1	0.36
12	10800000	0.019	141.7	9.7	1716	9.1	0.44
14	12600000	31	7568	16.0	5778	8.6	0.50

HV-VES Performance Test

Test architecture

Environment and Resources

Network Performance

Pod measurement method

Results of performed tests

Test Setup

Preconditions

HV-VES Performance test results

With dmaap Kafka

Conditions

Raw results data

Test Results - series 1

Test Results - series 2

No DMaaP Kafka SetUp

Install Kafka Docker on Kubernetes

Create config maps

Create deployments

Verify that pods are up and running

Run the test

Without DMaaP Kafka

Conditions

Raw results data

Test results - series 1

Test results - series 2