The purpose of this is to measure the load that a complete ONAP platform puts on a Kubernetes multi-node cluster orchestrated and deployed by a Rancher instance. Due to the nature of the deployment in an OOM-based ONAP platform which uses HELM for deployment, it is useful to have a standard set of initial resource requirements so that we know how much should be initially allocated for the Kubernetes node availability so as to not bring the cluster to a failure state due to resource availability concerns.
Rancher’s default Kubernetes template provides three components that have been used here to measure metrics. All of them run on pods within the Kubernetes cluster so the cluster itself can control it:
- Heapster: This is a component that collects metrics and events from all the Kubernetes nodes. It is designed to be modular in a way that multiple backends are supported, such as InfluxDB (used here), Kakfa, ElasticSearch, Google Cloud Monitoring, etc.
- InfluxDB: This is the backend component that Heapster uses to store the collected metric data (i.e. Heapster’s Sink). It is a time-series database that can be queried by many visualization GUIs, including Grafana and Kibana.
- Grafana: This is the frontend component that queries InfluxDB to obtain the metrics collected by Heapster. It can contain many dashboards broken up into panels that can be customized to show all types of data provided by the backend data source.
In order to run these measurements, the following was applied:
- An 11-node Kubernetes cluster (as currently being tested by the Integration team) is suggested to deploy a complete ONAP platform
- Docker version 17.03.2
- Kubernetes version 1.8.10
- A single Rancher instance (v1.6.14)
- Docker version 17.03.2
- Docker version 17.03.2
- The VM flavor for each of these VMs is OpenStack standard flavor “m1.xlarge” which is equivalent to 8 vCPUs, 16 GB RAM, 160 GB HDD
- ONAP "Beijing" Release (Complete, with all components enabled as part of the HELM chart deployment). Gerrit branch "BEIJING"
By using the components to measure resource usage (Heapster), store into a backend component (InfluxDB), and visualizing the data into a frontend (Grafana), the following relevant metrics were collected.
RESOURCE USAGE CHART
These metrics are from Kubernetes resources that do not stop after an amount of time (i.e. Kubernetes Jobs whose container exits as soon as the job request is completed), so focus is on Kubernetes Pods.
NOTE: All the numbers below are assuming that there is no additional actions on the ONAP applications, other than the resources running at an idle state.
- For the complete ONAP instance usage: this is measuring the mean value of overall kubernetes cluster usage after the sum of all the Kubernetes nodes resource usage reports. This takes into account the resource usage of a complete ONAP instance (with all its HELM Charts enabled and deployed).
| RESOURCE | vCPUs | FILESYSTEM | MEMORY (RAM) | NETWORK |
|---|---|---|---|---|
Complete ONAP Instance (All 25 HELM Chart Components - Beijing Release) | Min: 556 mcores Max: 23.46 cores Avg: 8.87 cores | Min: 2.84 GB Max: 25.22 GB Avg: 17.61 GB | Min: 8.5 GiB Max: 130.1 GB Avg: 118.7 GB | Min (Tx / Rx): 293 kBps / 282 kBps Max (Tx / Rx): 2.09 MBps / 9.73 MBps Avg (Tx / Rx): 371 kBps / 341 kBps |
OVERALL CLUSTER - CPU USAGE
OVERALL CLUSTER - FILESYSTEM USAGE
OVERALL CLUSTER - MEMORY (RAM) USAGE
OVERALL CLUSTER - NETWORK USAGE
- For measurement of aggregated pod usage (in other rows)
| RESOURCE | vCPUs (mcores) | FILESYSTEM (KiB) | RAM MEMORY (GiB) | NETWORK (kBps) |
|---|---|---|---|---|
HELM AAF - Overall Pod Usage 14 pods (13 in Running state / 1 in Completed state) | Min: 7 Max: 22 Avg: 11 | Min: 624 Max: 624 Avg: 624 | Min: 10.23 Max: 10.68 Avg: 10.43 | Min (Tx / Rx): 1.84 / 2.35 Max (Tx / Rx): 2.22 / 2.84 Avg (Tx / Rx): 2.00 / 2.52 |
HELM AAI - Overall Pod Usage 16 pods (15 in Running state / 1 in Completed state) | Min: 1068 Max: 1635 Avg: 1191 | Min: 732 Max: 732 Avg: 732 | Min: 10.52 Max: 11.39 Avg: 10.89 | Min (Tx / Rx): 26.64 / 36.31 Max (Tx / Rx): 50.60 / 45.78 Avg (Tx / Rx): 39.79 / 40.69 |
HELM APPC - Overall Pod Usage 4 pods (4 in Running state / 0 in Completed state) | Min: 14 Max: 25 Avg: 17 | Min: 192 Max: 192 Avg: 192 | Min: 2.71 Max: 2.81 Avg: 2.74 | Min (Tx / Rx): 0.249 / 0.321 Max (Tx / Rx): 0.335 / 0.394 Avg (Tx / Rx): 0.281 / 0.357 |
| HELM CLAMP - Overall Pod Usage | Min: 9 Max: 30 Avg: 13 | Min: 240 Max: 240 Avg: 240 | Min: 4.26 Max: 4.36 Avg: 4.3 | Min (Tx / Rx): 3.82 / 2.79 Max (Tx / Rx): 4.01 / 5.07 Avg (Tx / Rx): 3.92 / 4.98 |
| HELM CLI - Overall Pod Usage | Min: 0 Max: 0 Avg: 0 | Min: 48 Max: 48 Avg: 48 | Min: 6.637 MiB Max: 6.641 MiB Avg: 6.639 MiB | Min (Tx / Rx): 0.027 / 0.053 Max (Tx / Rx): 0.045 / 0.064 Avg (Tx / Rx): 0.029 / 0.055 |
| HELM CONSUL - Overall Pod Usage | Min: 408 Max: 514 Avg: 488 | Min: 192 Max: 192 Avg: 192 | Min: 0.507 Max: 2.49 Avg: 1.14 | Min (Tx / Rx): 44.33 / 758 Max (Tx / Rx): 51.03 / 856 Avg (Tx / Rx): 48.01 / 834 |
| HELM DCAEGEN2 - Overall Pod Usage | Min: (26) + (87) = Max: (40) + (551) = Avg: (30) + (251) = | Min: (480) + (480) = Max: (480) + (480) = Avg: (480) + (480) = | Min: (2.06) + (4.70) = Max: (2.11) + (6.38) = Avg: (2.08) + (5.50) = | Min (Tx / Rx): (32.12) + (0.951) = / (33.03) + (1.63) = Max (Tx / Rx): (34.95) + (1.40) = / (36.02) + (2.14) = Avg (Tx / Rx): (33.50) + (1.09) = / (34.49) + (1.80) = |
| HELM DMAAP - Overall Pod Usage | Min: 126 Max: 178 Avg: 155 | Min: 288 Max: 288 Avg: 288 | Min: 2.36 Max: 2.53 Avg: 2.44 | Min (Tx / Rx): 163.30 / 180.25 Max (Tx / Rx): 195.81 / 212.87 Avg (Tx / Rx): 188.81 / 205.66 |
| HELM ESR - Overall Pod Usage | Min: 1 Max: 4 Avg: 2 | Min: 96 Max: 96 Avg: 96 | Min: 545.6 MiB Max: 570.9 MiB Avg: 555 MiB | Min (Tx / Rx): 0.062 / 0.106 Max (Tx / Rx): 0.076 / 0.120 Avg (Tx / Rx): 0.069 / 0.109 |
| HELM LOG - Overall Pod Usage | Min: 232 Max: 26323 Avg: 3330 | Min: 624 Max: 624 Avg: 624 | Min: 12.89 Max: 18.62 Avg: 16.30 | Min (Tx / Rx): / 31.38 Max (Tx / Rx): / 97.80 Avg (Tx / Rx): / 44.73 |
| HELM MSB - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM MULTICLOUD - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM NBI - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM OOF - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM POLICY - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM PORTAL - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM ROBOT - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM SDC - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM SDNC - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM SNIRO-EMULATOR - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM SO - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM UUI - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM VFC - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM VID - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |
| HELM VNFSDK - Overall Pod Usage | Min: mcores Max: Avg: | Min: Max: Avg: | Min: Max: Avg: | Min (Tx / Rx): Max (Tx / Rx): Avg (Tx / Rx): |