Versions Compared

Old Version 29

changes.mady.by.user Halil Cakal

Saved on

compared with

New Version 36

changes.mady.by.user Halil Cakal

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Issue

Notes 

Decision

1

Agree on endurance performance KPI

The aim of the study is to early detection of bugs similar to

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-2430
or any memory leakage in the future.

Thus, we should spot the specific function/s currently used in k6 to be improved. These function/s will be used in the Endurance test suit.

Actions:

  1. Daniel Hanrahan and Lee Anjella Macabuhay can shed light on this item since they were resolving the bug.

  2. After identifying the test function/s, what can we functionally do to improve them?
    The reproduction of the bug includes two more data operations that are not available in the current k6 suit:

    1. Passthrough-write (PATCH)

    2. Passthrough (DELETE) / Delete Resource Data

  3. Regarding parallelism, how many VUs should be assigned to these tests?
    The suggested number of VUs with 'constant-vus' executor:

    1. Passthrough-read (READ): 4

    2. Passthrough-write (POST): 4

    3. Passthrough-write (PATCH) : 4

    4. Passthrough (DELETE) / Delete Resource Data: 4

CPS need to run all the test cases according to FS not according to what was tested in cps-2430 (Seeing they ran test that was outside of FS). Less focus on the KPI for this endurance run. Kolawole Adebisi-Adeolokun Halil Cakal

2

Agree on the test environment

Blocked by

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-2463

  1. if a new test server can be provided, then no need to have a secondary docker-compose deployment descriptor (port and container name changes)

  2. if no new server can be provided, then a secondary docker-compose deployment descriptor will be necessary

    1. There will be 3 different pipelines will be running on the physical test server:

      1. CPS performance tests (groovy-based)

      2. NCMP performance tests (k6-based)

      3. NCMP Endurance tests (k6-based, long-running)

  3. Team Kraken could set up test envi for us

  1. No parallelism. A conversation with Kraken to discuss option #3 Halil Cakal

  2. The discussion was made with Team Kraken and decided to have two docker-compose deployments on the test server. Daniel Hanrahan Halil Cakal Toine Siebelink

    1. The PoC has been done on Nordix.
      PoC using two docker-compose using an environment file

  3. The number of available executors should be sufficient to run a possible three jobs on the physical performance test server. Daniel Hanrahan Toine Siebelink Kolawole Adebisi-Adeolokun Halil Cakal

3

Visualization of memory trend

The most convenient way to represent the memory usage trend in a GUI is Grafana.

  1. Grafana

  2. GNUPlot (3pp library current in the use of k6 and Groovy test Visualization)

    1. If Gnuplot is to visualize memory usage, which parameters, logs, or output should be input?

    2. Issue #4 (external access) will no longer be valid. Since Jenkin’s HTML publisher can do it.

Agreed to use Grafana majorly because it allows storage for longer periods sticking with our A/C Kolawole Adebisi-Adeolokun Halil Cakal

The same decision was made by Daniel Hanrahan and Halil Cakal during the daily stand-up.

Note: A PoC for the GNUPlot option has been made, please see the relevant Jira ticket details. Daniel Hanrahan Halil Cakal Toine Siebelink

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-2466

4

Grafana access externally (If Grafana is selected at #3 then this issue should be discussed, otherwise ignore)

  1. Make local GUI link accessible externally

  2. Give server access to the CPS team

Team Kraken support is needed either way.

Since Grafana is the preferred option, access issues shall be discussed with team Kraken Halil Cakal (consider Jenkins plug-in option as well)7

5

Permanent storage (DB) for Grafana (If Grafana is selected at #3 then this issue should be discussed, otherwise, ignore)

Investigate the solution strategy for the storage e.g. a permanent service 7/24 running …

Halil Cakal to investigate how to tune the volume

...

Suggested Tasks

k6 performance suitAdd new k6 test suit

Description

Jira

Add new

test

profile ‘Endurance’

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-2464

Add new Jenkins job to run endurance test

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-2465

Add Grafana support to visualize memory usage pattern

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-2466

Two docker-compose deployments simultaneously

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-2463

Agree and Define new ‘Suite’ (js)

Jira Legacy
serverSystem Jira
serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
keyCPS-2493

Solution Proposal

...

For the sake of simplicity, the solution would be straightforward if a new test server is provided.

The functional/test-suit improvements in the current k6 performance tests if required

...

Agree and Define new ‘Suite’ (js)

  1. The existing K6 performance tests are enough to detect any memory leakage so that no new K6 test will be added. Daniel Hanrahan Halil Cakal
    There should be a new test suit Toine Siebelink Kolawole Adebisi-Adeolokun Halil Cakal

    Jira Legacy
    serverSystem Jira
    serverId4733707d-2057-3a0f-ae5e-4fd8aff50176
    keyCPS-2493

    HOW: Daniel and I decided to define a new test suite (endurance). Daniel Hanrahan Halil Cakal
    Have one ncmp-test-runner.js with different configs: kpi.json and endurance.json, moving the scenarios and thresholds settings into the json configs
    The endurance test should run all tests in parallel. Also, the same scenarios, executor types, and VUs should be the same in the Endurance suit except for the legacy_batch_consume_scenario. The executor type for this scenario should be changed to constant-arrival-rate with 1 req/second. Toine Siebelink Daniel Hanrahan Kolawole Adebisi-Adeolokun Halil Cakal
    As an example:

    Code Block
    languagejson
    {
  "hosts": {
    "ncmpBaseUrl": "http://localhost:8883",
    "dmiStubUrl": "http://ncmp-dmi-plugin-demo-and-csit-stub:8092",
    "kafkaBootstrapServer": "localhost:9092"
  },
  "kafka": {
    "legacyBatchTopic": "legacy_batch_topic"
  },
  "scenarios": {
    "passthrough_read_scenario": {
      "executor": "constant-vus",
      "exec": "passthroughReadScenario",
      "vus": 2,
      "duration": "15m"
    }
  },
  "thresholds": {
    "http_req_failed": ["rate == 0"],
    "cmhandles_created_per_second": ["avg >= 22"],
    "cmhandles_deleted_per_second": ["avg >= 22"],
    "ncmp_overhead_passthrough_read": ["avg <= 40"]
  }
}
    All tests should be run in parallel in the Endurance test. Also, the same scenarios, executor types, and VUs should be the same in the Endurance suit except for legacy_batch_consume_scenario. The executor type for legacy_batch_consume_scenario should be changed to constant-arrival-rate with 1 req/second. Toine Siebelink Daniel Hanrahan Kolawole Adebisi-Adeolokun Halil Cakal 
The current KPI and ENDURANCE (proposed) test parameters (Last updated  )
1
K P I
E N D U R A N C E
2
Test Stages
Scenario Name
Unit
Executor Type
VUs
Duration
Scenario Name
VUS
Executor Type
Duration
3
setup
create_cm_handles
CM-handles/second
N/A
N/A
20m
create_cm_handles
N/A
N/A
20m
4
scenario
passthrough_read_scenario
overhead
constant-vus
2
15m
passthrough_read_scenario
2
constant-vus
1h
5
passthrough_read_alt_id_scenario
overhead
constant-vus
2
passthrough_read_alt_id_scenario
2
constant-vus
6
passthrough_write_scenario
overhead
constant-vus
2
passthrough_write_scenario
2
constant-vus
7
passthrough_write_alt_id_scenario
overhead
constant-vus
2
passthrough_write_alt_id_scenario
2
constant-vus
8
cm_handle_id_search_nofilter_scenario
milliseconds
constant-vus
1
cm_handle_id_search_nofilter_scenario
1
constant-vus
9
 cm_handle_id_search_module_scenario
milliseconds
constant-vus
1
 cm_handle_id_search_module_scenario
1
constant-vus
10
cm_handle_id_search_property_scenario
milliseconds
constant-vus
1
cm_handle_id_search_property_scenario
1
constant-vus
11
cm_handle_id_search_cpspath_scenario
milliseconds
constant-vus
1
cm_handle_id_search_cpspath_scenario
1
constant-vus
12
cm_handle_id_search_trustlevel_scenario
milliseconds
constant-vus
1
cm_handle_id_search_trustlevel_scenario
1
constant-vus
13
cm_handle_search_nofilter_scenario
milliseconds
constant-vus
1
cm_handle_search_nofilter_scenario
1
constant-vus
14
cm_handle_search_module_scenario
milliseconds
constant-vus
1
cm_handle_search_module_scenario
1
constant-vus
15
cm_handle_search_property_scenario
milliseconds
constant-vus
1
cm_handle_search_property_scenario
1
constant-vus
16
cm_handle_search_cpspath_scenario
milliseconds
constant-vus
1
cm_handle_search_cpspath_scenario
1
constant-vus
17
cm_handle_search_trustlevel_scenario
milliseconds
constant-vus
1
cm_handle_search_trustlevel_scenario
1
constant-vus
18
legacy_batch_produce_scenario
milliseconds
shared-iterations
2
N/A
legacy_batch_produce_scenario
1 (1 req. sec)
constant-arrival-rate
19
legacy_batch_consume_scenario
events/second
per-vu-iterations
1
20
teardown
delete_cm_handles
CM-handles/second
N/A
N/A
20m
delete_cm_handles
N/A
10m
...
Visualizing the ENDURANCE test results 
As mentioned in issues/decisions, there are two alternative ways of representing memory trends: Grafana and 
...
GnuPlot.
  • Grafana support (without Grafana db) has already been in place in our docker-compose deployment descriptor and can be activated by --profile monitoring. 
    Then it will be accessible by http://localhost:3001/login
...
Grafana

EST has its own Prometheus and Grafana (externally accessible, no need to install Globalprotect) and it can be configured to show cps-and-ncmp memory trends.
Link to EST Grafana: https://monitoring.nordix.org/login

Through prometheus.yml (of EST), a new scrape_configs for the cps-and-ncmp microservice can be added.
 Code Block
languageyaml
scrape_configs:
- job_name: 'cps-and-ncmp'
  metrics_path: '/actuator/prometheus'
  scrape_interval: 5s
  static_configs:
    - targets:
      - 'cps-and-ncmp:8080' // replace by <physical-server-ip:port>
Also, the dashboard provider and dashboard config (jvm-micrometer-dashboard.json) can be added.
 Code Block
languageyaml
providers:
  - name: default
    orgId: 1
    type: file
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true
Then, the trend of G1 Old Gen space can be observed as seen below:
...
Permanent Storage Alternatives
Prometheus
Configuring the Prometheus with a persistent volume to retain data is possible.
This is an example service config for Prometheus:
 Code Block
languagejson
  prometheus:
    container_name: ${PROMETHEUS_CONTAINER_NAME:-prometheus}
    image: prom/prometheus:latest
    ports:
      - ${PROMETHEUS_PORT:-9090}:9090
    restart: always
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    profiles:
      - monitoring
      
volumes:
  prometheus_data:
    driver: local
GNUPlot
GnuPlot can also draw a plot for only G1 Old Gen space
...
Getting support from Team Kraken for external link/server access, if we decide to visualize memory trends by Grafana.
...
.