Stopping one scenario in k6 test run is not supported

• use of methods such as fail() or threshold abortOnFail aborts the current VU iteration not only one scenario

1. we can not stop just one scenario and continue on to the next scenarios

2. we can abort the whole test run only

Each VU has its own states and cannot access other VUs states (no shared mutable state) as it runs on its own JS runtime

1. if we wanted to control the number of errors for the whole k6 test run (all scenarios) i.e. only 10 errors for the whole test run and abort, we cannot create a global variable to control this.

Environmental variables are read only for each VU

1. we cannot use env variables as a shared variable for each VU (as a work around for limitation #2)

Threshold metrics are only evaluated after the k6 test run

1. use of metric such as http_req_failed or a custom metric cannot be used to abort the whole k6 test run or scenario

Use console.error('Logging message') to log every errors when response is != 200

1. Can modify to control what part of response is logged

   1. response.url

   2. response.status

   3. response.body

1. used on its own will overload logs if all errors are logged

Use console.error() + log errors to a file

Open

1. Only errors are saved into a file which means if there are no errors, file cant be as big

   1. Options reference | Grafana k6 documentation

2. Log file can be discarded next day in our Jenkins workspace or when we choose to discard it

   1. log has timestamp which allows us to relate which build it happened on

1. Possibility of having huge file for multiple errors if not controlled

2. Does not support multiple log output files

   1. info level logs will be missed if specified only errors

   2. see example where multiple log output files is used to separate info and error logs

Open

Use fail() when response is not OK

Sample result

1. Marks the current iteration failed

2. Useful for non rest call

   1. a failed rest call automatically updates http_req_failed

3. Can log error message with it

1. Aborts only the current iteration for the VU, not the entire scenario.

2. the VU is not stopped and continues onto the next iteration after

Limiting error log counts per VUs

i.e. each VU iteration

1. Controlled logs, won’t create a huge file filled with errors

2. if ‘do nothing’ when error limit is reached, continues other iterations and scenarios without adding to the metrics (stops VU from performing any actions)

3. if used ‘fail()’ then the threshold "http_req_failed" is still updated on the next iterations

1. if ‘do nothing’ the number of iterationsCompleted on the k6 instance summary will include the iterations where no actual rest call was made

Use exec.test.abort() to abort ALL test scenarios (all k6 tests) and do k6 test teardown() immediately

Below example shows that for an error limit of 5 per VU - when one VU reach its limit, whole k6 test run is cancelled. The action is logged at an error level and detail of which test caused it.

Open

1. Can be used when error limit on one VU is reached

2. Can abort with a message as well

3. Can be used to control on specific error types , i.e.

   1. if response code is 400 then we can do option #4

   2. if response code is 500 then abort all test

1. Error limit can only be tracked per VU which means that one VU reaching error limit can abort ALL test

Would be great if a shared mutable variable between VUs is available

Combination of the following:

1. console.error() / console.warning()

2. limiting logs per VUs based on status code i.e.

   1. errorsLimit = 3

   2. warningsLimit = 10

3. if errors limit hit exec.test.abort()

4. if warning limit hit do nothing, don’t log next warnings

1. we control the tolerated errors and warnings

2. does not abort all test due to warnings

3. we can see which test hit an error limit

Registration of CM-handles

De-registration of CM-handles

CM-handle ID search with No filter

CM-handle ID search with Module filter

CM-handle ID search with Property filter

CM-handle ID search with Cps Path filter

CM-handle ID search with Trust Level filter

CM-handle search with No filter

CM-handle search with Module filter

CM-handle search with Property filter

CM-handle search with Cps Path filter

NCMP overhead for Synchronous single CM-handle pass-through read with alternate id

NCMP overhead for Synchronous single CM-handle pass-through write with alternate id

Legacy batch read operation

Registration of CM-handles

De-registration of CM-handles

CM-handle ID search with No filter

CM-handle ID search with Module filter

CM-handle ID search with Property filter

CM-handle ID search with Cps Path filter

CM-handle ID search with Trust Level filter

CM-handle search with No filter

CM-handle search with Module filter

CM-handle search with Property filter

CM-handle search with Cps Path filter

NCMP overhead for Synchronous single CM-handle pass-through read with alternate id

NCMP overhead for Synchronous single CM-handle pass-through write with alternate id

Legacy batch produce operation