How to re-run a test on a daily platform?

Sometimes you need to troubleshoot the errors reported during a daily chain

You must first connect to the daily lab.

The controller can be reached over SSH master.onap.eu, istanbul.onap.eu, master-weekly.onap.eu.(you SSH key must obviously be referenced see How to add keys to access integration labs?).

Once connected, it is is a "usual ONAP" lab, so you can run your test as in any ONAP platform.

All the configurations of the tests executed during the daily chains can be found under /tmp/xtesting

tests

debian@control01-onap-master:~$ cd /tmp/xtesting
debian@control01-onap-master:/tmp/xtesting$ tree -L 2
.
├── config
├── infrastructure-healthcheck
│   ├── k8s
│   └── k8s-teardown
├── security
│   ├── jdpw_ports
│   ├── kube_hunter
│   ├── nonssl_endpoints
│   ├── root_pods
│   ├── unlimitted_pods
│   └── versions
├── smoke-usecases
│   ├── basic_cds
│   ├── basic_clamp
│   ├── basic_cnf
│   ├── basic_cnf_macro
│   ├── basic_network
│   ├── basic_onboard
│   ├── basic_vm
│   ├── basic_vm_macro
│   ├── cds_resource_resolution
│   └── pnf_macro
├── xtesting-healthcheck
│   ├── core
│   ├── cps-healthcheck
│   ├── full
│   ├── healthdist
│   └── postinstall
└── xtesting-smoke-usecases-robot
    ├── cmpv2
    ├── dcaemod
    ├── hv-ves
    ├── pnf-registrate
    └── ves-collector

33 directories, 1 file

The tests are run using 2 main mechanisms:

k8s job in the onap namespace
docker executed from the controller node

How to replay a kubernetes job?

k8s test

debian@control01-onap-master:~$ kubectl get job -n onap |grep integration
integration-onap-cmpv2                  0/1           9d         9d
integration-onap-core                   1/1           2m6s       9d
integration-onap-cps-healthcheck        1/1           32s        9d
integration-onap-dcaemod                0/1           9d         9d
integration-onap-full                   1/1           2m50s      9d
integration-onap-healthdist             1/1           5m58s      9d
integration-onap-hv-ves                 1/1           4m18s      9d
integration-onap-internal-check-certs   0/1           9d         9d
integration-onap-pnf-registrate         1/1           7m36s      9d
integration-onap-postinstall            1/1           101s       9d
integration-onap-ves-collector          1/1           11m        9d

# Delete the job of the test you want to replay
debian@control01-onap-master:~$ kubectl delete job -n onap integration-onap-cps-healthcheck
job.batch "integration-onap-cps-healthcheck" deleted

# Recreate the job (it will automatically relaunch the test pod)
debian@control01-onap-master:~$ cd oom/master/
debian@control01-onap-master:~/oom/master$ ls
deployment.yaml         healthcheck-cps-healthcheck.yaml  healthcheck-healthdist.yaml            healthcheck-pnf-registrate.yaml  onap-overrides.yaml
healthcheck-cmpv2.yaml  healthcheck-dcaemod.yaml          healthcheck-hv-ves.yaml                healthcheck-postinstall.yaml     xtesting-healthcheck
healthcheck-core.yaml   healthcheck-full.yaml             healthcheck-internal-check-certs.yaml  healthcheck-ves-collector.yaml   xtesting-smoke-usecases-robot
kubectl apply -f healthcheck-cps-healthcheck.yaml --validate=false

# Then look at the logs

kubectl logs -n onap integration-onap-cps-healthcheck--1-m7tbt -f
2022-03-07 13:27:41,030 - xtesting.ci.run_tests - INFO - Deployment description:

+-------------------------+----------------------------------------------------------+
|         ENV VAR         |                          VALUE                           |
+-------------------------+----------------------------------------------------------+
|         CI_LOOP         |                          daily                           |
|          DEBUG          |                          false                           |
|     DEPLOY_SCENARIO     |                   onap-nofeature-noha                    |
|      INSTALLER_TYPE     |                           oom                            |
|        BUILD_TAG        |      gitlab_ci-functest-kubespray-baremetal-daily-       |
|                         |                  master-479785435-onap                   |
|        NODE_NAME        |             onap_weekly_pod4_master-ONAP-oom             |
|       TEST_DB_URL       |     http://testresults.opnfv.org/onap/api/v1/results     |
|     TEST_DB_EXT_URL     |                                                          |
|     S3_ENDPOINT_URL     |                                                          |
|        S3_DST_URL       |                                                          |
|       HTTP_DST_URL      |                                                          |
+-------------------------+----------------------------------------------------------+

2022-03-07 13:27:41,061 - xtesting.ci.run_tests - INFO - Loading test case 'cps-healthcheck'...
2022-03-07 13:27:41,217 - xtesting.ci.run_tests - INFO - Running test case 'cps-healthcheck'...
2022-03-07 13:27:41,321 - xtesting.core.robotframework - INFO - 
==============================================================================
Actuator :: CPS - Actuator endpoints                                          
==============================================================================
Test CPS Enhanced Healthcheck :: Runs CPS Health Check. It will ch... | PASS |
------------------------------------------------------------------------------
Actuator :: CPS - Actuator endpoints                                  | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output:  /var/lib/xtesting/results/cps-healthcheck/output.xml

2022-03-07 13:27:41,326 - xtesting.core.robotframework - INFO - Results were successfully parsed
2022-03-07 13:27:41,395 - xtesting.core.robotframework - INFO - Results were successfully generated
2022-03-07 13:27:41,395 - xtesting.ci.run_tests - INFO - Test result:

+-------------------------+---------------------+------------------+----------------+
|        TEST CASE        |       PROJECT       |     DURATION     |     RESULT     |
+-------------------------+---------------------+------------------+----------------+
|     cps-healthcheck     |     integration     |      00:00       |      PASS      |
+-------------------------+---------------------+------------------+----------------+

2022-03-07 13:27:41,713 - xtesting.core.testcase - INFO - The results were successfully pushed to DB: 

http://testresults.opnfv.org/onap/api/v1/results/6226084d6c44f5001087e92d

2022-03-07 13:27:41,714 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_OK

# Note by  default it will push the results to the test DB..if you do not want, edit the manifest and remove the -- report option
            -   args:
                - --test
                - cps-healthcheck
                - --report

How to replay a docker based test?

Several tests are executed by running a docker file on the controller. It is used for

infrastructure tests
security tests
pythonsdk-test tests

docker tests

# Select the test you want to replay (e.g. basic_cds)
debian@control01-onap-master:~$ cd /tmp/xtesting//smoke-usecases/basic_cds/

# run the docker
debian@control01-onap-master:/tmp/xtesting/smoke-usecases/basic_cds$  cat /etc/hosts |grep onap.org && docker run -it --env-file env -v /home/debian/.kube/config:/root/.kube/config -v /home/debian/.config/openstack/clouds.yaml:/root/.config/openstack/clouds.yaml -v /tmp/xtesting/config:/usr/lib/python3.8/site-packages/onaptests/templates/artifacts/config -v /tmp/test:/var/lib/xtesting/results nexus3.onap.org:10003/onap/xtesting-smoke-usecases-pythonsdk:master bash

# you shall be in the docker
# for infrastructure and security tests you can directly run the test
# for pythonsdk you must copy/paste the content of the /etc/hosts (you could also use parameter in the docker exec command if you want), as onap url are not publicaly routable
# 
bash-5.1# vi /etc/hosts
# copy the content displayed by the command cat /etc/hosts |grep onap.org
bash-5.1# cat /etc/hosts
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
ff00::0	ip6-mcastprefix
ff02::1	ip6-allnodes
ff02::2	ip6-allrouters
172.17.0.2	265c7c83aaf0
10.253.0.217 portal.api.simpledemo.onap.org
10.253.0.217 vid.api.simpledemo.onap.org
10.253.0.217 sdc.api.fe.simpledemo.onap.org
10.253.0.217 sdc.api.be.simpledemo.onap.org
10.253.0.217 portal-sdk.simpledemo.onap.org
10.253.0.217 policy.api.simpledemo.onap.org
10.253.0.217 aai.api.sparky.simpledemo.onap.org
10.253.0.217 cli.api.simpledemo.onap.org
10.253.0.217 msb.api.simpledemo.onap.org
10.253.0.217 so.api.simpledemo.onap.org
10.253.0.217 appc.api.simpledemo.onap.org
10.253.0.217 sdnc.api.simpledemo.onap.org
10.253.0.217 nbi.api.simpledemo.onap.org
10.253.0.217 consul.api.simpledemo.onap.org
10.253.0.217 kibana.api.simpledemo.onap.org
10.253.0.217 mr.api.simpledemo.onap.org
10.253.0.217 uui.api.simpledemo.onap.org
10.253.0.217 aaf.api.simpledemo.onap.org
10.253.0.217 clamp.api.simpledemo.onap.org
10.253.0.217 robot.api.simpledemo.onap.org
10.253.0.217 dcae.api.simpledemo.onap.org
10.253.0.217 sdc.workflow.plugin.simpledemo.onap.org
10.253.0.217 sdc.dcae.plugin.simpledemo.onap.org

# execute the xtesting command
bash-5.1# run_tests -t basic_cds
2022-03-07 13:46:13,968 - xtesting.ci.run_tests - INFO - Deployment description:

+-------------------------+----------------------------------------------------------+
|         ENV VAR         |                          VALUE                           |
+-------------------------+----------------------------------------------------------+
|         CI_LOOP         |                          daily                           |
|          DEBUG          |                          False                           |
|     DEPLOY_SCENARIO     |                 os-nosdn-nofeature-noha                  |
|      INSTALLER_TYPE     |                           oom                            |
|        BUILD_TAG        |      gitlab_ci-functest-kubespray-baremetal-daily-       |
|                         |                  master-479785435-onap                   |
|        NODE_NAME        |             onap_weekly_pod4_master-ONAP-oom             |
|       TEST_DB_URL       |     http://testresults.opnfv.org/onap/api/v1/results     |
|     TEST_DB_EXT_URL     |                                                          |
|     S3_ENDPOINT_URL     |                                                          |
|        S3_DST_URL       |                                                          |
|       HTTP_DST_URL      |                                                          |
+-------------------------+----------------------------------------------------------+

2022-03-07 13:46:14,013 - xtesting.ci.run_tests - INFO - Loading test case 'basic_cds'...
2022-03-07 13:46:14,822 DEBUG 25:cds_blueprint_enrichment.py(13) - CDS blueprint enrichment initialization
2022-03-07 13:46:14,839 DEBUG 219:extension.py(13) - found extension EntryPoint(name='basic_clamp', value='onaptests.scenario.basic_clamp:BasicClamp', group='xtesting.testcase')
[....]
2022-03-07 13:46:33,393 INFO 176:run_tests.py(13) - Test result:

+-------------------+---------------------+------------------+----------------+
|     TEST CASE     |       PROJECT       |     DURATION     |     RESULT     |
+-------------------+---------------------+------------------+----------------+
|     basic_cds     |     integration     |      00:18       |      PASS      |
+-------------------+---------------------+------------------+----------------+

2022-03-07 13:46:33,397 INFO 51:cds_blueprint_enrichment.py(13) - Generate Test report
2022-03-07 13:46:33,477 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_OK
2022-03-07 13:46:33,477 INFO 275:run_tests.py(13) - Execution exit value: Result.EX_OK


# note 1: the results are in /tmp/reporting.html and the logs in /tmp/pythonsdk.debug.log
# note 2: the env var set in the env file shall corresponds to the test you run e.g for basic_cds => ONAP_PYTHON_SDK_SETTINGS=onaptests.configuration.cba_enrichment_settings
# note 3 the name of the test is in the test DB and defined as the entry point of xtesting https://github.com/onap/testsuite-pythonsdk-tests/blob/master/setup.cfg
# note 4, if you want to play directly with the python test or sdk code, it is respectively under /usr/lib/python3.8/site-packages/onaptests and /usr/lib/python3.8/site-packages/onapsdk

bash-5.1# ls /usr/lib/python3.8/site-packages/onaptests
__init__.py    __pycache__    configuration  scenario       steps          templates      utils
bash-5.1# ls /usr/lib/python3.8/site-packages/onapsdk
__init__.py      aai              clamp            constants.py     exceptions.py    nbi              sdc              so               version.py       vid
__pycache__      cds              configuration    dmaap            msb              onap_service.py  sdnc             utils            ves