Kohn - Debug

Owned by John Keeney (Ericsson EST)
Last updated: Jan 05, 2023
4 min read

When running the instructions from our Run wiki page, still errors can happen. This page will try to address some common issues we have seen that could help to successfully run all components.

Log collection

In order to collect logs from a docker container you can use the following command:

docker exec <container_name> cat <log_path>

Below there is a list with the container names follow by the log path, to replace the values in the command above. The container names correspond to the names used in the example Run page, to double check that the you are using the same names in your environment, you could use docker ps.

sdnc_container="/opt/opendaylight/data/log/karaf.log"

a1policymanagmentservice="/var/log/policy-agent/application.log"

nonrtric-control-panel="/logs/nonrtric-controlpanel.log"

nonrtric-gateway="/var/log/nonrtric-gateway/application.log"



The below command can be used to check if there is any error in the logs. For the a1policymanagmentservice container you can check the log using the following. Other containers can be checked in a similar way.

docker exec a1policymanagmentservice grep ERR /var/log/policy-agent/application.log

Warnings can be also checked, using WARN instead or ERR in the command above.

Did SDNC (A1 Controller) start correctly?

If using A1 Controller (SDNC & A1 Adapter), SDNC should be started. Specifically, mariadb and sdnc_image must have started correctly. 

Check MariaDB container 

If you can see the following message in the logs for DB container: 

sdnc_db_container | 2021-10-28 14:39:34+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' sdnc_db_container | 2021-10-28 14:39:34+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.12+maria~focal started. sdnc_db_container | 2021-10-28 14:39:35+00:00 [ERROR] [Entrypoint]: Database is uninitialized and password option is not specified sdnc_db_container |     You need to specify one of MARIADB_ROOT_PASSWORD, MARIADB_ALLOW_EMPTY_ROOT_PASSWORD and MARIADB_RANDOM_ROOT_PASSWORD sdnc_db_container exited with code 1

You probably are missing values for MARIADB_ROOT_PASSWORD, MARIADB_ALLOW_EMPTY_ROOT_PASSWORD and MARIADB_RANDOM_ROOT_PASSWORD. These values must be provided in order for docker compose to work correctly. You can use following values:

export MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-mySecretPassword} export MYSQL_USER=${MYSQL_USER:-sdnc} export MYSQL_PASSWORD=${MYSQL_PASSWORD:-test123} export MYSQL_DATABASE=${MYSQL_DATABASE:-sdncdb} export ODL_USER=${ODL_USER:-admin} export ODL_PASSWORD=${ODL_PASSWORD:-Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U}

To be sure that the container is up and running correctly, following message must be seen in the console:

sdnc_db_container | 2021-10-28 14:43:56 0 [Note] mysqld: ready for connections.

Check SDNC container

Make sure the following message has appear in the console of the docker-compose before continue the rest of the wiki page:

sdnc_controller | Apache Karaf starting up. Press Enter to open the shell now... 100% [========================================================================] sdnc_controller | sdnc_controller | Karaf started in 0s. Bundle stats: 19 active, 19 total sdnc_controller | Certificate installation in progress. Elapsed time - 10 secs. Waiting for 10 secs before checking the status.. sdnc_controller | Certificate installation in progress. Elapsed time - 20 secs. Waiting for 10 secs before checking the status.. sdnc_controller | Certificate installation in progress. Elapsed time - 30 secs. Waiting for 10 secs before checking the status.. sdnc_controller | Start cert provisioning. Log file: /opt/opendaylight/current/data/log/installCerts.log sdnc_controller | Certificate installation script completed execution sdnc_controller | Everything OK in Certificate Installation

Check SDNC logs 

Logs from sdnc_controller container can be found in /opt/opendaylight/data/log/karaf.log directory from the sdnc_controller container. To collect the logs, the following command can be used:

docker exec -t sdnc_controller cat /opt/opendaylight/data/log/karaf.log > karaf_log.log

Log will be store in current directory with the name karaf_log.log. Once we get the logs, we can verify if there is any error. 

If there is any error in SDNC you can contact CCSDK/SDNC team for help.

Did A1 Policy Management Service start correctly?

Volume not mounted properly

The A1 Policy Management Service is unlikely to operate as expected if its configuration files (volumes) are not loaded/mounted correctly. 

For example:

2021-10-29 07:23:04.878 DEBUG 1 --- [     parallel-2] o.o.c.o.a.dmaap.DmaapMessageConsumer     : InfiniteFlux.onRequest 1
2021-10-29 07:23:04.879 DEBUG 1 --- [     parallel-2] o.o.c.o.a.dmaap.DmaapMessageConsumer     : fetchFromDmaap, no action DMAAP not configured

To be sure that the configuration file was mounted properly execute the follow command:

$ docker inspect -f '{{ json .Mounts }}' a1policymanagmentservice | jq

The output below shows when the file was incorrectly mounted:

[
  {
    "Type": "volume",
    "Name": "f85d0d2c9c11dc9cd9cb20e6658711442fbd326ac8805ba2e5aa77d1c80af785",
    "Source": "/var/lib/docker/volumes/f85d0d2c9c11dc9cd9cb20e6658711442fbd326ac8805ba2e5aa77d1c80af785/_data",
    "Destination": ":/opt/app/policy-agent/data/application_configuration.json",
    "Driver": "local",
    "Mode": "",
    "RW": true,
    "Propagation": ""
  }
]

When the file is mounted correctly the output must look similar to the following:

$ docker inspect -f '{{ json .Mounts }}' a1policymanagmentservice | jq

[
  {
    "Type": "bind",
    "Source": "/home/ychacon/repo/onap/ccsdk/oran/a1-policy-management/config/application_configuration.json",
    "Destination": "/opt/app/policy-agent/data/application_configuration.json",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  }
]

where the source should specify the absolute path of the file, which should be recognizable for your environment. 

Cannot establish connection with RIC/Simulator

If when running A1 Policy Management Service you see log warning like those below, this indicates that the configured 'ric's cannot be contacted.

2021-10-29 07:33:06.134  WARN 1 --- [or-http-epoll-1] o.o.c.o.a.clients.A1ClientFactory        : Could not get protocol version from Near-RT RIC: ric3 2021-10-29 07:33:06.139  WARN 1 --- [or-http-epoll-1] o.o.c.o.a.tasks.RefreshConfigTask        : Failed to synchronize types in new RIC: ric3, reason: Protocol negotiation failed for ric3 2021-10-29 07:33:06.154  WARN 1 --- [or-http-epoll-1] o.o.c.o.a.clients.A1ClientFactory        : Could not get protocol version from Near-RT RIC: ric4 2021-10-29 07:33:06.154  WARN 1 --- [or-http-epoll-1] o.o.c.o.a.tasks.RefreshConfigTask        : Failed to synchronize types in new RIC: ric4, reason: Protocol negotiation failed for ric4 reactor.core.Exceptions$ErrorCallbackNotImplemented: org.onap.ccsdk.oran.a1policymanagementservice.exceptions.ServiceException: Protocol negotiation failed for ric1 Caused by: org.onap.ccsdk.oran.a1policymanagementservice.exceptions.ServiceException: Protocol negotiation failed for ric1

  • Check that all containers are running in same network?

    • You can use following command to check:

docker network inspect -f '{{ range $key, $value := .Containers }}{{printf "%s: %s\n" $key .Name}}{{ end }}' nonrtric-docker-net

  •     Check that the sdnc_controller container is on network nonrtric-docker-net, and if not use:

docker network connect nonrtric-docker-net sdnc_controller


Then restart the A1 Policy Management Service and try again.