CPS-802 Upgrade SDN-C and verify root access

CPS-802 - Getting issue details... STATUS

The SDN-C version is decided to upgrade from version 1.8.1 to 2.2.3.

References:

CCSDK-3582 - Getting issue details... STATUS

The original result for the legacy URL using SDNC version 1.8.1 is the following:

Legacy URL

Result

http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount/turing-machine:turing-machine

{
    "turing-machine:turing-machine": {
        "transition-function": {
            "delta": [
                {
                    "label": "separator",
                    "input": {
                        "state": 0,
                        "symbol": "0"
                    },
                    "output": {
                        "state": 1,
                        "symbol": "1"
                    }
                },
                {
                    "label": "write separator",
                    "input": {
                        "state": 2,
                        "symbol": "1"
                    },
                    "output": {
                        "state": 3,
                        "symbol": "0",
                        "head-move": "left"
                    }
                },
                {
                    "label": "right summand",
                    "input": {
                        "state": 1,
                        "symbol": "1"
                    }
                },
                {
                    "label": "final step",
                    "input": {
                        "state": 3,
                        "symbol": ""
                    },
                    "output": {
                        "state": 4
                    }
                },
                {
                    "label": "go home",
                    "input": {
                        "state": 3,
                        "symbol": "1"
                    },
                    "output": {
                        "head-move": "left"
                    }
                },
                {
                    "label": "right end",
                    "input": {
                        "state": 1,
                        "symbol": ""
                    },
                    "output": {
                        "state": 2,
                        "head-move": "left"
                    }
                },
                {
                    "label": "left summand",
                    "input": {
                        "state": 0,
                        "symbol": "1"
                    }
                }
            ]
        }
    }
}

Endpoint Test Results

The following are the results of using the URLs to get nodes using the new version SDNC 2.2.3.

URL	Result	Notes
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount/turing-machine:turing-machine	Expand source { "turing-machine:turing-machine": { "transition-function": { "delta": [ { "label": "separator", "output": { "state": 1, "symbol": "1" }, "input": { "state": 0, "symbol": "0" } }, { "label": "right end", "output": { "state": 2, "head-move": "left" }, "input": { "state": 1, "symbol": "" } }, { "label": "write separator", "output": { "state": 3, "head-move": "left", "symbol": "0" }, "input": { "state": 2, "symbol": "1" } }, { "label": "right summand", "input": { "state": 1, "symbol": "1" } }, { "label": "go home", "output": { "head-move": "left" }, "input": { "state": 3, "symbol": "1" } }, { "label": "final step", "output": { "state": 4 }, "input": { "state": 3, "symbol": "" } }, { "label": "left summand", "input": { "state": 0, "symbol": "1" } } ] } } }	Size = 796 B 76 lines in total Starts at specified node (turing-machine)
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount		Size = 19.59 KB 883 lines in total Starts at the root node and includes all other child nodes including the node 'turing-machine'
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount/		Size = 19.79 KB 883 lines in total Starts at the root node and includes all other child nodes including the node 'turing-machine'
http://localhost:8282/restconf/config/network-topology:network-topology/topology/topology-netconf/node/PNFDemo/yang-ext:mount		Size = 5.25 KB 223 lines in total Starts at specified node (turing-machine) and rest of the nodes
http://localhost:8282/restconf/config/network-topology:network-topology/topology/topology-netconf/node/PNFDemo/yang-ext:mount/turing-machine:turing-machine		Size = 921 B 76 lines in total Starts at specified node (turing-machine)

CSIT/CCSDK Automation Issues

Ticket logged: SDNC-1667 - Getting issue details... STATUS

Where were we?

Our integration (and manual) testing using SDN-C v.1.8.1 worked fine. At a high level the setup followed these steps

pre-generated (?) zip (csit/plans/cps/sdnc/certs) extract to /opt/opendaylight/current/certs
Install SDN-C v 1.8.1
Mount a node
Execute /rests and /restconf requests to nodes successfully either manual and directly to SND-C or using CPS services

Old CPS SDNC docker-compose.yml

# ============LICENSE_START=======================================================
# Copyright (C) 2021 Nordix Foundation
# ================================================================================
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============LICENSE_END=========================================================

version: '3'

services:
  mariadb:
    image: mariadb:10.1.11
    ports:
      - "3306:3306"
    container_name: mariadb
    environment:
      - MYSQL_ROOT_PASSWORD=password
    hostname:
      mariadb.so.testlab.onap.org
    logging:
      driver: "json-file"
      options:
        max-size: "30m"
        max-file: "5"
  sdnc:
    image: onap/sdnc-image:1.8.1
    container_name: sdnc
    volumes: 
      - /etc/localtime:/etc/localtime:ro
      - ./certs:/opt/opendaylight/current/certs
    entrypoint: ["/opt/onap/sdnc/bin/startODL.sh"]
    ports:
      - "8282:8181"
    hostname:
      sdnc
    depends_on:
      - mariadb
    environment:
      - MYSQL_ROOT_PASSWORD=password
      - SDNC_CONFIG_DIR=/opt/onap/sdnc/data/properties
      - MYSQL_PASSWD=password
      - ODL_CERT_DIR=/opt/opendaylight/current/certs
      - ODL_ADMIN_USERNAME=admin
      - ODL_ADMIN_PASSWORD=Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U
    dns:
      - ${DNS_IP_ADDR-10.0.100.1}
    logging:
      driver:   "json-file"
      options:
        max-size: "30m"
        max-file: "5"
    extra_hosts:
      - sdnctldb02:${LOCAL_IP}
      - sdnctldb01:${LOCAL_IP}
      - dbhost:${LOCAL_IP}

Where are we now?

~~Installing pre-existing certs.~~ This caused issues with SDN-C v. 2.2.3 installation, so we removed this step[ (we assume SDN-C now includes its own and/or ODL certs)

Install SDN-C (output includes details on ODL certification installation)

SDNC Certificate Success

100% [========================================================================]
Karaf started in 44s. Bundle stats: 433 active, 434 total
Certificate installation in progress. Elapsed time - 60 secs. Waiting for 10 secs before checking the status.. 
Certificate installation in progress. Elapsed time - 70 secs. Waiting for 10 secs before checking the status.. 
Certificate installation in progress. Elapsed time - 80 secs. Waiting for 10 secs before checking the status.. 
Certificate installation in progress. Elapsed time - 90 secs. Waiting for 10 secs before checking the status.. 
Start cert provisioning. Log file: /opt/opendaylight/current/data/log/installCerts.log
Certificate installation script completed execution 
Everything OK in Certificate Installation

Mount Node
RestConf queries work fine:
We can also query SDNC to return all nodes using http://localhost:8282/restconf/config/network-topology:network-topology/topology/topology-netconf

the nodes can also be retrieved using /restconf

/rest based request fail
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=DemoNode/yang-ext:mount/turing-machine:turing-machine we receive the following error:

Postman Response

{
    "errors": {
        "error": [
            {
                "error-tag": "resource-denied-transport",
                "error-type": "protocol",
                "error-message": "Mount point does not exist."
            }
        ]
    }
}

CPS CSIT test fail with same root cause
CSIT Error
09:49:08.028 [http-nio-8080-exec-8] ERROR o.o.c.n.d.e.DmiExceptionHandler - Exception occurred
org.springframework.web.client.HttpServerErrorException$ServiceUnavailable: 503 Service Unavailable: [{"errors":{"error":[{"error-tag":"resource-denied-transport","error-type":"protocol","error-message":"Mount point does not exist."}]}}]

Summary

Perhaps there is a change in the way /rests behaves that we are unfamiliar with or perhaps our configuration is incorrect. To sum up: we can successfully start SDNC, mount a node, query nodes using /restconf but all /rests calls seem to fail. This could be an issue with certs or TLS.

Open Questions

#

Question/Issue

Notes/Decision

1

Are we to generate certs for SDNC ourselves or can we rely on the certs used as part of SDNC itself?

As mentioned on https://docs.onap.org/projects/onap-sdnc-oam/en/istanbul/cert_installation.html certs folder is required as part of installing SDNC through docker-compose

2

Do we have some incorrect config in our docker-compose file?

CPS SDNC docker-compose.yml

# ============LICENSE_START=======================================================
# Copyright (C) 2022 Nordix Foundation
# ================================================================================
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============LICENSE_END=========================================================

version: '3'

services:
  mariadb:
    image: mariadb:10.5
    container_name: sdnc_db_container
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-password}
      - MYSQL_ROOT_HOST=%
      - MYSQL_USER=${MYSQL_USER:-sdnc}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD:-password}
      - MYSQL_DATABASE=${MYSQL_DATABASE:-sdncdb}
    logging:
      driver: "json-file"
      options:
        max-size: "30m"
        max-file: "5"

  ansible:
    image: onap/sdnc-ansible-server-image:2.2.2
    depends_on :
      - mariadb
    container_name: sdnc_ansible_container
    entrypoint: ["/opt/ansible-server/startAnsibleServer.sh"]
    ports:
      - "8000"
    links:
      - mariadb:dbhost
      - mariadb:sdnctldb01
      - mariadb:sdnctldb02
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-password}
      - MYSQL_USER=${MYSQL_USER:-sdnc}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD:-password}
      - MYSQL_DATABASE=${MYSQL_DATABASE:-sdncdb}
      - ANSIBLE_TRUSTSTORE_PASSWORD=${ANSIBLE_TRUSTSTORE_PASSWORD:-changeit}
    logging:
      driver:   "json-file"
      options:
        max-size: "30m"
        max-file: "5"

  sdnc:
    image: onap/sdnc-image:${VERSION:-2.2.3}
    depends_on :
      - mariadb
      - ansible
    container_name: sdnc_controller
    entrypoint: ["/opt/onap/sdnc/bin/startODL.sh"]
    ports:
      - "8282:8181"
    links:
      - mariadb:dbhost
      - mariadb:sdnctldb01
      - mariadb:sdnctldb02
      - ansible:ansiblehost
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-password}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD-password}
      - MYSQL_DATABASE=${MYSQL_DATABASE:-sdncdb}
      - SDNC_CONFIG_DIR=/opt/onap/sdnc/data/properties
      - SDNC_BIN=/opt/onap/sdnc/bin
      - ODL_CERT_DIR=/tmp
      - ODL_ADMIN_USERNAME=${ODL_USER:-admin}
      - ODL_ADMIN_PASSWORD=${ODL_PASSWORD:-Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U}
      - ODL_USER=${ODL_USER:-admin}
      - ODL_PASSWORD=${ODL_PASSWORD:-Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U}
      - SDNC_DB_INIT=true
      - HONEYCOMB_USER=${HONEYCOMB_USER:-admin}
      - HONEYCOMB_PASSWORD=${HONEYCOMB_PASSWORD:-admin}
      - TRUSTSTORE_PASSWORD=${TRUSTSTORE_PASSWORD:-changeit}
      - KEYSTORE_PASSWORD=${KEYSTORE_PASSWORD:-adminadmin}
      - SO_USER=${SO_USER:-sdncaBpmn}
      - SO_PASSWORD=${SO_PASSWORD:-password1$$}
      - NENG_USER=${NENG_USER:-ccsdkapps}
      - NENG_PASSWORD=${NENG_PASSWORD:-ccsdkapps}
      - CDS_USER=${CDS_USER:-ccsdkapps}
      - CDS_PASSWORD=${CDS_PASSWORD:-ccsdkapps}
      - ANSIBLE_USER=${ANSIBLE_USER:-sdnc}
      - ANSIBLE_PASSWORD=${ANSIBLE_PASSWORD:-sdnc}
      - SQL_CRYPTKEY=${SQL_CRYPTKEY:-fakECryptKey}
      - A1_TRUSTSTORE_PASSWORD=a1adapter
    dns:
      - ${DNS_IP_ADDR-10.0.100.1}
    logging:
      driver:   "json-file"
      options:
        max-size: "30m"
        max-file: "5"
    extra_hosts:
      aaf.osaaf.org: 10.12.6.214

Need mount to specific files, see https://gerrit.onap.org/r/c/cps/+/126945/14..15/csit/plans/cps/sdnc/docker-compose.yml

Original guide used for sdnc docker-compose can be found here: Istanbul - Run.

3

CPS has certs within our repo which were generated for previous versions of SDNC. If we mount the volume as such:

volumes:
- $SDNC_CERT_PATH:/opt/opendaylight/current/certs

where SDNC_CERT_PATH is the absolute path of the certs within the cps repo, we get the following error in SDNC cert logs:

18:23:42 2022-02-07 18:09:57,310 - root - ERROR - Error while extracting zip file(s). Exiting Certificate Installation.
18:23:42 2022-02-07 18:09:57,310 - root - INFO - Error details : [Errno 13] Permission denied: '/opt/opendaylight/current/certs/keys0'
18:23:42 Stoppping SDNR container due to failure in installing Certificates

This is how we installed and used certs for SDNC 1.8.1 so has the process of accessing the certs changed?

This was resolved by adding separate volume mounts for the files contained with the certs folder.

Old:

volumes:
     - $SDNC_CERT_PATH:/opt/opendaylight/current/certs

New:

volumes:
      - ./certs/certs.properties:/opt/opendaylight/certs/certs.properties
      - ./certs/keys0.zip:/opt/opendaylight/certs/keys0.zip