CPS-802 Upgrade SDN-C and verify root access

CPS-802 - Getting issue details... STATUS

The SDN-C version is decided to upgrade from version 1.8.1 to 2.2.3.

References:

CCSDK-3582 - Getting issue details... STATUS

The original result for the legacy URL using SDNC version 1.8.1 is the following:

Legacy URL

Result

http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount/turing-machine:turing-machine

{
    "turing-machine:turing-machine": {
        "transition-function": {
            "delta": [
                {
                    "label": "separator",
                    "input": {
                        "state": 0,
                        "symbol": "0"
                    },
                    "output": {
                        "state": 1,
                        "symbol": "1"
                    }
                },
                {
                    "label": "write separator",
                    "input": {
                        "state": 2,
                        "symbol": "1"
                    },
                    "output": {
                        "state": 3,
                        "symbol": "0",
                        "head-move": "left"
                    }
                },
                {
                    "label": "right summand",
                    "input": {
                        "state": 1,
                        "symbol": "1"
                    }
                },
                {
                    "label": "final step",
                    "input": {
                        "state": 3,
                        "symbol": ""
                    },
                    "output": {
                        "state": 4
                    }
                },
                {
                    "label": "go home",
                    "input": {
                        "state": 3,
                        "symbol": "1"
                    },
                    "output": {
                        "head-move": "left"
                    }
                },
                {
                    "label": "right end",
                    "input": {
                        "state": 1,
                        "symbol": ""
                    },
                    "output": {
                        "state": 2,
                        "head-move": "left"
                    }
                },
                {
                    "label": "left summand",
                    "input": {
                        "state": 0,
                        "symbol": "1"
                    }
                }
            ]
        }
    }
}

Endpoint Test Results

The following are the results of using the URLs to get nodes using the new version SDNC 2.2.3.

URL	Result	Notes
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount/turing-machine:turing-machine	Expand source { "turing-machine:turing-machine": { "transition-function": { "delta": [ { "label": "separator", "output": { "state": 1, "symbol": "1" }, "input": { "state": 0, "symbol": "0" } }, { "label": "right end", "output": { "state": 2, "head-move": "left" }, "input": { "state": 1, "symbol": "" } }, { "label": "write separator", "output": { "state": 3, "head-move": "left", "symbol": "0" }, "input": { "state": 2, "symbol": "1" } }, { "label": "right summand", "input": { "state": 1, "symbol": "1" } }, { "label": "go home", "output": { "head-move": "left" }, "input": { "state": 3, "symbol": "1" } }, { "label": "final step", "output": { "state": 4 }, "input": { "state": 3, "symbol": "" } }, { "label": "left summand", "input": { "state": 0, "symbol": "1" } } ] } } }	Size = 796 B 76 lines in total Starts at specified node (turing-machine)
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount		Size = 19.59 KB 883 lines in total Starts at the root node and includes all other child nodes including the node 'turing-machine'
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount/		Size = 19.79 KB 883 lines in total Starts at the root node and includes all other child nodes including the node 'turing-machine'
http://localhost:8282/restconf/config/network-topology:network-topology/topology/topology-netconf/node/PNFDemo/yang-ext:mount		Size = 5.25 KB 223 lines in total Starts at specified node (turing-machine) and rest of the nodes
http://localhost:8282/restconf/config/network-topology:network-topology/topology/topology-netconf/node/PNFDemo/yang-ext:mount/turing-machine:turing-machine		Size = 921 B 76 lines in total Starts at specified node (turing-machine)

CSIT/CCSDK Automation Issues

Where were we?

Our integration (and manual) testing using SDN-C v.1.8.1 worked fine. At a high level the setup followed these steps

pre-generated (?) zip (csit/plans/cps/sdnc/certs) extract to /opt/opendaylight/current/certs
Install SDN-C v 1.8.1
Mount a node
Execute /rests and /restconf requests to nodes successfully either manual and directly to SND-C or using CPS services

Old CPS SDNC docker-compose.yml

# ============LICENSE_START=======================================================
# Copyright (C) 2021 Nordix Foundation
# ================================================================================
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============LICENSE_END=========================================================

version: '3'

services:
  mariadb:
    image: mariadb:10.1.11
    ports:
      - "3306:3306"
    container_name: mariadb
    environment:
      - MYSQL_ROOT_PASSWORD=password
    hostname:
      mariadb.so.testlab.onap.org
    logging:
      driver: "json-file"
      options:
        max-size: "30m"
        max-file: "5"
  sdnc:
    image: onap/sdnc-image:1.8.1
    container_name: sdnc
    volumes: 
      - /etc/localtime:/etc/localtime:ro
      - ./certs:/opt/opendaylight/current/certs
    entrypoint: ["/opt/onap/sdnc/bin/startODL.sh"]
    ports:
      - "8282:8181"
    hostname:
      sdnc
    depends_on:
      - mariadb
    environment:
      - MYSQL_ROOT_PASSWORD=password
      - SDNC_CONFIG_DIR=/opt/onap/sdnc/data/properties
      - MYSQL_PASSWD=password
      - ODL_CERT_DIR=/opt/opendaylight/current/certs
      - ODL_ADMIN_USERNAME=admin
      - ODL_ADMIN_PASSWORD=Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U
    dns:
      - ${DNS_IP_ADDR-10.0.100.1}
    logging:
      driver:   "json-file"
      options:
        max-size: "30m"
        max-file: "5"
    extra_hosts:
      - sdnctldb02:${LOCAL_IP}
      - sdnctldb01:${LOCAL_IP}
      - dbhost:${LOCAL_IP}

Where are we now?

~~Installing pre-existing certs.~~ This caused issues with SDN-C v. 2.2.3 installation, so we removed this step[ (we assume SDN-C now includes its own and/or ODL certs)

Install SDN-C (output includes details on ODL certification installation)

SDNC Certificate Success

100% [========================================================================]
Karaf started in 44s. Bundle stats: 433 active, 434 total
Certificate installation in progress. Elapsed time - 60 secs. Waiting for 10 secs before checking the status.. 
Certificate installation in progress. Elapsed time - 70 secs. Waiting for 10 secs before checking the status.. 
Certificate installation in progress. Elapsed time - 80 secs. Waiting for 10 secs before checking the status.. 
Certificate installation in progress. Elapsed time - 90 secs. Waiting for 10 secs before checking the status.. 
Start cert provisioning. Log file: /opt/opendaylight/current/data/log/installCerts.log
Certificate installation script completed execution 
Everything OK in Certificate Installation

Mount Node
RestConf queries work fine:
We can also query SDNC to return all nodes using http://localhost:8282/restconf/config/network-topology:network-topology/topology/topology-netconf

the nodes can also be retrieved using /restconf

/rest based request fail
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=DemoNode/yang-ext:mount/turing-machine:turing-machine we receive the following error:

Postman Response

{
    "errors": {
        "error": [
            {
                "error-tag": "resource-denied-transport",
                "error-type": "protocol",
                "error-message": "Mount point does not exist."
            }
        ]
    }
}

CPS CSIT test fail with same root cause
CSIT Error
09:49:08.028 [http-nio-8080-exec-8] ERROR o.o.c.n.d.e.DmiExceptionHandler - Exception occurred
org.springframework.web.client.HttpServerErrorException$ServiceUnavailable: 503 Service Unavailable: [{"errors":{"error":[{"error-tag":"resource-denied-transport","error-type":"protocol","error-message":"Mount point does not exist."}]}}]

Summary

Perhaps there is a change in the way /rests behaves that we are unfamiliar with or perhaps our configuration is incorrect. To sum up: we can successfully start SDNC, mount a node, query nodes using /restconf but all /rests calls seem to fail. This could be an issue with certs or TLS.

Open Questions

#

Question/Issue

Notes/Decision

1

Are we to generate certs for SDNC ourselves or can we rely on the certs used as part of SDNC itself?

As mentioned on https://gerrit.onap.org/r/gitweb?p=sdnc/oam.git;a=blob;f=docs/cert_installation.rst;h=44dfe05306adb316a03ea3ca1c05077757fd9574;hb=refs/heads/istanbul certs folder is required as part of installing SDNC through docker-compose

2

Do we have some incorrect config in our docker-compose file?

CPS SDNC docker-compose.yml

# ============LICENSE_START=======================================================
# Copyright (C) 2022 Nordix Foundation
# ================================================================================
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============LICENSE_END=========================================================

version: '3'

services:
  mariadb:
    image: mariadb:10.5
    container_name: sdnc_db_container
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-password}
      - MYSQL_ROOT_HOST=%
      - MYSQL_USER=${MYSQL_USER:-sdnc}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD:-password}
      - MYSQL_DATABASE=${MYSQL_DATABASE:-sdncdb}
    logging:
      driver: "json-file"
      options:
        max-size: "30m"
        max-file: "5"

  ansible:
    image: onap/sdnc-ansible-server-image:2.2.2
    depends_on :
      - mariadb
    container_name: sdnc_ansible_container
    entrypoint: ["/opt/ansible-server/startAnsibleServer.sh"]
    ports:
      - "8000"
    links:
      - mariadb:dbhost
      - mariadb:sdnctldb01
      - mariadb:sdnctldb02
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-password}
      - MYSQL_USER=${MYSQL_USER:-sdnc}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD:-password}
      - MYSQL_DATABASE=${MYSQL_DATABASE:-sdncdb}
      - ANSIBLE_TRUSTSTORE_PASSWORD=${ANSIBLE_TRUSTSTORE_PASSWORD:-changeit}
    logging:
      driver:   "json-file"
      options:
        max-size: "30m"
        max-file: "5"

  sdnc:
    image: onap/sdnc-image:${VERSION:-2.2.3}
    depends_on :
      - mariadb
      - ansible
    container_name: sdnc_controller
    entrypoint: ["/opt/onap/sdnc/bin/startODL.sh"]
    ports:
      - "8282:8181"
    links:
      - mariadb:dbhost
      - mariadb:sdnctldb01
      - mariadb:sdnctldb02
      - ansible:ansiblehost
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-password}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD-password}
      - MYSQL_DATABASE=${MYSQL_DATABASE:-sdncdb}
      - SDNC_CONFIG_DIR=/opt/onap/sdnc/data/properties
      - SDNC_BIN=/opt/onap/sdnc/bin
      - ODL_CERT_DIR=/tmp
      - ODL_ADMIN_USERNAME=${ODL_USER:-admin}
      - ODL_ADMIN_PASSWORD=${ODL_PASSWORD:-Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U}
      - ODL_USER=${ODL_USER:-admin}
      - ODL_PASSWORD=${ODL_PASSWORD:-Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U}
      - SDNC_DB_INIT=true
      - HONEYCOMB_USER=${HONEYCOMB_USER:-admin}
      - HONEYCOMB_PASSWORD=${HONEYCOMB_PASSWORD:-admin}
      - TRUSTSTORE_PASSWORD=${TRUSTSTORE_PASSWORD:-changeit}
      - KEYSTORE_PASSWORD=${KEYSTORE_PASSWORD:-adminadmin}
      - SO_USER=${SO_USER:-sdncaBpmn}
      - SO_PASSWORD=${SO_PASSWORD:-password1$$}
      - NENG_USER=${NENG_USER:-ccsdkapps}
      - NENG_PASSWORD=${NENG_PASSWORD:-ccsdkapps}
      - CDS_USER=${CDS_USER:-ccsdkapps}
      - CDS_PASSWORD=${CDS_PASSWORD:-ccsdkapps}
      - ANSIBLE_USER=${ANSIBLE_USER:-sdnc}
      - ANSIBLE_PASSWORD=${ANSIBLE_PASSWORD:-sdnc}
      - SQL_CRYPTKEY=${SQL_CRYPTKEY:-fakECryptKey}
      - A1_TRUSTSTORE_PASSWORD=a1adapter
    dns:
      - ${DNS_IP_ADDR-10.0.100.1}
    logging:
      driver:   "json-file"
      options:
        max-size: "30m"
        max-file: "5"
    extra_hosts:
      aaf.osaaf.org: 10.12.6.214

3

CPS has certs within our repo which were generated for previous versions of SDNC. If we mount the volume as such:

volumes:
- $SDNC_CERT_PATH:/opt/opendaylight/current/certs

where SDNC_CERT_PATH is the absolute path of the certs within the cps repo, we get the following error in SDNC cert logs:

18:23:42 2022-02-07 18:09:57,310 - root - ERROR - Error while extracting zip file(s). Exiting Certificate Installation.
18:23:42 2022-02-07 18:09:57,310 - root - INFO - Error details : [Errno 13] Permission denied: '/opt/opendaylight/current/certs/keys0'
18:23:42 Stoppping SDNR container due to failure in installing Certificates

This is how we installed and used certs for SDNC 1.8.1 so has the process of accessing the certs changed?