CPS-802 Upgrade SDN-C and verify root access

https://lf-onap.atlassian.net/browse/CPS-802

The SDN-C version is decided to upgrade from version 1.8.1 to 2.2.3.

1 Endpoint Test Results
2 CSIT/CCSDK Automation Issues
- 2.1 Where were we?
  - 2.1.1 Old CPS SDNC docker-compose.yml
- 2.2 Where are we now?
- 2.3 Summary
- 2.4 Open Questions
  - 2.4.1 CPS SDNC docker-compose.yml

References:

https://lf-onap.atlassian.net/browse/CCSDK-3582

The original result for the legacy URL using SDNC version 1.8.1 is the following:

Legacy URL	Result

Legacy URL

Result

http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount/turing-machine:turing-machine

{
    "turing-machine:turing-machine": {
        "transition-function": {
            "delta": [
                {
                    "label": "separator",
                    "input": {
                        "state": 0,
                        "symbol": "0"
                    },
                    "output": {
                        "state": 1,
                        "symbol": "1"
                    }
                },
                {
                    "label": "write separator",
                    "input": {
                        "state": 2,
                        "symbol": "1"
                    },
                    "output": {
                        "state": 3,
                        "symbol": "0",
                        "head-move": "left"
                    }
                },
                {
                    "label": "right summand",
                    "input": {
                        "state": 1,
                        "symbol": "1"
                    }
                },
                {
                    "label": "final step",
                    "input": {
                        "state": 3,
                        "symbol": ""
                    },
                    "output": {
                        "state": 4
                    }
                },
                {
                    "label": "go home",
                    "input": {
                        "state": 3,
                        "symbol": "1"
                    },
                    "output": {
                        "head-move": "left"
                    }
                },
                {
                    "label": "right end",
                    "input": {
                        "state": 1,
                        "symbol": ""
                    },
                    "output": {
                        "state": 2,
                        "head-move": "left"
                    }
                },
                {
                    "label": "left summand",
                    "input": {
                        "state": 0,
                        "symbol": "1"
                    }
                }
            ]
        }
    }
}

Endpoint Test Results

The following are the results of using the URLs to get nodes using the new version SDNC 2.2.3.

URL	Result	Notes

URL	Result	Notes
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount/turing-machine:turing-machine	{ "turing-machine:turing-machine": { "transition-function": { "delta": [ { "label": "separator", "output": { "state": 1, "symbol": "1" }, "input": { "state": 0, "symbol": "0" } }, { "label": "right end", "output": { "state": 2, "head-move": "left" }, "input": { "state": 1, "symbol": "" } }, { "label": "write separator", "output": { "state": 3, "head-move": "left", "symbol": "0" }, "input": { "state": 2, "symbol": "1" } }, { "label": "right summand", "input": { "state": 1, "symbol": "1" } }, { "label": "go home", "output": { "head-move": "left" }, "input": { "state": 3, "symbol": "1" } }, { "label": "final step", "output": { "state": 4 }, "input": { "state": 3, "symbol": "" } }, { "label": "left summand", "input": { "state": 0, "symbol": "1" } } ] } } }	Size = 796 B 76 lines in total Starts at specified node (turing-machine)
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount		Size = 19.59 KB 883 lines in total Starts at the root node and includes all other child nodes including the node 'turing-machine'
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=PNFDemo/yang-ext:mount/		Size = 19.79 KB 883 lines in total Starts at the root node and includes all other child nodes including the node 'turing-machine'
http://localhost:8282/restconf/config/network-topology:network-topology/topology/topology-netconf/node/PNFDemo/yang-ext:mount		Size = 5.25 KB 223 lines in total Starts at specified node (turing-machine) and rest of the nodes
http://localhost:8282/restconf/config/network-topology:network-topology/topology/topology-netconf/node/PNFDemo/yang-ext:mount/turing-machine:turing-machine		Size = 921 B 76 lines in total Starts at specified node (turing-machine)

CSIT/CCSDK Automation Issues

Ticket logged: https://lf-onap.atlassian.net/browse/SDNC-1667

Where were we?

Our integration (and manual) testing using SDN-C v.1.8.1 worked fine. At a high level the setup followed these steps

pre-generated (?) zip (csit/plans/cps/sdnc/certs) extract to /opt/opendaylight/current/certs
Install SDN-C v 1.8.1
Mount a node
Execute /rests and /restconf requests to nodes successfully either manual and directly to SND-C or using CPS services

Old CPS SDNC docker-compose.yml

# ============LICENSE_START=======================================================
# Copyright (C) 2021 Nordix Foundation
# ================================================================================
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============LICENSE_END=========================================================

version: '3'

services:
  mariadb:
    image: mariadb:10.1.11
    ports:
      - "3306:3306"
    container_name: mariadb
    environment:
      - MYSQL_ROOT_PASSWORD=password
    hostname:
      mariadb.so.testlab.onap.org
    logging:
      driver: "json-file"
      options:
        max-size: "30m"
        max-file: "5"
  sdnc:
    image: onap/sdnc-image:1.8.1
    container_name: sdnc
    volumes: 
      - /etc/localtime:/etc/localtime:ro
      - ./certs:/opt/opendaylight/current/certs
    entrypoint: ["/opt/onap/sdnc/bin/startODL.sh"]
    ports:
      - "8282:8181"
    hostname:
      sdnc
    depends_on:
      - mariadb
    environment:
      - MYSQL_ROOT_PASSWORD=password
      - SDNC_CONFIG_DIR=/opt/onap/sdnc/data/properties
      - MYSQL_PASSWD=password
      - ODL_CERT_DIR=/opt/opendaylight/current/certs
      - ODL_ADMIN_USERNAME=admin
      - ODL_ADMIN_PASSWORD=Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U
    dns:
      - ${DNS_IP_ADDR-10.0.100.1}
    logging:
      driver:   "json-file"
      options:
        max-size: "30m"
        max-file: "5"
    extra_hosts:
      - sdnctldb02:${LOCAL_IP}
      - sdnctldb01:${LOCAL_IP}
      - dbhost:${LOCAL_IP}

Where are we now?

Installing pre-existing certs. This caused issues with SDN-C v. 2.2.3 installation, so we removed this step[ (we assume SDN-C now includes its own and/or ODL certs)
Install SDN-C (output includes details on ODL certification installation)

SDNC Certificate Success
100% [========================================================================] Karaf started in 44s. Bundle stats: 433 active, 434 total Certificate installation in progress. Elapsed time - 60 secs. Waiting for 10 secs before checking the status.. Certificate installation in progress. Elapsed time - 70 secs. Waiting for 10 secs before checking the status.. Certificate installation in progress. Elapsed time - 80 secs. Waiting for 10 secs before checking the status.. Certificate installation in progress. Elapsed time - 90 secs. Waiting for 10 secs before checking the status.. Start cert provisioning. Log file: /opt/opendaylight/current/data/log/installCerts.log Certificate installation script completed execution Everything OK in Certificate Installation
Mount Node
RestConf queries work fine:
We can also query SDNC to return all nodes using http://localhost:8282/restconf/config/network-topology:network-topology/topology/topology-netconf

the nodes can also be retrieved using /restconf
/rest based request fail
http://localhost:8282/rests/data/network-topology:network-topology/topology=topology-netconf/node=DemoNode/yang-ext:mount/turing-machine:turing-machine we receive the following error:

Postman Response
{ "errors": { "error": [ { "error-tag": "resource-denied-transport", "error-type": "protocol", "error-message": "Mount point does not exist." } ] } }
CPS CSIT test fail with same root cause

Summary

Perhaps there is a change in the way /rests behaves that we are unfamiliar with or perhaps our configuration is incorrect. To sum up: we can successfully start SDNC, mount a node, query nodes using /restconf but all /rests calls seem to fail. This could be an issue with certs or TLS.

Open Questions

	Question/Issue	Notes/Decision

	Question/Issue	Notes/Decision
1	Are we to generate certs for SDNC ourselves or can we rely on the certs used as part of SDNC itself?	As mentioned on https://docs.onap.org/projects/onap-sdnc-oam/en/istanbul/cert_installation.html certs folder is required as part of installing SDNC through docker-compose
2	Do we have some incorrect config in our docker-compose file? CPS SDNC docker-compose.yml # ============LICENSE_START======================================================= # Copyright (C) 2022 Nordix Foundation # ================================================================================ # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============LICENSE_END========================================================= version: '3' services: mariadb: image: mariadb:10.5 container_name: sdnc_db_container ports: - "3306:3306" environment: - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-password} - MYSQL_ROOT_HOST=% - MYSQL_USER=${MYSQL_USER:-sdnc} - MYSQL_PASSWORD=${MYSQL_PASSWORD:-password} - MYSQL_DATABASE=${MYSQL_DATABASE:-sdncdb} logging: driver: "json-file" options: max-size: "30m" max-file: "5" ansible: image: onap/sdnc-ansible-server-image:2.2.2 depends_on : - mariadb container_name: sdnc_ansible_container entrypoint: ["/opt/ansible-server/startAnsibleServer.sh"] ports: - "8000" links: - mariadb:dbhost - mariadb:sdnctldb01 - mariadb:sdnctldb02 environment: - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-password} - MYSQL_USER=${MYSQL_USER:-sdnc} - MYSQL_PASSWORD=${MYSQL_PASSWORD:-password} - MYSQL_DATABASE=${MYSQL_DATABASE:-sdncdb} - ANSIBLE_TRUSTSTORE_PASSWORD=${ANSIBLE_TRUSTSTORE_PASSWORD:-changeit} logging: driver: "json-file" options: max-size: "30m" max-file: "5" sdnc: image: onap/sdnc-image:${VERSION:-2.2.3} depends_on : - mariadb - ansible container_name: sdnc_controller entrypoint: ["/opt/onap/sdnc/bin/startODL.sh"] ports: - "8282:8181" links: - mariadb:dbhost - mariadb:sdnctldb01 - mariadb:sdnctldb02 - ansible:ansiblehost environment: - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-password} - MYSQL_USER=${MYSQL_USER} - MYSQL_PASSWORD=${MYSQL_PASSWORD-password} - MYSQL_DATABASE=${MYSQL_DATABASE:-sdncdb} - SDNC_CONFIG_DIR=/opt/onap/sdnc/data/properties - SDNC_BIN=/opt/onap/sdnc/bin - ODL_CERT_DIR=/tmp - ODL_ADMIN_USERNAME=${ODL_USER:-admin} - ODL_ADMIN_PASSWORD=${ODL_PASSWORD:-Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U} - ODL_USER=${ODL_USER:-admin} - ODL_PASSWORD=${ODL_PASSWORD:-Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U} - SDNC_DB_INIT=true - HONEYCOMB_USER=${HONEYCOMB_USER:-admin} - HONEYCOMB_PASSWORD=${HONEYCOMB_PASSWORD:-admin} - TRUSTSTORE_PASSWORD=${TRUSTSTORE_PASSWORD:-changeit} - KEYSTORE_PASSWORD=${KEYSTORE_PASSWORD:-adminadmin} - SO_USER=${SO_USER:-sdncaBpmn} - SO_PASSWORD=${SO_PASSWORD:-password1$$} - NENG_USER=${NENG_USER:-ccsdkapps} - NENG_PASSWORD=${NENG_PASSWORD:-ccsdkapps} - CDS_USER=${CDS_USER:-ccsdkapps} - CDS_PASSWORD=${CDS_PASSWORD:-ccsdkapps} - ANSIBLE_USER=${ANSIBLE_USER:-sdnc} - ANSIBLE_PASSWORD=${ANSIBLE_PASSWORD:-sdnc} - SQL_CRYPTKEY=${SQL_CRYPTKEY:-fakECryptKey} - A1_TRUSTSTORE_PASSWORD=a1adapter dns: - ${DNS_IP_ADDR-10.0.100.1} logging: driver: "json-file" options: max-size: "30m" max-file: "5" extra_hosts: aaf.osaaf.org: 10.12.6.214	Need mount to specific files, see https://gerrit.onap.org/r/c/cps/+/126945/14..15/csit/plans/cps/sdnc/docker-compose.yml Original guide used for sdnc docker-compose can be found here: Istanbul - Run.
3	CPS has certs within our repo which were generated for previous versions of SDNC. If we mount the volume as such: volumes: - $SDNC_CERT_PATH:/opt/opendaylight/current/certs where SDNC_CERT_PATH is the absolute path of the certs within the cps repo, we get the following error in SDNC cert logs: 18:23:42 2022-02-07 18:09:57,310 - root - ERROR - Error while extracting zip file(s). Exiting Certificate Installation. 18:23:42 2022-02-07 18:09:57,310 - root - INFO - Error details : [Errno 13] Permission denied: '/opt/opendaylight/current/certs/keys0' 18:23:42 Stoppping SDNR container due to failure in installing Certificates This is how we installed and used certs for SDNC 1.8.1 so has the process of accessing the certs changed?	This was resolved by adding separate volume mounts for the files contained with the certs folder. Old: `volumes: - $SDNC_CERT_PATH:/opt/opendaylight/current/certs` New: `volumes: - ./certs/certs.properties:/opt/opendaylight/certs/certs.properties - ./certs/keys0.zip:/opt/opendaylight/certs/keys0.zip`