DMaaP Bus Controller Topic and Feed provisioning support via helm

This document highlights design considered for to Provision authenticated, authorized DMaaP topics on Message Router and feeds on Data Router .

Overview

Bus Controller is a RESTful web service used to provision DMaaP topics (on Message Router) and feeds (on Data Router), with associated authorization (on AAF).

Endpoints are used to provision :

Authorized topic on MR, and to create and grant permission for Publishers and Subscribers.
Feed on Data Router, with associated user authentication.
Bus Controller API link : https://docs.onap.org/projects/onap-dmaap-buscontroller/en/latest/apis/api.html

Gerrit review links

Ticket No.	Gerrit link	Comments	Status
DMAAP-1561	https://gerrit.onap.org/r/c/dmaap/buscontroller/+/121543	Updated dmaap-dbc script	OPEN
DCAEGEN2-2715	https://gerrit.onap.org/r/c/oom/+/120492	To be tested with 5gBulkpm gaiting testcase	OPEN
DCAEGEN2-2714	https://gerrit.onap.org/r/c/oom/+/120213	To be tested with 5gBulkpm gaiting testcase	OPEN

Dmaap dbc-client docker Image (onap/dmaap/dbc-client:1.0.9)

This dbc-client image is utility (containing shell script) by making use of which we can initiate HTTP REST Request towards Dmaap Bus Controller app for creating dynamic Feeds, Topics.

Design details

Existing design consist fo kubernetes Job Manifest which consist of onap/dmaap/dbc-client docker image for making http request toward dmaap-bc (Dmaap Bus Controller) pod.
In DCAE Design-1 highlighted in below pic is considered where we need to create dmmap provisionning Init-Container which will be making use of same onap/dmaap/dbc-client docker image to make Http Rest Request towards dmap-db pod.
1. Once the http request is made response will be written back to shared EmptyDir voulme.
2. Second Init-container (Update Config) will be reading the response from share EmptyDir voulme and will merge it with application config.

Feed + Dr_pubs _Dr_subs Input via Values.yaml

Data Router Feed, Publisher, Subscriber creation input details.

Old approach-
------------------------------------------
feedConfig:
  owner: dcaecm
  feedVersion: 0.0
  feedName: bulk_pm_feed
  asprClassification: unclassified
  feedDescription: DFC Feed Creation
  pubs:
    - username: dcaepub
      userpwd: tpJN3gjaAzPAiAa4
      dcaeLocationName: loc00
------------------------------------------

New approach-
------------------------------------------
# DataRouter Feed Configuration
drFeedConfig:
  - feedName: bulk_pm_feed
    owner: dcaecm
    feedVersion: 0.0
    asprClassification: unclassified
    feedDescription: DFC Feed Creation

# DataRouter Publisher Configuration
drPubConfig:
  - feedName: bulk_pm_feed
    dcaeLocationName: loc00

# DataRouter Subscriber Configuration
drSubConfig:
  - feedName: bulk_pm_feed
    dcaeLocationName: loc00
	decompress: True
    privilegedSubscriber: True
    deliveryURL: https://dcae-pm-mapper:8443/delivery
------------------------------------------

ConfigMap Configuration for Feed, Dr_Publisher
------------------------------------------
volumes:
  - name: feeds-config
    path: /opt/app/config/feeds/
  - name: drpub-config
    path: /opt/app/config/dr_pubs/
  - name: drsub-config
    path: /opt/app/config/dr_subs/

Code Snip for Updating Application config with dmaap Feed/Pub/Sub detals.

InitContainer Code Snip

{{- define "common.dmaap.provisioning.initContainer" -}}
{{- $dot := default . .dot -}}
{{- $drFeedConfig := default $dot.Values.drFeedConfig .drFeedConfig -}}
{{- $drPubConfig := default $dot.Values.drPubConfig .drPubConfig -}}
{{- if or $drFeedConfig $drPubConfig -}}
- name: {{ include "common.name" $dot }}-init-dmaap-provisioning
  image: {{ include "repositoryGenerator.image.dbcClient" $dot }}
  imagePullPolicy: {{ $dot.Values.global.pullPolicy | default $dot.Values.pullPolicy }}
  env:
  - name: RESP_CACHE
    value: /opt/app/config/cache
  - name: REQUESTID
    value: "{{ include "common.name" $dot }}-dmaap-provisioning"
  resources: {{ include "common.resources" $dot | nindent 1 }}
  volumeMounts:
  {{- include "common.dmaap.provisioning._volumeMounts" $dot | trim | nindent 2 }}
- name: {{ include "common.name" $dot }}-init-merge-config
  image: {{ include "repositoryGenerator.image.envsubst" $dot }}
  imagePullPolicy: {{ $dot.Values.global.pullPolicy | default $dot.Values.pullPolicy }}
  command:
  - /bin/sh
  args:
  - -c
  - |
    if [ -d /opt/app/config/cache/ ]; then
      cd /opt/app/config/cache/
      for file in $(ls feed*); do
        NUM=$(echo "$file" | sed 's/feedConfig-\([0-9]\+\)-resp.json/\1/')
        export DR_LOG_URL_"$NUM"="$(grep -o '"logURL":"[^"]*' "$file" | cut -d '"' -f4)"
        export DR_FILES_PUBLISHER_URL_"$NUM"="$(grep -o '"publishURL":"[^"]*' "$file" | cut -d '"' -f4)"
      done
      for file in $(ls drpub*); do
        NUM=$(echo "$file" | sed 's/drpubConfig-\([0-9]\+\)-resp.json/\1/')
        export DR_USERNAME_"$NUM"="$(grep -o '"username":"[^"]*' "$file" | cut -d '"' -f4)"
        export DR_PASSWORD_"$NUM"="$(grep -o '"userpwd":"[^"]*' "$file" | cut -d '"' -f4)"
        export DR_FILES_PUBLISHER_ID_"$NUM"="$(grep -o '"pubId":"[^"]*' "$file" | cut -d '"' -f4)"
      done
      for file in $(ls drsub*); do
        NUM=$(echo "$file" | sed 's/drsubConfig-\([0-9]\+\)-resp.json/\1/')
		export DR_USERNAME_"$NUM"="$(grep -o '"username":"[^"]*' "$file" | cut -d '"' -f4)"
        export DR_PASSWORD_"$NUM"="$(grep -o '"userpwd":"[^"]*' "$file" | cut -d '"' -f4)"
        export DR_FILES_SUBSCRIBER_ID_"$NUM"="$(grep -o '"subId":"[^"]*' "$file" | cut -d '"' -f4)"
      done
    else
      echo "No Response logged for Dmaap BusController Http POST Request..!"
    fi
    cd /config-input && for PFILE in `ls -1`; do envsubst <${PFILE} >/config/${PFILE}; done
  volumeMounts:
  - mountPath: /opt/app/config/cache/
    name: dbc-response-cache
  - mountPath: /config-input
    name: app-config-input
  - mountPath: /config
    name: app-config
  resources:
    limits:
      cpu: 200m
      memory: 250Mi
    requests:
      cpu: 100m
      memory: 200Mi
{{- end -}}
{{- end -}}

Open Issues

Data Router Publisher/Subscriber behaviour is not consistent

On dr_publiser creation if we dont provide username/password a default usename/password is created, but the same is not happening for dr_subscriber.
On dr_publisher creation if username/password is provided as part of input dmaap-dbc client Api returns username as output and password is defaulted ot "null".
On dr_subscriber creation if we supply username/password as part of input dmaap-dbc client Api returns both username/password as null.

Cmpv2Certificates generation

While testing with 5gbulkpm testcase Cmpv2Certificates processing is failing send mail for clarification.

**Further analysis is on going will be updating this wiki as we move along.