This document highlights design considered for to Provision authenticated, authorized DMaaP topics on Message Router and feeds on Data Router .
...
- Authorized topic on MR, and to create and grant permission for Publishers and Subscribers.
- Feed on Data Router, with associated user authentication.
- Bus Controller API link : https://docs.onap.org/projects/onap-dmaap-buscontroller/en/latest/apis/api.html
Gerrit review links
Ticket No. | Gerrit link | Comments | 5gbulkpm Test Result | Status |
---|---|---|---|---|
DMAAP-1561 | https://gerrit.onap.org/r/c/dmaap/buscontroller/+/121543 | Updated dmaap-dbc script | Pass |
Merged | |||
DCAEGEN2-2715 (DCAE Data File Collector) | https://gerrit.onap.org/r/c/oom/+/120492 | To be tested with 5gBulkpm gaiting testcase | Pass |
Merged | |||
DCAEGEN2-2714 (DCAE PM-Mapper) | https://gerrit.onap.org/r/c/oom/+/120213 | To be tested with 5gBulkpm gaiting testcase | Pass |
Merged |
Dmaap dbc-client docker Image (onap/dmaap/dbc-client:
...
2.0.
...
7)
This dbc-client image is utility (containing shell script) by making use of which we can initiate HTTP REST Request towards Dmaap Bus Controller app for creating dynamic Feeds, Topics.
...
- Existing design consist fo kubernetes Job Manifest which consist of onap/dmaap/dbc-client docker image for making http request toward dmaap-bc (Dmaap Bus Controller) pod.
- In DCAE Design-1 highlighted in below pic is considered where we need to create dmmap provisionning Init-Container which will be making use of same onap/dmaap/dbc-client docker image to make Http Rest Request towards dmap-db pod.
- Once the http request is made response will be written back to shared EmptyDir voulme.
- Second Init-container (Update Config) will be reading the response from share EmptyDir voulme and will merge it with application config.
...
Draw.io design-1 :
- Dmaap Provisioning init-container sends POST request to Dmaap Bus-Controller Pod for creation of Topics, Feeds.
- DCAE Merge init config container merges dcae application config and response received back from dmaap provisioning init-container.
Drawio | ||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
Data Router Feed, Publisher, Subscriber creation input details. Old approach- ------------------------------------------ feedConfig: owner: dcaecm feedVersion: 0.0 feedName: bulk_pm_feed asprClassification: unclassified feedDescription: DFC Feed Creation pubs: - username: dcaepub userpwd: tpJN3gjaAzPAiAa4 dcaeLocationName: loc00 ------------------------------------------ New approach- ------------------------------------------ # DataRouter Feed Configuration drFeedConfig: - feedName: bulk_pm_feed owner: dcaecm feedVersion: 0.0 asprClassification: unclassified feedDescription: DFC Feed Creation # DataRouter Publisher Configuration drPubConfig: - feedName: bulk_pm_feed dcaeLocationName: loc00 # DataRouter Subscriber Configuration drSubConfig: - feedName: bulk_pm_feed decompress: true username: ${DR_USERNAME} userpwd: ${DR_PASSWORD} dcaeLocationName: loc00 privilegedSubscriber: true deliveryURL: https://dcae-pm-mapper:8443/delivery # MessageRouter Topic, Publisher Configuration mrTopicsConfig: - topicName: PERFORMANCE_MEASUREMENTS topicDescription: PM Mapper publishes perf3gpp VES PM Events to authenticated MR topic owner: dcaecm tnxEnabled: false clients: - dcaeLocationName: san-francisco clientRole: org.onap.dcae.pmPublisher action: - pub - view ------------------------------------------ ConfigMap Configuration for Feed, Dr_Publisher ------------------------------------------ volumes: - name: feeds-config path: /opt/app/config/feeds/ - name: drpub-config path: /opt/app/config/dr_pubs/ - name: drsub-config path: /opt/app/config/dr_subs/ - name: topics-config path: /opt/app/config/topics |
...
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
$ kubectl logs -f -n onap dev-dcae-pm-mapper-6d84cb4757-mljn9 -c dcae-pm-mapper-init-dmaap-provisioning + export 'PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin' + RESP_CACHE=/opt/app/config/cache + RESP=/dev/null + APP_ROOT=/opt/app/dbc-client + CONFIGMAP_ROOT=/opt/app/config + PORT=8443 + DBC=dmaap-bc + PROTO=https + PARAM='useExisting=true' + REQUESTID=dcae-pm-mapper-dmaap-provisioning + URL=https://dmaap-bc:8443/webapi/ + CA_PEM=ca.pem + KEY_PEM=key.pem + CLIENT_PEM=client.pem + PEM_DIR=/opt/app/osaaf/local + CERT_PWD='2U[iOZzMHI:.#tdCwlBqc;}S' + BA_PWD='demo123456!' + AUTH_METHOD=basicAuth + BA_IDENTITY=dmaap-bc@dmaap-bc.onap.org + init_config + '[' '!' -d /opt/app/dbc-client -a '!' -d /opt/app/config ] + cd /opt/app/osaaf/local + '[' basicAuth '=' basicAuth ] + echo '-u dmaap-bc@dmaap-bc.onap.org:demo123456!' + CURL_CRED='-K /opt/app/osaaf/local/curl.cred' + init_dbc_provisioning + cd /opt/app/config + '[' -d dmaap ] + '[' -d dcaeLocations ] + '[' -d mr_clusters ] + '[' -d topics ] + ls topics/topicsConfig-0.json + do_http_post topics/topicsConfig-0.json topics + RETRY_TIME=60 + '[' -n /opt/app/config/cache ] + echo+ topicsConfig-0.json cut -d . -f1 + RESP=/opt/app/config/cache/topicsConfig-0-resp.json + true + '[' topics '!=' feeds -a topics '!=' topics ] + xcurl -o /opt/app/config/cache/topicsConfig-0-resp.json -d @topics/topicsConfig-0.json 'https://dmaap-bc:8443/webapi/topics/?useExisting=true' + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dcae-pm-mapper-dmaap-provisioning' -H 'Content-Type: application/json' -o /opt/app/config/cache/topicsConfig-0-resp.json -d @topics/topicsConfig-0.json 'https://dmaap-bc:8443/webapi/topics/?useExisting=true' + rc=201 + '[' 201 '=' 200 -o 201 '=' 201 -o 201 '=' 409 ] + echo 'Http Post request for feed creation is successful with response code=201' + break + '[' -d mr_clients ] + '[' -d dr_nodes ] + '[' -d feeds ] Http Post request for feed creation is successful with response code=201 + ls feeds/feedConfig-0.json + do_http_post feeds/feedConfig-0.json feeds + RETRY_TIME=60 + '[' -n /opt/app/config/cache ] + echo feedConfig-0.json + cut -d . -f1 + RESP=/opt/app/config/cache/feedConfig-0-resp.json + true + '[' feeds '!=' feeds -a feeds '!=' topics ] + xcurl -o /opt/app/config/cache/feedConfig-0-resp.json -d @feeds/feedConfig-0.json 'https://dmaap-bc:8443/webapi/feeds/?useExisting=true' + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dcae-pm-mapper-dmaap-provisioning' -H 'Content-Type: application/json' -o /opt/app/config/cache/feedConfig-0-resp.json -d @feeds/feedConfig-0.json 'https://dmaap-bc:8443/webapi/feeds/?useExisting=true' + rc=200 + '[' 200 '=' 200 -o 200 '=' 201 -o 200 '=' 409 ] + echo 'Http Post request for feed creation is successful with response code=200' + break + '[' -d dr_pubs ] + '[' -d dr_subs ] Http Post request for feed creation is successful with response code=200 + ls dr_subs/drsubConfig-0.json + do_http_post dr_subs/drsubConfig-0.json dr_subs + RETRY_TIME=60 + '[' -n /opt/app/config/cache ] + echo drsubConfig-0.json + cut -d . -f1 + RESP=/opt/app/config/cache/drsubConfig-0-resp.json + true + '[' dr_subs '!=' feeds -a dr_subs '!=' topics ] + cat dr_subs/drsubConfig-0.json + envsubst + json_input='{"dcaeLocationName":"loc00","decompress":true,"deliveryURL":"https://dcae-pm-mapper:8443/delivery","feedName":"bulk_pm_feed","privilegedSubscriber":true,"username":"username","userpwd":"password"}' + xcurl -o /opt/app/config/cache/drsubConfig-0-resp.json -d '{"dcaeLocationName":"loc00","decompress":true,"deliveryURL":"https://dcae-pm-mapper:8443/delivery","feedName":"bulk_pm_feed","privilegedSubscriber":true,"username":"username","userpwd":"password"}' https://dmaap-bc:8443/webapi/dr_subs + curl -X POST -s '-K /opt/app/osaaf/local/curl.cred' -w '%{http_code}' -H 'X-ECOMP-RequestID: dcae-pm-mapper-dmaap-provisioning' -H 'Content-Type: application/json' -o /opt/app/config/cache/drsubConfig-0-resp.json -d '{"dcaeLocationName":"loc00","decompress":true,"deliveryURL":"https://dcae-pm-mapper:8443/delivery","feedName":"bulk_pm_feed","privilegedSubscriber":true,"username":"username","userpwd":"password"}' https://dmaap-bc:8443/webapi/dr_subs Http Post request is successful with response code=201 + rc=201 + '[' 201 '=' 200 -o 201 '=' 201 -o 201 '=' 409 ] + echo 'Http Post request is successful with response code=201' + break |
Open Issues
Data Router Publisher/Subscriber behaviour is not consistent
...
4. GET Request for DR Feeds, MR Topics from Bus Controller API.
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
1. >> GET : https://172.16.0.70:30242/webapi/feeds/
[
{
"lastMod": 1625906852119,
"status": "VALID",
"feedId": "1",
"feedName": "bulk_pm_feed",
"feedVersion": "0",
"feedDescription": "DFC Feed Creation",
"owner": "dcaecm",
"asprClassification": "unclassified",
"publishURL": "https://dmaap-dr-prov/publish/1",
"subscribeURL": "https://dmaap-dr-prov/subscribe/1",
"suspended": false,
"logURL": "https://dmaap-dr-prov/feedlog/1",
"formatUuid": null,
"pubs": [
{
"lastMod": 1625906852105,
"status": "VALID",
"dcaeLocationName": "loc00",
"username": "tmp_edftprplxki85pw",
"userpwd": "gi6nvqz4skzepa1",
"feedId": "1",
"pubId": "1.da72f",
"feedName": null,
"feedVersion": null,
"statusValid": true
},
{
"lastMod": 1625906851630,
"status": "VALID",
"dcaeLocationName": "san-francisco",
"username": "tmp_rmu408ke6c47cjd",
"userpwd": "3ae2ahymjl6sndw",
"feedId": "1",
"pubId": "1.9hl81",
"feedName": null,
"feedVersion": null,
"statusValid": true
}
],
"subs": [
{
"lastMod": 1625906974644,
"status": "VALID",
"dcaeLocationName": "loc00",
"username": "username",
"userpwd": "password",
"feedId": "1",
"deliveryURL": "https://dcae-pm-mapper:8443/delivery",
"logURL": "https://dmaap-dr-prov/sublog/1",
"subId": "1",
"use100": false,
"suspended": false,
"owner": "DGL",
"guaranteedDelivery": false,
"guaranteedSequence": false,
"privilegedSubscriber": true,
"decompress": true,
"feedName": null,
"feedVersion": null,
"statusValid": true
}
],
"bytes": "eyJuYW1lIjogImJ1bGtfcG1fZmVlZCIsICJ2ZXJzaW9uIjogIjAiLCAiZGVzY3JpcHRpb24iOiAiREZDIEZlZWQgQ3JlYXRpb24iLCAic3VzcGVuZCI6IGZhbHNlLCAiYXV0aG9yaXphdGlvbiI6IHsgImNsYXNzaWZpY2F0aW9uIjogInVuY2xhc3NpZmllZCIsICJlbmRwb2ludF9hZGRycyI6IFtdLCJlbmRwb2ludF9pZHMiOiBbCXsiaWQiOiAidG1wX2VkZnRwcnBseGtpODVwdyIsInBhc3N3b3JkIjogImdpNm52cXo0c2t6ZXBhMSJ9CSx7ImlkIjogInRtcF9ybXU0MDhrZTZjNDdjamQiLCJwYXNzd29yZCI6ICIzYWUyYWh5bWpsNnNuZHcifV19fQ==",
"statusValid": true
}
]
2. >> GET Topics : https://172.16.0.70:30242/webapi/topics/
[
{
"lastMod": 1625906819480,
"status": "VALID",
"fqtn": "org.onap.dmaap.mr.PNF_REGISTRATION",
"topicName": "PNF_REGISTRATION",
"topicDescription": "the VES collector will be publishing pnfRegistration events in this topic",
"tnxEnabled": "false",
"owner": "VEScollector",
"formatUuid": null,
"replicationCase": "REPLICATION_NONE",
"globalMrURL": null,
"fqtnStyle": null,
"version": null,
"partitionCount": "2",
"replicationCount": "1",
"publisherRole": "org.onap.dmaap.mr.PNF_REGISTRATION.publisher",
"subscriberRole": "org.onap.dmaap.mr.PNF_REGISTRATION.subscriber",
"clients": [
{
"lastMod": 1625906820030,
"status": "VALID",
"dcaeLocationName": "san-francisco",
"topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.PNF_REGISTRATION",
"fqtn": "org.onap.dmaap.mr.PNF_REGISTRATION",
"clientRole": "org.onap.dmaap.mr.PNF_REGISTRATION.sub",
"action": [
"sub",
"view"
],
"mrClientId": "1625906820030",
"clientIdentity": null,
"publisher": false,
"subscriber": true,
"statusValid": true
}
],
"numClients": 1,
"bytes": "eyAidG9waWNOYW1lIjogIm9yZy5vbmFwLmRtYWFwLm1yLlBORl9SRUdJU1RSQVRJT04iLCAidG9waWNEZXNjcmlwdGlvbiI6ICJ0aGUgVkVTIGNvbGxlY3RvciB3aWxsIGJlIHB1Ymxpc2hpbmcgcG5mUmVnaXN0cmF0aW9uIGV2ZW50cyBpbiB0aGlzIHRvcGljIiwgInBhcnRpdGlvbkNvdW50IjogIjIiLCAicmVwbGljYXRpb25Db3VudCI6ICIxIiB9IA==",
"statusValid": true
},
{
"lastMod": 1625906820603,
"status": "VALID",
"fqtn": "org.onap.dmaap.mr.mirrormakeragent",
"topicName": "mirrormakeragent",
"topicDescription": "the topic used to provision the MM agent whitelist",
"tnxEnabled": "false",
"owner": "dmaap",
"formatUuid": null,
"replicationCase": "REPLICATION_NONE",
"globalMrURL": null,
"fqtnStyle": null,
"version": null,
"partitionCount": "1",
"replicationCount": "1",
"publisherRole": "org.onap.dmaap.mr.mirrormakeragent.publisher",
"subscriberRole": "org.onap.dmaap.mr.mirrormakeragent.subscriber",
"clients": [
{
"lastMod": 1625906821157,
"status": "VALID",
"dcaeLocationName": "san-francisco",
"topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.mirrormakeragent",
"fqtn": "org.onap.dmaap.mr.mirrormakeragent",
"clientRole": null,
"action": [
"pub",
"sub",
"view"
],
"mrClientId": "1625906821157",
"clientIdentity": "dmaap-bc-mm-prov@dmaap-bc-mm-prov.onap.org",
"publisher": true,
"subscriber": true,
"statusValid": true
},
{
"lastMod": 1625906827509,
"status": "VALID",
"dcaeLocationName": "san-francisco",
"topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.mirrormakeragent",
"fqtn": "org.onap.dmaap.mr.mirrormakeragent",
"clientRole": null,
"action": [
"pub",
"sub",
"view"
],
"mrClientId": "1625906827509",
"clientIdentity": "demo@people.osaaf.org",
"publisher": true,
"subscriber": true,
"statusValid": true
},
{
"lastMod": 1625906826491,
"status": "VALID",
"dcaeLocationName": "san-francisco",
"topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.mirrormakeragent",
"fqtn": "org.onap.dmaap.mr.mirrormakeragent",
"clientRole": null,
"action": [
"pub",
"sub",
"view"
],
"mrClientId": "1625906826491",
"clientIdentity": "dmaap-bc-topic-mgr@dmaap-bc-topic-mgr.onap.org",
"publisher": true,
"subscriber": true,
"statusValid": true
}
],
"numClients": 3,
"bytes": "eyAidG9waWNOYW1lIjogIm9yZy5vbmFwLmRtYWFwLm1yLm1pcnJvcm1ha2VyYWdlbnQiLCAidG9waWNEZXNjcmlwdGlvbiI6ICJ0aGUgdG9waWMgdXNlZCB0byBwcm92aXNpb24gdGhlIE1NIGFnZW50IHdoaXRlbGlzdCIsICJwYXJ0aXRpb25Db3VudCI6ICIxIiwgInJlcGxpY2F0aW9uQ291bnQiOiAiMSIgfSA=",
"statusValid": true
},
{
"lastMod": 1625906971092,
"status": "VALID",
"fqtn": "org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS",
"topicName": "PERFORMANCE_MEASUREMENTS",
"topicDescription": "PM Mapper publishes perf3gpp VES PM Events to authenticated MR topic",
"tnxEnabled": "false",
"owner": "dcaecm",
"formatUuid": null,
"replicationCase": "REPLICATION_NONE",
"globalMrURL": null,
"fqtnStyle": null,
"version": null,
"partitionCount": "2",
"replicationCount": "1",
"publisherRole": "org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS.publisher",
"subscriberRole": "org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS.subscriber",
"clients": [
{
"lastMod": 1625906971956,
"status": "VALID",
"dcaeLocationName": "san-francisco",
"topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS",
"fqtn": "org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS",
"clientRole": "org.onap.dcae.pmPublisher",
"action": [
"pub",
"view"
],
"mrClientId": "1625906971956",
"clientIdentity": null,
"publisher": true,
"subscriber": false,
"statusValid": true
}
],
"numClients": 1,
"bytes": "eyAidG9waWNOYW1lIjogIm9yZy5vbmFwLmRtYWFwLm1yLlBFUkZPUk1BTkNFX01FQVNVUkVNRU5UUyIsICJ0b3BpY0Rlc2NyaXB0aW9uIjogIlBNIE1hcHBlciBwdWJsaXNoZXMgcGVyZjNncHAgVkVTIFBNIEV2ZW50cyB0byBhdXRoZW50aWNhdGVkIE1SIHRvcGljIiwgInBhcnRpdGlvbkNvdW50IjogIjIiLCAicmVwbGljYXRpb25Db3VudCI6ICIxIiB9IA==",
"statusValid": true
},
{
"lastMod": 1625906816791,
"status": "VALID",
"fqtn": "org.onap.dmaap.mr.PNF_READY",
"topicName": "PNF_READY",
"topicDescription": "This topic will be used to publish the PNF_READY events generated by the PNF REgistration Handler service in the DCAE platform.",
"tnxEnabled": "false",
"owner": "PNFRegistrationHandler",
"formatUuid": null,
"replicationCase": "REPLICATION_NONE",
"globalMrURL": null,
"fqtnStyle": null,
"version": null,
"partitionCount": "2",
"replicationCount": "1",
"publisherRole": "org.onap.dmaap.mr.PNF_READY.publisher",
"subscriberRole": "org.onap.dmaap.mr.PNF_READY.subscriber",
"clients": [
{
"lastMod": 1625906817506,
"status": "VALID",
"dcaeLocationName": "san-francisco",
"topicURL": "http://message-router:3904/events/org.onap.dmaap.mr.PNF_READY",
"fqtn": "org.onap.dmaap.mr.PNF_READY",
"clientRole": "org.onap.dmaap.mr.PNF_READY.pub",
"action": [
"pub",
"view"
],
"mrClientId": "1625906817506",
"clientIdentity": null,
"publisher": true,
"subscriber": false,
"statusValid": true
}
],
"numClients": 1,
"bytes": "eyAidG9waWNOYW1lIjogIm9yZy5vbmFwLmRtYWFwLm1yLlBORl9SRUFEWSIsICJ0b3BpY0Rlc2NyaXB0aW9uIjogIlRoaXMgdG9waWMgd2lsbCBiZSB1c2VkIHRvIHB1Ymxpc2ggdGhlIFBORl9SRUFEWSBldmVudHMgZ2VuZXJhdGVkIGJ5IHRoZSBQTkYgUkVnaXN0cmF0aW9uIEhhbmRsZXIgc2VydmljZSBpbiB0aGUgRENBRSBwbGF0Zm9ybS4iLCAicGFydGl0aW9uQ291bnQiOiAiMiIsICJyZXBsaWNhdGlvbkNvdW50IjogIjEiIH0g",
"statusValid": true
}
] |
Test details
- Create Sftp server.
- Upload PM Files to xNF SFTP Server.
- Send File Ready Event to VES Collector.
- Verify Ves Collector logs for Event send to unauthenticated.VES_NOTIFICATION_OUTPUT : Ves Collector log file.
- Verify Data File Collector Logs if it contains message > Publishing file A1625946870.67.xml.gz to DR successful! : DFC log file.
- Vefify PM-Mapper log if it contains > org.onap.dcaegen2.services.pmmapper.messagerouter.VESPublisher. Successfully published VES events to messagerouter : Pm-Mapper log file.
- Verify Message-Router log if it contains Publisher Log Details : [publisherId=dcae, topicId=org.onap.dmaap.mr.PERFORMANCE_MEASUREMENTS, messageTimestamp=null, publisherIp=10.42.6.27, messageBatchId=10072021195549, messageSequence=1, messageLengthInBytes=2530, transactionEnabled=true, transactionId=10-07-2021::07:55:44:917::10.42.6.27::10072021195549::1, publishTimestamp=10-07-2021::07:55:44:918, serverIp=10.42.3.28]
Improvements
- Datafile Collector not sending dr publisher credentails as secret.