Page Comparison

Create 'bulk' version for org.onap.cps.ncmp.rest.controller.NetworkCmProxyController#getResourceDataForCmHandle to support multiple cm-handles (~ multiple anchors in in CPS-Core)

Table of Contents

References

Image RemovedCPS-1515 - Spike: Support multiple CM-Handles for NCMP Get Operation
CPS-NCMP ↔ DMI-Plugin Interface Details Jakarta-R10
CPS Events Structure

Requirements

Functional

...

Error Handling

...

Capabilities

...

Async response available on client topic
No delay in DMI PLugin (tested/measured using stub DMI Plugin)

...

Expand

CPS and NCMP

requests: cpu: 2000m memory: 2Gi limits: memory: 3Gi cpu: 3000m

2. Postgres

requests:
    cpu: 4000m
    memory: 1Gi
limits:
    memory: 3Gi
    cpu: 6000m

...

Out-of-scope

Support for multiple resource identifiers in one batch operation
Support cached data batch requests: only passthrough datastores will be supported (see decision #2)
NCMP does NOT keep track of request status to see if it is completed, amalgamate responses or anything like that. It wil simply forward responses from the internal topic to the client topic.
Access control

Assumptions

...

#

...

Assumption

...

Notes

...

Proprietary options or not in the scope of this analysis.

...

agreed 08 Mar 2023 with kieran mccarthy ; these are optional parameters (name-value pairs) not interpreted by NCMP but can be interpreted by proprietary plugins
In the future "scope" might become standardized instead of proprietary but that wil be achieved through a separate requirement

...

same xpath (resourceIdentifierInQuery) for all cm handles or different for each cm handle

...

agreed 08 Mar 2023 with kieran mccarthy ; if different resources are required on the same cm-handle the client wil send another (batch) request

...

options, resourceIdentifier is optional for bulk operations.
operation, datastore and cmhandleIds are mandatory fields

...

agreed 24 Apr 2023 with CPS team.

...

agreed 24 Apr 2023

...

Get batch operation does not support includeDescendants as it is impl. for

pass through datastore only

ncmp-datastore:passthrough-running and

ncmp-datastore:passthrough-operational

...

agreed on 15 May 2023

Issues & Decisions

...

#

...

Issue

...

Notes

...

Decision

...

Get
Create
Update (Put)
Patch
Delete

if many what is the priority?

...

agreed 08 Mar 2023 with kieran mccarthy

Only Get (read)

(in future other operations might be support batch option too)

...

Which datasources should be supported?

...

Do we need to support passthrough-only no-cached() data only ?
(maybe just start with that, support cached data bulk request later)

...

agreed 08 Mar 2023 with kieran mccarthy :

all passthrough datastores will be supported

Not implemented (yet) response, for non passthrough datastores

...

We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment. The proposal is that we should not distinguish batch from bulk or other flavors of read/write.

The aim is to only have a single flavor of interface for read or write for clients. Therefore the proposal is to drop “batch” from the interface URL and just act toward “data”

...

Agreed with team 23 May 2023 kieran mccarthy

POST http://localhost:8080/ncmp/v1/data&topic=my-topic-name

...

CPS prefers keep interface similar as single cm handle interface (consistency and cost)

Existing : ...&topic=topicParamInQuery

...

agreed 08 Mar 2023 with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency

...

support in ONAP DMI-plugin

...

agreed 08 Mar 2023 with Toine Siebelink : ONAP plugin can respond with not implemented yet code,

...

Response always Async ie. topic is compulsory ?

...

Agreed 08 Mar 2023 with kieran mccarthy :

Topic is optional but system will respond with 'Not implemented (yet when not specified or blank

...

Agreed 08 Mar 2023 with kieran mccarthy : NCMP wil only forward to client topic no handling tracking or any responses or status of request

...

Agreed 08 Mar 2023 with kieran mccarthy :
No response for 4b then send an error response to the topic given by client

...

Agreed 08 Mar 2023 with kieran mccarthy :
NCMP can log error when forward to 'client topic' Security not in scope (yet)

...

Suggestions

Silently ignore
(initial) error response
1. should we combine all errors in one message?

...

Agreed 08 Mar 2023 with kieran mccarthy :

Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI

...

Overlap/clash with Deutsche Telekom user story:

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1377

?!

...

08 Mar 2023 discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors)

...

Schema of bulk response

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1556

...

1) Agreement required on the structure of the response. Please see response structure below.
2) Does 'eventTime' field which holds the timestamp of the bulk response event, required or can it be dropped?

...

Need to follow schema structure there in #15 under Issues & Decisions section and it is agreed on 11 May 2023

...

~~Schema for Bulk Response event forwarding to client specified topic~~

Image RemovedCPS-1557 - NCMP : forward bulk response messages to client topic

...

~~Not keeping the 'eventTarget' which comes from the (BulkResponseEvent(In progress of the structure agreement)).~~

~~Note: Will consider 'eventTarget', If it finalized the schema from~~

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1556

...

Agreed on 11 May 2023 to keep one schema for both events (DMI → NCMP) and (NCMP → ClientApps)

...

How NCMP would forward response? (Response data : ref. message flow #6)

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1556

...

kieran mccarthy Confirmed in meeting 07 Jun 2023

...

correlationid
target

23 May 2023 kieran mccarthy

...

topic and requestId will be send as a path param from NCMP to DMI-plugin.

NCMP batch endpoint : http://localhost:8080/ncmp/v1/data?&topic=my-topic-name

DMI-Plugin batch endpoint : http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=e6fa4d26-4dc1-4877-aa3c-45e99f840708

Mandatory Fields:
- - "operation": "read"
  - "operationId": "12"
    "datastore": "ncmp-datastore:passthrough-operational"
  - "targetIds": [ "0df4d39af6514d99b816758148389cfd", "ec2e9495679a43c58659c07d87025e72" ]

Optional Fields:
- - "options": "(fields=schemas/schema)"
  - "resourceIdentifier": "parent/child"

Agreed 11 May 2023 with Csaba Kocsis and kieran mccarthy

...

event type proposal : from ~~BatchDataEvent~~ to

DataOperationEvent

For example : DataOperationResponseEvent

'eventType' value as below 'org.onap.cps.ncmp.events.~~Batch~~DataEvent'

proposed Operation like :

'org.onap.cps.ncmp.events.DataOperationEvent'

'batch' was the keyword use in the URL for this feature (now gone) but much code (classnames, variable names) etc uses this name still...

...

kieran mccarthy and Team agreed on 07 Jun 2023 and some cost (1-2 days) is acceptable

...

kieran mccarthy please confirm : we are using 'responseContent' as name of this field as 'data' is already used for cloud event standard payload.

Now it is like : data.responses[0].responseContent

Sourabh Sourabh Suggest to use "result". This is after all where the actual result output. the other tags in responses[x] are status related to the request :

data.responses[0].result

Event Format Definitions

Event Headers

...

Event Body (data)

The current proposal is to use the CNFC Event structure and library as detail in CPS Events Structure#CNCFCloudEventalignment
To prevent problems with large data request multiple response event will be sent for a successful read for each cm handle (id)
Error messages can contain information about single cm handle ids

...

Field

...

Type

...

Description

...

Mandatory?

...

Notes

...

data

...

Mandatory

...

data.responses[0, 1, 2, ...]

...

Mandatory

...

data.responses[0].operationId

...

Mandatory

...

data.responses[0].ids

...

Mandatory

...

Example : ["0df4d39af6514d99b816758148389cfd"]
Note: Ids array should contain only a single element in the array in case of success messages and In case of error it can have any number of elements.

...

data.responses[0].statusCode

...

Mandatory

...

Common NCMP defined error codes:

status-code 0-99 is reserved for any success response
status-code from 100 to 199 is reserved for any failed response.

...

data.responses[0].statusMessage

...

Mandatory

Examples for code & message :

statusCode	statusMessage
1	"Successfully applied changes"
101	"cmHandle(s) do not exist"

...

Optional

...

In case of success :
- Optional, for write operations then no need to return configurations
  application/yang-patch+json | application/yang-data+json
In case of failure :
- Optional, any supplementary error data matching the error status-code

Implementation

Bulk Request Message Flow

Gliffy

macroId	5966a68a-1641-430a-8c78-27c938f06e15
displayName	Cm Handle Batch Message Flow
name	Cm Handle Batch Interface Design
pagePin	21

Message Flow details

...

Flow Step

...

Short description

...

Message Details

...

Notes

...

Bulk Get Request

...

language	bash
title	NCMP batch endpoint
collapse	true

...

Create 'bulk' version for org.onap.cps.ncmp.rest.controller.NetworkCmProxyController#getResourceDataForCmHandle to support multiple cm-handles (~ multiple anchors in in CPS-Core)

Table of Contents

References

Image AddedCPS-1515 - Spike: Support multiple CM-Handles for NCMP Get Operation
CPS-NCMP ↔ DMI-Plugin Interface Details Jakarta-R10
CPS Events Structure

Requirements

Functional

#	Interface	Requirement	Additional Information
1	REST CPS(-NCMP)-E-05	Support batch read operation (new) using an asynchronous response on a client specified topic	payload includes list of cm handle ids
2	REST DMI-I-01	Support batch read operation (new) using an asynchronous response to an internal topic	payload includes list of (associated for this plugin) cm handles ids with their private (additional) properties

Error Handling

#	Error Scenario	Expected behavior
1	DMI Not respond to initial synchronous request with normal HTTP timeout	Special Error message (Kafka event) including affected, and reason for all the handles in the message cm handles send to client topic detailing cm-handles (per dmi plugin)
2	Topic not supplied on CPS-E-05	Return HTTP 501 (not implemented)
3	Client specided Topic is not configured	Log error message only
4	Non-existing cm-handle id	Similar message but different reason as specified #1 above

Capabilities

Excerpt

# Parameter Expectation Notes

1

Response Time 1 Batch request

<2 seconds (average)

Async response available on client topic
No delay in DMI Plugin (tested/measured using stub DMI Plugin)

2 Batch-size 200 cm handles No hardcoded limit

3 Response payload size ~2 KB per cm handles Performance test for capability should be tested with this average response size

4 Maximum registered #cm handles 20,000 This will effect the internal query time

5 Supported # DMI PLugins 10 This might effect processing times

6

Test Environment

Expand

CPS and NCMP

requests: cpu: 2000m memory: 2Gi limits: memory: 3Gi cpu: 3000m

2. Postgres

requests:
    cpu: 4000m
    memory: 1Gi
limits:
    memory: 3Gi
    cpu: 6000m

7 Concurrent request 12 12 clients requests toward 1 NCMP simultaneously

8 Request Frequency 100 request/min Should not affect performance, does not need to be tested

Out-of-scope

Support for multiple resource identifiers in one batch operation
Support cached data batch requests: only passthrough datastores will be supported (see decision #2)
NCMP does NOT keep track of request status to see if it is completed, amalgamate responses or anything like that. It wil simply forward responses from the internal topic to the client topic.
Access control

Assumptions

#	Assumption	Notes
1	Proprietary options or not in the scope of this analysis.	agreed 08 Mar 2023 with kieran mccarthy ; these are optional parameters (name-value pairs) not interpreted by NCMP but can be interpreted by proprietary plugins In the future "scope" might become standardized instead of proprietary but that wil be achieved through a separate requirement
2	same xpath (resourceIdentifierInQuery) for all cm handles or different for each cm handle	agreed 08 Mar 2023 with kieran mccarthy ; if different resources are required on the same cm-handle the client wil send another (batch) request
3	options, resourceIdentifier is optional for bulk operations. operation, datastore and cmhandleIds are mandatory fields	agreed 24 Apr 2023 with CPS team.
4	We are not merging any duplicate cm handles while sending request to dmi-plugin	agreed 24 Apr 2023
5	Get batch operation does not support includeDescendants as it is impl. for pass through datastore only ncmp-datastore:passthrough-running and ncmp-datastore:passthrough-operational	agreed on 15 May 2023

Issues & Decisions

#

Issue

Notes

Decision

1

Which operation(s) need support for multiple cm handles?

Get
Create
Update (Put)
Patch
Delete

if many what is the priority?

agreed 08 Mar 2023 with kieran mccarthy

Only Get (read)

(in future other operations might be support batch option too)

2

Which datasources should be supported?

Do we need to support passthrough-only no-cached() data only ?
(maybe just start with that, support cached data bulk request later)

agreed 08 Mar 2023 with kieran mccarthy :

all passthrough datastores will be supported

Not implemented (yet) response, for non passthrough datastores

3

URL pattern for NCMP bulk endpoints

We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment. The proposal is that we should not distinguish batch from bulk or other flavors of read/write.

The aim is to only have a single flavor of interface for read or write for clients. Therefore the proposal is to drop “batch” from the interface URL and just act toward “data”

Agreed with team 23 May 2023 kieran mccarthy

POST http://localhost:8080/ncmp/v1/data&topic=my-topic-name

4

keep datastore, topic and optional parameters in the URL itself instead into body.

CPS prefers keep interface similar as single cm handle interface (consistency and cost)

Existing : ...&topic=topicParamInQuery

agreed 08 Mar 2023 with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency

5

support in ONAP DMI-plugin

agreed 08 Mar 2023 with Toine Siebelink : ONAP plugin can respond with not implemented yet code,

6

Response always Async ie. topic is compulsory ?

Assume topic is compulsory (defined in OPenApi) → Response therefore wil be 400 if not supplied

Agreed 08 Mar 2023 with kieran mccarthy :

Topic is optional but system will respond with 'Not implemented (yet when not specified or blank

7

Should NCMP Amalgamate Async responses from DMI-Plugin before forwarding ?

step 6 in flow diagram

Agreed 08 Mar 2023 with kieran mccarthy : NCMP wil only forward to client topic no handling tracking or any responses or status of request

8

Handle non responding dmi-plugin

Agreed 08 Mar 2023 with kieran mccarthy :
No response for 4b then send an error response to the topic given by client

9

Should (can) NCMP check if 'MyTopic' specified by client exist

Consider Access Control too. For now NCMP can log error. Client is responsible for topic setup

Agreed 08 Mar 2023 with kieran mccarthy :
NCMP can log error when forward to 'client topic' Security not in scope (yet)

10

How to handle non-existing CM-Handles (id)

Suggestions

Silently ignore
(initial) error response
1. should we combine all errors in one message?

Agreed 08 Mar 2023 with kieran mccarthy :

Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI

11

Overlap/clash with Deutsche Telekom user story:

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1377

?!

08 Mar 2023 discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors)

12

Schema of bulk response

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1556

1) Agreement required on the structure of the response. Please see response structure below.
2) Does 'eventTime' field which holds the timestamp of the bulk response event, required or can it be dropped?

Need to follow schema structure there in #15 under Issues & Decisions section and it is agreed on 11 May 2023

13

~~Schema for Bulk Response event forwarding to client specified topic~~

Image AddedCPS-1557 - NCMP : forward bulk response messages to client topic

~~Not keeping the 'eventTarget' which comes from the (BulkResponseEvent(In progress of the structure agreement)).~~

~~Note: Will consider 'eventTarget', If it finalized the schema from~~

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1556

Agreed on 11 May 2023 to keep one schema for both events (DMI → NCMP) and (NCMP → ClientApps)

14

How NCMP would forward response? (Response data : ref. message flow #6)

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1556

1. Do we need to send only one response containing all the requested cm handles ?
2. Send single response for each cm handle? This could lead to to much data in the event for large batch requests. Csaba Kocsis , Sourabh Sourabh and Toine Siebelink proposed to us single event for each cm handle
3. Single response message per DMI plugin? Just is a list of IDS should be OK.

kieran mccarthy

kieran mccarthy Confirmed in meeting 07 Jun 2023

15

Message format for the batch interface

We have had some internal discussions including with some O-RAN standards representatives and one of the outcomes is that it would be good if we aligned with the some community standards for event header definitions. IT is proposed (initially from AT&T) that we should follow Cloud Native Computing Foundation (CNCF) specification as defined in their cloudevents incubator project.
Some non-standard headers wil need to be implemented as 'extensions' and names to be confirmed

correlationid
target

23 May 2023 kieran mccarthy

More details see CPS Events Structure#CNCFCloudEventalignment

16

NCMP will send only one request to each DMI

topic and requestId will be send as a path param from NCMP to DMI-plugin.

NCMP batch endpoint : http://localhost:8080/ncmp/v1/data?&topic=my-topic-name

DMI-Plugin batch endpoint : http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=e6fa4d26-4dc1-4877-aa3c-45e99f840708

Mandatory Fields:
- - "operation": "read"
  - "operationId": "12"
    "datastore": "ncmp-datastore:passthrough-operational"
  - "targetIds": [ "0df4d39af6514d99b816758148389cfd", "ec2e9495679a43c58659c07d87025e72" ]

Optional Fields:
- - "options": "(fields=schemas/schema)"
  - "resourceIdentifier": "parent/child"

Agreed 11 May 2023 with Csaba Kocsis and kieran mccarthy

17

Feature Name (events)

event type proposal : from ~~BatchDataEvent~~ to

DataOperationEvent

For example : DataOperationResponseEvent

'eventType' value as below 'org.onap.cps.ncmp.events.~~Batch~~DataEvent'

proposed Operation like :

'org.onap.cps.ncmp.events.DataOperationEvent'

'batch' was the keyword use in the URL for this feature (now gone) but much code (classnames, variable names) etc uses this name still...

kieran mccarthy and Team agreed on 07 Jun 2023 and some cost (1-2 days) is acceptable

18

What would be the field name of response data?

data.responses[0].data

kieran mccarthy please confirm : we are using 'responseContent' as name of this field as 'data' is already used for cloud event standard payload.

Now it is like : data.responses[0].responseContent

Sourabh Sourabh Suggest to use "result". This is after all where the actual result output. the other tags in responses[x] are status related to the request :

data.responses[0].result

19

Exact format of timestamp of cloud events.

CloudEvent documentation (https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md#time) refers on ISO8601 standard

(see org.onap.cps.ncmp.api.impl.utils.EventDateTimeFormatter )

Toine Siebelink Although not clearly communicated before ISO8601 is the standard to be used and it seems the current implementation in NCMP conforms to this standard

Event Format Definitions

Event Headers

Insert excerpt

	CPS Events Structure
	CPS Events Structure

Event Body (data)

The current proposal is to use the CNFC Event structure and library as detail in CPS Events Structure#CNCFCloudEventalignment
To prevent problems with large data request multiple response event will be sent for a successful read for each cm handle (id)
Error messages can contain information about single cm handle ids

Field

Type

Description

Mandatory?

Notes

data

Event

The payload of an event

Mandatory

Cloud event standard (data)

data.responses[0, 1, 2, ...]

Array

contains an array or batch response that includes both success and failure.

Mandatory

data.responses[0].operationId

String

specified to distinguish multiple operations using same cmhandleId

Mandatory

data.responses[0].ids

String

cmhandle-ids

Mandatory

Example : ["0df4d39af6514d99b816758148389cfd"]
Note: Ids array should contain only a single element in the array in case of success messages and In case of error it can have any number of elements.

data.responses[0].statusCode

String

Mandatory

Common NCMP defined error codes:

status-code 0-99 is reserved for any success response
status-code from 100 to 199 is reserved for any failed response.

data.responses[0].statusMessage

String

Mandatory

Examples for code & message :

statusCode	statusMessage
0	Successfully applied changes.
100	cm handle id(s) is(are) not found.
101	cm handle id(s) is(are) in non ready state.

data.responses[0].data

Object

Optional

In case of success :
- Optional, for write operations then no need to return configurations
  application/yang-patch+json | application/yang-data+json
In case of failure :
- Optional, any supplementary error data matching the error status-code

Implementation

Bulk Request Message Flow

Gliffy

macroId	5966a68a-1641-430a-8c78-27c938f06e15
displayName	Cm Handle Batch Message Flow
name	Cm Handle Batch Interface Design
pagePin	21

Message Flow details

PriorityComponentDescriptionJIRAsEstimateStatus1DMI, NCMPData operation response event (DMI → NCMP) to Comply with Cloud Events operation targetIds[836bb62201f34a7aa056a47bd95a81ed",202acb75b4a54e43bb1ff8c0c17a8e08]},{"operation":"read",operationId14datastore"ncmp-datastore:passthrough-running","targetIds":["ec2e9495679a43c58659c07d87025e72","0df4d39af6514d99b816758148389cfd" ] }'

Flow Step

Short description

Message Details

Notes

1

Bulk Get Request

Code Block

language	bash
title	NCMP batch endpoint
collapse	true

curl --location 'http: //localhost:8080/ncmp/v1/data?topic=my-topic-name' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic Y3BzdXNlcjpjcHNyMGNrcyE=' \
--data '{
    "operations": [
        {
            "operation": "read",
            "operationId": "12",
            "datastore": "ncmp-datastore:passthrough-operational",
            "options": "some option",
            "resourceIdentifier": "parent/child",
            "targetIds": [
                "836bb62201f34a7aa056a47bd95a81ed",
                "202acb75b4a54e43bb1ff8c0c17a8e08"
            ]
        },
        {
            "operation": "read",
            "operationId": "14",
            "datastore": "ncmp-datastore:passthrough-running",
            "targetIds": [
                "ec2e9495679a43c58659c07d87025e72",
                "0df4d39af6514d99b816758148389cfd"
            ]
        }
    ]
}'

Define new get operation "getResourceDataForCmHandles" into ncmp.yml

2

Ack client Request

Code Block

language	bash
title	NCMP batch endpoint response (202->Accepted)
collapse	true

{
    "requestId": "4753fc1f-7de2-449a-b306-a6204b5370b3"
}

3

DMI Bulk Request

Code Block

language	bash
title	DMI service batch endpoint
collapse	true

DMI Service 1 (POST): http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> 

[
    {
        "operationType": "read",
        "operationId": "14",
        "datastore": "ncmp-datastore:passthrough-running",
        "cmHandles": [
            {
                "id": "ec2e9495679a43c58659c07d87025e72",
                "cmHandleProperties": {
                    "neType": "RadioNode"
                }
            },
            {
                "id": "0df4d39af6514d99b816758148389cfd",
                "cmHandleProperties": {
                    "neType": "RadioNode"
                }
            }
        ]
    }
]

DMI Service 2 (POST) : http://172.26.202.26:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> 

   [
    {

operationType": "read",


        "operationId": "12",

"datastore": "ncmp-datastore:passthrough-operational",


        "options": "some option",


        "resourceIdentifier": "parent/child",
        "cmHandles": [
            {
                "

id":

"836bb62201f34a7aa056a47bd95a81ed",
                "

cmHandleProperties": {
                    "neType": "

RadioNode"

},

id": "

202acb75b4a54e43bb1ff8c0c17a8e08",
                "

cmHandleProperties":

       "neType": "RadioNode"

    }
        ]

Define new get operation "getResourceDataForCmHandles" into ncmp.yml

2

Ack client Request

Code Block

language	bash
title	NCMP batch endpoint response (202->Accepted)
collapse	true

{
    "requestId": "4753fc1f-7de2-449a-b306-a6204b5370b3"
}

3DMI Bulk Request

Code Block

language	bash
title	DMI service batch endpoint
collapse	true

DMI Service 1 (POST): http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> 

[
    {
        "operationType": "read",
        "operationId": "14",
        "datastore": ]

08 Aug 2023 New JIRA

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1833

is created to address it.

Discussed with Csaba Kocsis and agreed to send identical data request body as NCMP where data operation requat body is as follows:

Client App→ NCMP	NCMP→ DMI	Agreed
operation	operationType	operation
operation details are wrapped into parent attribute "operations" like "operations": [ ...	operation details ar not wrapped	wrap operation details into parent attribute "operations"

NCMP→ DMI	Current NCMP→ DMI	Proposed NCMP→ DMI
{ "operations": [ { "operation": "read", "operationId": "operational-10", "datastore": "ncmp-datastore:passthrough-running", "resourceIdentifier": "parent/child", "targetIds": [ "cd4473a732724092a97cebe157604a83" ] } ] }	[ { "operationType": "read", "operationId": "operational-10", "datastore": "ncmp-datastore:passthrough-running",

"cmHandles":

[ { "id": "ec2e9495679a43c58659c07d87025e72", "cmHandleProperties": { "neType": "RadioNode" } }, { "id": "0df4d39af6514d99b816758148389cfd", "cmHandleProperties": { "neType": "RadioNode" } } ] } ] DMI Service 2 (POST) : http://172.26.202.26:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> [ { "operationType": "read", "operationId": "12",

[
            {
                "id": "cd4473a732724092a97cebe157604a83",
                "cmHandleProperties": {
                    "neType": "RadioNode"
                }
            }
        ]
    }
]

{
    "operations": [
        {
            "operation": "read",
            "operationId": "operational-10",
            "datastore": "ncmp-datastore:passthrough-running",
            "resourceIdentifier": "parent/child",
            "cmHandles": [
                {
                    "id": "cd4473a732724092a97cebe157604a83",
                    "cmHandleProperties": {
                        "neType": "RadioNode"
                    }
                }
            ]
        }
    ]
}

4Ack NCMP Request

Code Block

language	java
title	Response 202
collapse	true

202 is non-committal, meaning that there is no way for the HTTP to later send an asynchronous response indicating the outcome of processing the request. It is intended for cases where another process or server handles the request, or for batch processing.5Kafka Messages from DMI to NCMP

Code Block

title	Batch Event Payload Schema
collapse	true

{
  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "$id": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",
  "$ref": "#/definitions/BatchDataResponseEvent",
  "definitions": {
    "BatchDataResponseEvent": {
      "description": "The payload of batch event.",
      "datastoretype": "ncmp-datastore:passthrough-operationalobject",

       "optionsjavaType" : "some option",
 org.onap.cps.ncmp.events.async.v1.BatchEvent",
      "resourceIdentifierproperties": "parent/child",{
        "cmHandlesevent": [{
          "description": "The {content of     Batch event.",
          "idtype": "836bb62201f34a7aa056a47bd95a81edobject",
                "cmHandleProperties"existingJavaType": {"java.lang.Object",
          "additionalProperties": false
        "neType": "RadioNode"}
      },
      "required": [
  }      "event"
      }],
      "additionalProperties": false
    }
 { }
}

Code Block

title	Batch Event Payload
collapse	true

{     
  "event": {
     "

id

payload": "

202acb75b4a54e43bb1ff8c0c17a8e08",

response of batch cm handles"
  }

6Kafka Message(s) from NCMP to Client

Code Block

title	Batch Event Payload
collapse	true

{     
  "cmHandlePropertiesevent": {
     "payload": "response of batch cm handles"
          "neType": "RadioNode"
                }
            }
        ]
    }
]

4Ack NCMP Request Code Block

language	java
title	Response 202
collapse	true

202 is non-committal, meaning that there is no way for the HTTP to later send an asynchronous response indicating the outcome of processing the request. It is intended for cases where another process or server handles the request, or for batch processing.5Kafka Messages from DMI to NCMP

Code Block

title	Batch Event Payload Schema
collapse	true

{
  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "$id": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",
  "$ref": "#/definitions/BatchDataResponseEvent",
  "definitions": {
    "BatchDataResponseEvent": {
      "description": "The payload of batch event.",
      "type": "object",
      "javaType" : "org.onap.cps.ncmp.events.async.v1.BatchEvent",
      "properties": {
        "event": {
          "description": "The content of Batch event.",
          "type": "object",
          "existingJavaType": "java.lang.Object",
          "additionalProperties": false
        }
      },
      "required": [
        "event"
      ],
      "additionalProperties": false
    }
  }
}

Code Block

title	Batch Event Payload
collapse	true

{     
  "event": {
     "payload": "response of batch cm handles"
  } 
}

6Kafka Message(s) from NCMP to Client

Code Block

title	Batch Event Payload
collapse	true

{     
  "event": {
     "payload": "response of batch cm handles"
  } 
}

Single response format for all scenarios bot positive and error, just using optional fields instead7Error for 4/5 → Non responding DMI, non existing CM Handles or any other error. NCMP will have to create error message detailing all cm-handle ids affect by that error.

Code Block

title	Non responding DMI-plugin
collapse	true

{
  "timestamp":"2023-03-01T23:00:00.345-0400", 
  "requestId":123,
  "error": "<error-message>",
  "Cmhandles":["cm-1",...,"cm-n"]
  }

Single response format for all scenarios bot positive and error, just using optional fields instead
See decision # 8

Proposed JIRAs :

} 
}

Single response format for all scenarios bot positive and error, just using optional fields instead

7

Error for 4/5 → Non responding DMI, non existing CM Handles or any other error. NCMP will have to create error message detailing all cm-handle ids affect by that error.

Code Block

title	Non responding DMI-plugin
collapse	true

{
  "timestamp":"2023-03-01T23:00:00.345-0400", 
  "requestId":123,
  "error": "<error-message>",
  "Cmhandles":["cm-1",...,"cm-n"]
  }

Single response format for all scenarios bot positive and error, just using optional fields instead
See decision # 8

Proposed JIRAs :

Expose REST endpoint to accept collection of cm handles for GET operation (Passthrough only)key,summary,type,created,updated,due,assignee,reporter,priority,status,resolutionDone

15571558

Priority

Component

Description

JIRAs

Estimate

Status

1

DMI, NCMP

Data operation response event (DMI → NCMP) to Comply with Cloud Events

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1723

5 Days (M)

Done

2

NCMP

Data operation response event (NCMP → Client App)

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1724

TBD

Done

3

DMI-Plugin

Accept datastore name as param into URL

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1550

5 Days

Done

4

NCMP

Expose REST endpoint to accept collection of cm handles for GET operation (Passthrough only)

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1553

15 Days

Done

5

DMI-Plugin

Expose endpoint for ONAP not impl. and Stub impl. for testing/demo

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1555

5 Days

Done

6

NCMP

NCMP: Update existing REST endpoint that accepts data operation request for GET operation

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-17231635

5 Days (M)

New Requirement.

2

NCMP

Data operation response event (NCMP → Client App)

Done

7

DMI-Plugin

DMI-Plugin : Update endpoint to accept data operation request

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-17241636

TBD 3 Days

Done

38

Stubbed DMI-Plugin

Accept datastore name as param into URL

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1550

5 Days

Done

4

NCMP

Include code to send response messages to internal kafka topic with delay

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-15531556

15 10 Days

Delivered , Pending task id RTD document only

5

DMI-Plugin

Expose endpoint for ONAP not impl. and Stub impl. for testing/demo

Done

9

NCMP

Forward response messages to client given kafka topic

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-15551557

5 Days

Done

610

NCMP

NCMP: Update existing REST endpoint that accepts data operation request for GET operationNCMP

Handle non-existing cm handles

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-16351583

5 Days

new because Requirements changed
+5 days

7

DMI-Plugin

DMI-Plugin : Update endpoint to accept data operation request

Jira Legacy

server

System Jira

columnIds

issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution

columns

11

NCMP

Error handling for non-ready cm handle state

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-16361614

3 5 Days

Done

12

8NCMP Stubbed

Handle non responding DMI-Plugin

Include code to send response messages to internal kafka topic with delay

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-15561558

10 5 Days

Open

Done

9

NCMP

Forward response messages to client given kafka topic13

CSIT test for demo

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-

1560

5 Days

Done

10

NCMP

Handle non-existing cm handles14

Update RTD

Error handling for non-ready cm handle state

Jira Legacy

server	System Jira
columnIds	issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1583

5 Days

11

NCMP

Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1785

4 Daya

Done

https://docs.onap.org/projects/onap-cps/en/latest/modeling.html

https://docs.onap.org/projects/onap-cps/en/latest/cps-events.html

https://docs.onap.org/projects/onap-cps/en/latest/data-operation-events.html

https://docs.onap.org/projects/onap-cps/en/latest/cps-ncmp-message-status-codes.html

15

Test performance

Capabilities

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-16141786

5 Days

12

NCMP

Handle non responding DMI-Plugin3 Days

Done

(NCMP: Read data operation resource API Performance for multiple cm-handles)

16

NCMP

Modify DMI data operation request body.

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-

1833

5 Days

13

CSIT test for demo

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	CPS-1560

5 Days

Planning :

...

2 Days

Done

Planning :

Allow for 2 more user stories each may take 1 weeks
Based on 2 resource working parallelly may take approx.. ~~8 weeks from 29 March 2023~~
(updated after additional items where added) The estimated date for completion is 28 Jun 2023
The release for cps-and-ncmp 3.3.3 version is completed on Friday, 30 Jun 2023
Release tag : https://gerrit.onap.org/r/gitweb?p=cps.git;a=tag;h=refs%2Ftags%2F3.3.3
Demo recording can be referred here : https://lf-onap.atlassian.net/wiki/display/DW/CPS+User+Story+Demos

Versions Compared

Old Version 174

New Version Current

Key

References

Requirements

Functional

Error Handling

Capabilities

Out-of-scope

Assumptions

Issues & Decisions

Event Format Definitions

Event Headers

Event Body (data)

Implementation

Bulk Request Message Flow

Message Flow details

References

Requirements

Functional

Error Handling

Capabilities

Out-of-scope

Assumptions

Issues & Decisions

Event Format Definitions

Event Headers

Event Body (data)

Implementation

Bulk Request Message Flow

Message Flow details

Proposed JIRAs :

Proposed JIRAs :

Planning :

Planning :