CPS-1515: Spike: Support Multiple CM-Handles for NCMP Get Operation (Data operations)

1

REST CPS(-NCMP)-E-05

Support batch read operation (new) using an asynchronous response on a client specified topic

payload includes list of cm handle ids

2

REST DMI-I-01

Support batch read operation (new) using an asynchronous response to an internal topic

payload includes list of (associated for this plugin) cm handles ids with their private (additional) properties

1

DMI Not respond to initial synchronous request with normal HTTP timeout

Special Error message (Kafka event)  including affected, and reason for all the handles in the message cm handles send to client topic detailing cm-handles
(per dmi plugin)

2

Topic not supplied on CPS-E-05

Return HTTP 501 (not implemented)

3

Client specided Topic is not configured

Log error message only

4

Non-existing cm-handle id

Similar message but different reason as specified #1 above

1

Proprietary options or not in the scope of this analysis.

agreed Mar 8, 2023 with @kieran mccarthy ; these are optional parameters (name-value pairs) not interpreted by NCMP but can be interpreted by proprietary plugins
In the future "scope" might become standardized instead of proprietary but that wil be achieved through a separate requirement 

2

same xpath (resourceIdentifierInQuery) for all cm handles or different for each cm handle

agreed Mar 8, 2023 with @kieran mccarthy ; if different resources are required on the same cm-handle the client wil send another (batch) request

3

• options, resourceIdentifier is optional for bulk operations.

• operation, datastore and cmhandleIds are mandatory fields

agreed Apr 24, 2023 with CPS team.

4

We are not merging any duplicate cm handles while sending request to dmi-plugin

agreed Apr 24, 2023 

5

Get batch operation does not support includeDescendants as it is impl. for 

pass through datastore only

ncmp-datastore:passthrough-running and

ncmp-datastore:passthrough-operational

agreed on May 15, 2023 

1

Which operation(s) need support for multiple cm handles?

1. Get

2. Create

3. Update (Put)

4. Patch

5. Delete

if many what is the priority?

agreed Mar 8, 2023 with @kieran mccarthy 

Only Get (read)

(in future other operations might be support batch option too)

2

Which datasources should be supported?

Do we need to support passthrough-only no-cached() data only ?
(maybe just start with that, support cached data bulk request later)

agreed Mar 8, 2023 with @kieran mccarthy :

all passthrough datastores will be supported

Not implemented (yet) response, for non passthrough datastores 

3

URL  pattern for NCMP bulk endpoints

We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment.  The proposal is that we should not distinguish batch from bulk or other flavors of read/write. 

The aim is to only have a single flavor of interface for read or write for clients.  Therefore the proposal is to drop “batch” from the interface URL and just act toward “data”

Agreed with team May 23, 2023 @kieran mccarthy 

POST http://localhost:8080/ncmp/v1/data&topic=my-topic-name

4

keep datastore, topic and optional parameters in the URL itself instead into body.

CPS prefers keep interface similar as single cm handle interface (consistency and cost)

Existing : ...&topic=topicParamInQuery

agreed Mar 8, 2023 with @kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency

5

support in ONAP DMI-plugin

agreed Mar 8, 2023 with @Toine Siebelink : ONAP plugin can respond with not implemented yet code,

6

Response always Async ie. topic is compulsory ?

Assume topic is compulsory (defined in OPenApi) → Response therefore wil be 400 if not supplied

Agreed Mar 8, 2023 with @kieran mccarthy :

Topic is optional but system will respond with 'Not implemented (yet when not specified or blank

7

Should NCMP Amalgamate Async responses from DMI-Plugin before forwarding ?

step 6 in flow diagram

Agreed Mar 8, 2023 with @kieran mccarthy : NCMP wil only forward to client topic no handling tracking or any responses or status of request

8

Handle non responding dmi-plugin

Agreed Mar 8, 2023 with @kieran mccarthy :
No response for 4b then send an error response to the topic given by client

9

Should (can) NCMP check if 'MyTopic' specified by client exist

Consider Access Control too. For now NCMP can log error. Client is responsible for topic setup

Agreed Mar 8, 2023 with @kieran mccarthy :
NCMP can log error when forward to 'client topic' Security not in scope (yet)

10

How to handle non-existing CM-Handles (id)

Suggestions

1. Silently ignore

2. (initial) error response

   1. should we combine all errors in one message?

Agreed Mar 8, 2023 with @kieran mccarthy :

Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI

11

Overlap/clash with Deutsche Telekom user story: https://lf-onap.atlassian.net/browse/CPS-1377?!

Mar 8, 2023 discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached  only?!) instead of a given list of cm-handles (~anchors)

12

Schema of bulk response https://lf-onap.atlassian.net/browse/CPS-1556

1) Agreement required on the structure of the response.  Please see response structure below.
2) Does 'eventTime' field which holds the timestamp of the bulk response event,  required or can it be dropped?

Need to follow schema structure there in  #15 under Issues & Decisions section and it  is  agreed on May 11, 2023   

13

Schema for Bulk Response event forwarding to client specified  topic

Not keeping the 'eventTarget' which comes from the (BulkResponseEvent(In progress of the structure agreement)).

Note:  Will consider  'eventTarget', If it finalized the schema from https://lf-onap.atlassian.net/browse/CPS-1556

Agreed on May 11, 2023 to keep one schema for both events (DMI → NCMP) and (NCMP → ClientApps)

14

How NCMP would forward response? (Response data : ref.  message flow #6)

https://lf-onap.atlassian.net/browse/CPS-1556

    1. Do we need to send only one response containing all the requested cm handles ?
    2. Send single response for each cm handle? This could lead to to much data in the event for large batch requests. @Csaba Kocsis , @Sourabh Sourabh and @Toine Siebelink proposed to us single event for each cm handle
    3. Single response message per DMI plugin? Just is a list of IDS should be OK.

@kieran mccarthy

@kieran mccarthy  Confirmed in meeting Jun 7, 2023 

15

Message format for the batch interface

We have had some internal discussions including with some O-RAN standards representatives and one of the outcomes is that it would be good if we aligned with the some community standards for event header definitions.  IT is proposed (initially from AT&T) that we should follow Cloud Native Computing Foundation (CNCF) specification as defined in their cloudevents incubator project.
Some non-standard headers wil need to be implemented as 'extensions' and names to be confirmed

• correlationid

• target

May 23, 2023 @kieran mccarthy 

More details see CPS Events Structure#CNCFCloudEventalignment

16

NCMP will send only one request to each DMI

topic and requestId will be send as a path param from NCMP to DMI-plugin.

NCMP batch endpoint : http://localhost:8080/ncmp/v1/data?&topic=my-topic-name

DMI-Plugin batch endpoint : http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=e6fa4d26-4dc1-4877-aa3c-45e99f840708

• Mandatory Fields:

  •  

    • "operation": "read"

    • "operationId": "12"
      "datastore": "ncmp-datastore:passthrough-operational"

    • "targetIds": [ "0df4d39af6514d99b816758148389cfd", "ec2e9495679a43c58659c07d87025e72" ]

• Optional Fields: 

  •  

    • "options": "(fields=schemas/schema)"

    • "resourceIdentifier": "parent/child"

Agreed May 11, 2023 with @Csaba Kocsis  and @kieran mccarthy

17

Feature Name (events)

event type proposal : from BatchDataEvent  to

DataOperationEvent

For example : DataOperationResponseEvent 

 'eventType' value as below 'org.onap.cps.ncmp.events.BatchDataEvent' 

proposed Operation like :

'org.onap.cps.ncmp.events.DataOperationEvent' 

'batch' was the keyword use in the URL for this feature (now gone) but much code (classnames, variable names) etc uses this name still...

@kieran mccarthy and Team agreed on Jun 7, 2023 and some cost (1-2 days) is acceptable 

18

What would be the field name of response data?

data.responses[0].data

@kieran mccarthy please confirm : we are using 'responseContent' as name of this field as 'data' is already used for cloud event standard payload.

Now it is like : data.responses[0].responseContent

@Sourabh Sourabh Suggest to use "result".  This is after all where the actual result output.  the other tags in responses[x] are status related to the request  :

                        data.responses[0].result

19

Exact format of timestamp of cloud events.

CloudEvent documentation (https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/spec.md#time) refers on ISO8601 standard

(see org.onap.cps.ncmp.api.impl.utils.EventDateTimeFormatter ) 

@Toine Siebelink Although not clearly communicated before ISO8601 is the standard to be used and it seems the current implementation in NCMP conforms to this standard 

data

Event

The payload of an event

Cloud event standard (data) 

data.responses[0, 1, 2, ...]

Array 

contains an array or batch response that includes both success and failure.

data.responses[0].operationId

String