CPS-1515: Spike: Support Multiple CM-Handles for NCMP Get Operation (batch operations)

Create 'bulk' version for org.onap.cps.ncmp.rest.controller.NetworkCmProxyController#getResourceDataForCmHandle to support multiple cm-handles (~ multiple anchors in in CPS-Core)

References

CPS-1515 - Spike: Support multiple CM-Handles for NCMP Get Operation
CPS-NCMP ↔ DMI-Plugin Interface Details Jakarta-R10
CPS Events Structure

Requirements

Functional

#	Interface	Requirement	Additional Information
1	REST CPS(-NCMP)-E-05	Support batch read operation (new) using an asynchronous response on a client specified topic	payload includes list of cm handle ids
2	REST DMI-I-01	Support batch read operation (new) using an asynchronous response to an internal topic	payload includes list of (associated for this plugin) cm handles ids with their private (additional) properties

Error Handling

#	Error Scenario	Expected behavior
1	DMI Not respond to initial synchronous request with normal HTTP timeout	Special Error message (Kafka event) including affected, and reason for all the handles in the message cm handles send to client topic detailing cm-handles (per dmi plugin)
2	Topic not supplied on CPS-E-05	Return HTTP 501 (not implemented)
3	Client specided Topic is not configured	Log error message only
4	Non-existing cm-handle id	Similar message but different reason as specified #1 above

Capabilities

#	Parameter	Expectation	Notes
1	Response Time 1 Batch request	<2 seconds (average)	Async response available on client topic No delay in DMI PLugin (tested/measured using stub DMI Plugin)
2	Batch-size	200 cm handles	No hardcoded limit
3	Response payload size	~2 KB per cm handles	Performance test for capability should be tested with this average response size
4	Maximum registered #cm handles	20,000	This will effect the internal query time
5	Supported # DMI PLugins	10	This might effect processing times
6	Test Environment	Click here to expand... CPS and NCMP `requests: cpu: 2000m memory: 2Gi limits: memory: 3Gi cpu: 3000m` 2. Postgres `requests:` `cpu: 4000m` `memory: 1Gi` `limits:` `memory: 3Gi` `cpu: 6000m`
7	Concurrent request	12	12 clients requests toward 1 NCMP simultaneously
8	Request Frequency	100 request/min	Should not affect performance, does not need to be tested

Out-of-scope

Support for multiple resource identifiers in one batch operation
Support cached data batch requests: only passthrough datastores will be supported (see decision #2)
NCMP does NOT keep track of request status to see if it is completed, amalgamate responses or anything like that. It wil simply forward responses from the internal topic to the client topic.
Access control

Assumptions

#	Assumption	Notes
1	Proprietary options or not in the scope of this analysis.	agreed 08 Mar 2023 with kieran mccarthy ; these are optional parameters (name-value pairs) not interpreted by NCMP but can be interpreted by proprietary plugins In the future "scope" might become standardized instead of proprietary but that wil be achieved through a separate requirement
2	same xpath (resourceIdentifierInQuery) for all cm handles or different for each cm handle	agreed 08 Mar 2023 with kieran mccarthy ; if different resources are required on the same cm-handle the client wil send another (batch) request
3	options, resourceIdentifier is optional for bulk operations. operation, datastore and cmhandleIds are mandatory fields	agreed 24 Apr 2023 with CPS team.
4	We are not merging any duplicate cm handles while sending request to dmi-plugin	agreed 24 Apr 2023
5	Get batch operation does not support includeDescendants as it is impl. for pass through datastore only ncmp-datastore:passthrough-running and ncmp-datastore:passthrough-operational	agreed on 15 May 2023

Issues & Decisions

#

Issue

Notes

Decision

1

Which operation(s) need support for multiple cm handles?

Get
Create
Update (Put)
Patch
Delete

if many what is the priority?

agreed 08 Mar 2023 with kieran mccarthy

Only Get (read)

(in future other operations might be support batch option too)

2

Which datasources should be supported?

Do we need to support passthrough-only no-cached() data only ?
(maybe just start with that, support cached data bulk request later)

agreed 08 Mar 2023 with kieran mccarthy :

all passthrough datastores will be supported

Not implemented (yet) response, for non passthrough datastores

3

URL pattern for NCMP bulk endpoints

08 Mar 2023

~~Existing : /v1/ch/{cm-handle}/data/ds/{datastore-name}~~

~~CPS Proposed : /v1/batch/data/ds/{datastore-name}~~

~~( include cm handles into payload / body and leave datastore into url)~~

-----------------------------------------------------------------------------------------------------------------------

17 Apr 2023 Meeting : kafka message schema & batch interface extension

~~POST http://localhost:8080/ncmp/v1/batch/data&topic=my-topic-name~~

{
operations: [
{
operation: read,
datastore: "...",
options: "...",
resourceIdentifier: "...",
cmhandleIds: [4, 6]
},
{
operation: read,
datastore: "...",
options: "...",
resourceIdentifier: "...",
cmhandleIds: [1, 2, 3]
}
]
}

23 May 2023

POST http://localhost:8080/ncmp/v1/data&topic=my-topic-name

~~agreed~~ 08 Mar 2023 ~~with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency~~

------------------------------------------------------------------------------------------------------------

23 May 2023 kieran mccarthy

We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment.

The proposal is that we should not distinguish batch from bulk or other flavours of read/write.

The aim is to only have a single flavour of interface for read or write for clients. Therefore the proposal is to drop “batch” from the interface URL and just act toward “data” (reads/writes/actions)

Before	After
~~POST http://localhost:8080/ncmp/v1/batch/data&topic=my-topic-name~~	POST http://localhost:8080/ncmp/v1/data&topic=my-topic-name

4

keep datastore, topic and optional parameters in the URL itself instead into body.

CPS prefers keep interface similar as single cm handle interface (consistency and cost)

Existing : ...&topic=topicParamInQuery

agreed 08 Mar 2023 with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency

5

support in ONAP DMI-plugin

agreed 08 Mar 2023 with Toine Siebelink : ONAP plugin can respond with not implemented yet code,

6

Response always Async ie. topic is compulsory ?

Assume topic is compulsory (defined in OPenApi) → Response therefore wil be 400 if not supplied

Agreed 08 Mar 2023 with kieran mccarthy :

Topic is optional but system will respond with 'Not implemented (yet when not specified or blank

7

Should NCMP Amalgamate Async responses from DMI-Plugin before forwarding ?

step 6 in flow diagram

Agreed 08 Mar 2023 with kieran mccarthy : NCMP wil only forward to client topic no handling tracking or any responses or status of request

8

Handle non responding dmi-plugin

Agreed 08 Mar 2023 with kieran mccarthy :
No response for 4b then send an error response to the topic given by client

9

Should (can) NCMP check if 'MyTopic' specified by client exist

Consider Access Control too. For now NCMP can log error. Client is responsible for topic setup

Agreed 08 Mar 2023 with kieran mccarthy :
NCMP can log error when forward to 'client topic' Security not in scope (yet)

10

How to handle non-existing CM-Handles (id)

Suggestions

Silently ignore
(initial) error response
1. should we combine all errors in one message?

Agreed 08 Mar 2023 with kieran mccarthy :

Additional error (messages) response with all cm-handles that cannot be resolved, also a separate error message wil be sent for each failed DMI

11

Overlap/clash with Deutsche Telekom user story: CPS-1377 - Getting issue details... STATUS ?!

08 Mar 2023 discussed in weekly ONAP meeting the DT user story is affect CPS-Core interface (not NCMP) and the requirement is to execute a query over ALL cm-handles (cached only?!) instead of a given list of cm-handles (~anchors)

12

Schema of bulk response

CPS-1556 - Getting issue details... STATUS

1) Agreement required on the structure of the response. Please see response structure below.
2) Does 'eventTime' field which holds the timestamp of the bulk response event, required or can it be dropped?

Batch Event Payload Schema

{
  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "$id": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",
  "$ref": "#/definitions/BatchDataResponseEvent",
  "definitions": {
    "BatchDataResponseEvent": {
      "description": "The payload of batch event.",
      "type": "object",
      "javaType" : "org.onap.cps.ncmp.events.async.v1.BatchEvent",
      "properties": {
        "event": {
          "description": "The content of Batch event.",
          "type": "object",
          "existingJavaType": "java.lang.Object",
          "additionalProperties": false
        }
      },
      "required": [
        "event"
      ],
      "additionalProperties": false
    }
  }
}

Sample Batch Event Payload

{     
  "event": {
     "payload": "response of batch cm handles"
  } 
}

Need to follow schema structure there in #15 under Issues & Decisions section and it is agreed on 11 May 2023

13

~~Schema for Bulk Response event forwarding to client specified topic~~

CPS-1557 - NCMP : forward bulk response messages to client topic

~~Not keeping the 'eventTarget' which comes from the (BulkResponseEvent(In progress of the structure agreement)).~~

~~Note: Will consider 'eventTarget', If it finalized the schema from~~ CPS-1556 - Getting issue details... STATUS

Agreed on 11 May 2023 to keep one schema for both events (DMI → NCMP) and (NCMP → ClientApps)

14

How NCMP would forward response? (Response data : ref. message flow #6)

CPS-1556 - Getting issue details... STATUS

1. Do we need to send only one response containing all the requested cm handles ?
2. Send single response for each cm handle?
3. Single response message per DMI plugin?

kieran mccarthy - Please share your inputs on the above listed questions.

15

Message format for the batch interface

some non-standard headers wil need to be implemented as 'extensions' and names to be confirmed

correlationid
target

23 May 2023 kieran mccarthy

We have had some internal discussions including with some O-RAN standards representatives and one of the outcomes is that it would be good if we aligned with the some community standards for event header definitions. IT is proposed (initially from AT&T) that we should follow Cloud Native Computing Foundation (CNCF) specification as defined in their cloudevents incubator project.
More details see CPS Events Structure#CNCFCloudEventalignment

16

NCMP wil send only one request to each DMI

NCMP batch endpoint

curl --location 'http: //localhost:8080/ncmp/v1/data?topic=my-topic-name' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic Y3BzdXNlcjpjcHNyMGNrcyE=' \
--data '{
    "operations": [
        {
            "operation": "read",
            "operationId": "12",
            "datastore": "ncmp-datastore:passthrough-operational",
            "options": "(fields=schemas/schema)",
            "resourceIdentifier": "parent/child",
            "targetIds": [
                "836bb62201f34a7aa056a47bd95a81ed",
                "202acb75b4a54e43bb1ff8c0c17a8e08"
            ]
        },
        {
            "operation": "read",
            "operationId": "14",
            "datastore": "ncmp-datastore:passthrough-running",
            "targetIds": [
                "ec2e9495679a43c58659c07d87025e72",
                "0df4d39af6514d99b816758148389cfd"
            ]
        }
    ]
}'

NCMP batch endpoint response (202->Accepted)

{
    "requestId": "4753fc1f-7de2-449a-b306-a6204b5370b3"
}

DMI service batch endpoint

DMI Service 1 (POST): http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> 

[
    {
        "operationType": "read",
        "operationId": "14",
        "datastore": "ncmp-datastore:passthrough-running",
        "cmHandles": [
            {
                "id": "ec2e9495679a43c58659c07d87025e72",
                "cmHandleProperties": {
                    "neType": "RadioNode"
                }
            },
            {
                "id": "0df4d39af6514d99b816758148389cfd",
                "cmHandleProperties": {
                    "neType": "RadioNode"
                }
            }
        ]
    }
]

DMI Service 2 (POST) : http://172.26.202.26:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> 

   [
    {
        "operationType": "read",
        "operationId": "12",
        "datastore": "ncmp-datastore:passthrough-operational",
        "options": "(fields=schemas/schema)",
        "resourceIdentifier": "parent/child",
        "cmHandles": [
            {
                "id": "836bb62201f34a7aa056a47bd95a81ed",
                "cmHandleProperties": {
                    "neType": "RadioNode"
                }
            },
            {
                "id": "202acb75b4a54e43bb1ff8c0c17a8e08",
                "cmHandleProperties": {
                    "neType": "RadioNode"
                }
            }
        ]
    }
]

Agreed 11 May 2023 with Csaba Kocsis and kieran mccarthy

requestId would be send as a path param from NCMP to DMI-plugin.

Example

NCMP batch endpoint : http://localhost:8080/ncmp/v1/data?&topic=my-topic-name

DMI-Plugin batch endpoint : http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=e6fa4d26-4dc1-4877-aa3c-45e99f840708

Mandatory Fileds :
- - "operation": "read"
  - "operationId": "12"
    "datastore": "ncmp-datastore:passthrough-operational"
  - "targetIds": [ "0df4d39af6514d99b816758148389cfd", "ec2e9495679a43c58659c07d87025e72" ]

Optional Fields :
- - "options": "(fields=schemas/schema)"
  - "resourceIdentifier": "parent/child"

17

Feature Name (events, URL)

event type proposal : from ~~BatchDataXXXEvent~~ to

DataOperationXXXEvent

For example : DataOperationResponseEvent

'eventType' value as below 'org.onap.cps.ncmp.events.~~Batch~~DataXXXEvent'

proposed Operation like :

'org.onap.cps.ncmp.events.DataOperationXXXEvent'

'batch' was the keyword use in the URL for this feature (now gone) but much code (classnames, variable names) etc uses this name still...

Event Format Definitions

Event Headers

AVC Subscription Event

Subscription Event Object

Field	Type	Description	Header	Notes	CloudEvents Mapping
~~version~~	~~string~~	~~The event type version~~	X		~~included in dataschema~~
eventType	string	The event type	X		standard (`type`)
source			X	NEW see decision #18, value <clientId> (same as in payload) TBC	standard (`source`)
event	Event	The payload for an event	X	Defined by event object below	standard (`data`)

Event Object (cps:org.onap.cps.ncmp.events:avc-subscription-event:v1)

Field	Type	Description	Payload	Notes
subscription	Subscription	The subscription details	X	Defined by subscription object below
dataType	DataType	The datatype content	X	Defined by dataType object below
predicates	Predicates	Additional values to be added into the subscription	X	Defined by predicates object below

Type: Subscription

Field	Type	Description	Payload	Notes
clientID	string	The client ID	X
name	string	The name of the subscription	X
isTagged	boolean	optional parameter, default is false	X	default: false

Type: DataType

Field	Type	Description	Payload
dataspace	string	The dataspace name	X
dataCategory	string	The category type of the data	X
dataProvider	string	The provider name of the data	X
schemaName	string	The name of the schema	X
schemaVersion	string	The version of the schema	X

Type: Predicates

Field	Type	Description	Payload
targets	array	CM Handles to be targeted by the subscription	X
datastore	string	datastore which is to be used by the subscription	X
Xpath-filter	string	filter to be applied to the CM Handles through this event	X

Event Body (data)

The current proposal is to use the CNFC Event structure and library as detail in CPS Events Structure#CNCFCloudEventalignment
To prevent problems with large data request a single response event will be sent for a successful read for each cm handle (id)
Error message can contain information about multiple cm handle ids

Field

Type

Description

Mandatory?

Notes

event

Event

The payload of an event

Mandatory

event.responses[0, 1, 2, ...]

Array

contains an array or batch response that includes both success and failure.

Mandatory

event.responses[0].operationId

String

specified to distinguish multiple operations using same cmhandleId

Mandatory

event.responses[0].ids

String

cmhandle-ids

Mandatory

Example : ["0df4d39af6514d99b816758148389cfd"]
Note: Ids array should contain only a single element in the array in case of success messages and In case of error it can have any number of elements.

event.responses[0].statusCode

String

Mandatory

Common NCMP defined error codes:

status-code 0-99 is reserved for any success response
status-code from 100 to 199 is reserved for any failed response.

event.responses[0].statusMessage

String

Mandatory

Examples for code & message :

statusCode	statusMessage
1	"Successfully applied changes"
101	"cmHandle(s) do not exist"

event.responses[0].data

Object

Optional

In case of success :
- Optional, for write operations then no need to return configurations
  application/yang-patch+json | application/yang-data+json
In case of failure :
- Optional, any supplementary error data matching the error status-code

Implementation

Bulk Request Message Flow

Message Flow details

Flow Step	Short description	Message Details	Notes
1	Bulk Get Request	/v1/batch/data/ds/{datastore-name} Expand source body: ["cm-1",...,"cm-n"] Example CURL cmd : curl --location --request POST Expand source 'http://localhost:8080/ncmp/v1/batch/data/ds/ncmp-datastore:passthrough-running?resourceIdentifier=parent/child%26options=(a=1,b=2)&topic=my-topic-name&options=(fields=schemas/schema)' \ --header 'Authorization: Basic Y3BzdXNlcjpjcHNyMGNrcyE=' \ --header 'Content-Type: application/json' \ --data '[ "40137a9771f84459affa795fa1d633ab", "f5a92ec7a7db4d6fbb0e0ce2803a86cc" ]'	Define new get operation "getResourceDataForCmHandles" into ncmp.yml
2	Ack clent Request	Response 202 (Accepted) Expand source {"requestId":"123"}
3	DMI Bulk Request	/v1/ch/batch/data/ds/{datastore-name}&topic=topicParamInQuery Expand source body: {"Cmhandles":["cm-1",...,"cm-n"],"requestId":123}	The DMI PLugin should be told (included in request) the client topic so that NCMP does not have to 'remember' to relation between request id and client topic!
4	Ack NCMP Request	Response 202 Expand source	202 is non-committal, meaning that there is no way for the HTTP to later send an asynchronous response indicating the outcome of processing the request. It is intended for cases where another process or server handles the request, or for batch processing.
5	Kafka Messages from DMI to NCMP	Batch Event Headers Expand source { "eventId": "4cb32729-85e3-44d1-aa6e-c923b9b059a5", "eventCorrelationId": "68f15800-8ed4-4bae-9e53-27a9e03e1911", "eventTime": "2023-03-28T14:29:23.876+0000", "eventTarget": "client-topic" "eventSource": "dmi-plugin:enm-1"(dmi service name) "eventType": "org.onap.cps.ncmp.event.model.BulkResponseEvent", "eventSchema": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0", "schemaVersion": "1.0.0", } Batch Event Payload Expand source { "event": { "payload": "response of batch cm handles" } }
6	Kafka Message(s) from NCMP to Client Table	Batch Event Headers Expand source { "eventId": "4cb32729-85e3-44d1-aa6e-c923b9b059a5", "eventCorrelationId": "68f15800-8ed4-4bae-9e53-27a9e03e1911", "eventTime": "2023-03-28T14:29:23.876+0000", "eventSource": "dmi-plugin:enm-1"(dmi service name) "eventType": "org.onap.cps.ncmp.event.model.BulkResponseEvent", "eventSchema": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0", "schemaVersion": "1.0.0", } Batch Event Payload Expand source { "event": { "payload": "response of batch cm handles" } }
7	Alternative for 4/5 → Non responding DMI. NCMP will have to create error message detailing cm-handles		See decision # 8 and 9
8	Response message structure ? (Flow no. 5) Non responding DMI-plugin	Non responding DMI-plugin Expand source { "timestamp":"2023-03-01T23:00:00.345-0400", "requestId":123, "error": "DMI Service Unavailable, {service-name}", "Cmhandles":["cm-1",...,"cm-n"] }
9	Response message structure ? (Flow no. 5) Non existing cm handles	Non existing cm handles Expand source { "timestamp":"2023-03-01T23:00:00.345-0400", "requestId":123, "error":"Cm-Handle not found", "Cmhandles":["cm-1",...,"cm-n"] }
10	Non Ready cm handles	Non READY existing cm handles Expand source { "timestamp":"2023-03-01T23:00:00.345-0400", "requestId":123, "error":"Cm-Handle not in READY state.", "Cmhandles":["cm-1",...,"cm-n"] }
11	URL pattern for DMI-Plugin bulk endpoints	Existing DMI endpoints are : /v1/ch/{cmHandle}/data/ds/{datastore-name} datastore-name: ncmp-datastore:passthrough-operational ncmp-datastore:passthrough-running ...&topic=topicParamInQuery CPS Proposed : /v1/ch/batch/data/ds/{datastore-name} ...&topic=topicParamInQuery cm handle ids and requestid into body

Proposed JIRAs :

Priority	Component	Description	JIRAs	Estimate	Status
1	DMI, NCMP	Batch Response Event (DMI → NCMP) to Comply with Cloud Events	CPS-1723 - Getting issue details... STATUS	5 Days (M)	New Requirement.
2	NCMP	Batch Response Event (NCMP → Client App)	CPS-1724 - Getting issue details... STATUS	TBD	New Requirement.
3	DMI-Plugin	Accept datastore name as param into URL	CPS-1550 - Getting issue details... STATUS	5 Days	Done
4	NCMP	Expose REST endpoint to accept collection of cm handles for GET operation (Passthrough only)	CPS-1553 - Getting issue details... STATUS	15 Days	Code is merged and Delivered, Pending task id RTD document only
5	DMI-Plugin	Expose endpoint for ONAP not impl. and Stub impl. for testing/demo	CPS-1555 - Getting issue details... STATUS	5 Days	Done
6	NCMP	NCMP: Update existing REST endpoint that accepts bulk request for GET operation	CPS-1635 - Getting issue details... STATUS	5 Days	new because Requirements changed +5 days
7	DMI-Plugin	DMI-Plugin : Update endpoint to accept bulk request	CPS-1636 - Getting issue details... STATUS	3 Days	Submitted (In code review state)
8	Stubbed DMI-Plugin	Include code to send response messages to internal kafka topic with delay	CPS-1556 - Getting issue details... STATUS	10 Days	In progress
9	NCMP	Forward response messages to client given kafka topic	CPS-1557 - Getting issue details... STATUS	5 Days	Re-opened
10	NCMP	Handle non-existing cm handles	CPS-1583 - Getting issue details... STATUS	5 Days
11	NCMP	Error handling for non-ready cm handle state	CPS-1614 - Getting issue details... STATUS	5 Days	In progress
12	NCMP	Handle non responding DMI-Plugin	CPS-1558 - Getting issue details... STATUS	5 Days
13	CSIT test for demo		CPS-1560 - Getting issue details... STATUS	5 Days

Planning :

Allow for 2 more user stories each may take 1 weeks
Based on 2 resource working parallelly may take approx.. ~~8 weeks from 29 March 2023~~
(updated after additional items where added) The estimated date for completion is 14 Jun 2023