Page Comparison

Create 'bulk' version for org.onap.cps.ncmp.rest.controller.NetworkCmProxyController#getResourceDataForCmHandle to support multiple cm-handles (~ multiple anchors in in CPS-Core)

References

1. Image RemovedCPS-1515 - Spike: Support multiple CM-Handles for NCMP Get Operation
2. CPS-NCMP ↔ DMI-Plugin Interface Details Jakarta-R10

Requirements

Functional

...

Error Handling

...

Capabilities

...

• Async response available on client topic
• No delay in DMI PLugin (tested/measured using stub DMI Plugin)

...

...

Out-of-scope

1. Support for multiple resource identifiers in one batch operation
2. Support cached data batch requests: only passthrough datastores will be supported (see decision #2)
3. NCMP does NOT keep track of request status to see if  it is completed, amalgamate responses or anything like that. It wil simply forward responses from the internal topic to the client topic.
4. Access control

Assumptions

...

#

...

Assumption

...

Notes

...

Proprietary options or not in the scope of this analysis.

...

...

same xpath (resourceIdentifierInQuery) for all cm handles or different for each cm handle

...

...

• options, resourceIdentifier is optional for bulk operations.
• operation, datastore and cmhandleIds are mandatory fields

...

...

...

Get batch operation does not support includeDescendants as it is impl. for 

pass through datastore only

ncmp-datastore:passthrough-running and

ncmp-datastore:passthrough-operational

...

Issues & Decisions

...

#

...

Issue

...

Notes 

...

Decision

...

1. Get
2. Create
3. Update (Put)
4. Patch
5. Delete

if many what is the priority?

...

...

Which datasources should be supported?

...

Do we need to support passthrough-only no-cached() data only ?
(maybe just start with that, support cached data bulk request later)

...

...

Existing : /v1/ch/{cm-handle}/data/ds/{datastore-name}

CPS Proposed : /v1/batch/data/ds/{datastore-name}

      ( include cm handles into payload / body and leave datastore into url)

-----------------------------------------------------------------------------------------------------------------------

17 Apr 2023 Meeting : kafka message schema & batch interface extension

POST http://localhost:8080/ncmp/v1/batch/data&topic=my-topic-name

{
              operations: [
                             {
                                           operation: read,
                                           datastore: "...",
                                           options: "...",
                                           resourceIdentifier: "...",
                                           cmhandleIds: [4, 6]
                             },
                             {
                                           operation: read,
                                           datastore: "...",
                                           options: "...",
                                           resourceIdentifier: "...",
                                           cmhandleIds: [1, 2, 3]
                             }
              ]
}

23 May 2023 

POST http://localhost:8080/ncmp/v1/data&topic=my-topic-name

agreed 08 Mar 2023 with kieran mccarthy : Follow the existing interfaces as much as possible for consistency and efficiency

------------------------------------------------------------------------------------------------------------

23 May 2023 kieran mccarthy 

We had an internal review with some of our rApp colleagues around some of the recent proposed NCMP batch interface and they came back with some valid comment. 

The proposal is that we should not distinguish batch from bulk or other flavours of read/write. 

The aim is to only have a single flavour of interface for read or write for clients.  Therefore the proposal is to drop “batch” from the interface URL and just act toward “data”  (reads/writes/actions)

...

CPS prefers keep interface similar as single cm handle interface (consistency and cost)

Existing : ...&topic=topicParamInQuery

...

...

support in ONAP DMI-plugin

...

...

Response always Async ie. topic is compulsory ?

...

...

...

...

...

Suggestions

1. Silently ignore
2. (initial) error response
   1. should we combine all errors in one message?

...

...

...

...

Schema of bulk response

1) Agreement required on the structure of the response.  Please see response structure below.
2) Does 'eventTime' field which holds the timestamp of the bulk response event,  required or can it be dropped?

{
  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "$id": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-headers:1.0.0",
  "$ref": "#/definitions/BatchEventHeaders",
  "definitions": {
    "BatchEventHeaders": {
      "description": "The header information of the Batch event.",
      "type": "object",
      "javaType" : "org.onap.cps.ncmp.events.async.v1.BatchEventHeaders",
      "properties": {
        "eventId": {
          "description": "The unique id for identifying the event.",
          "type": "string"
        },
        "eventCorrelationId": {
          "description": "The request id received by NCMP as an acknowledgement.",
          "type": "string"
        },
        "eventTime": {
          "description": "The time of the event. It should be in RFC format ('yyyy-MM-		     dd'T'HH:mm:ss.SSSZ').",
          "type": "string"
        },
        "eventTarget": {
          "description": "The destination topic to forward the consumed event.",
          "type": "string"
        },
        "eventSource": {
          "description": "The source of the event.",
          "type": "string"
        },
        "eventType": {
          "description": "The type of the Batch event.",
          "type": "string"
        },
        "eventSchema": {
          "description": "The schema of the Batch event payload.",
          "type": "string"
        },
        "eventSchemaVersion": {
          "description": "The schema version of the Batch event payload.",
          "type": "string"
        }
      },
      "required": [
        "eventId",
        "eventCorrelationId",
        "eventTarget",
        "eventType",
        "eventSchema",
        "eventSchemaVersion"
      ],
      "additionalProperties": false
    }
  }
}

{
  "eventId":            "4cb32729-85e3-44d1-aa6e-c923b9b059a5",
  "eventCorrelationId": "68f15800-8ed4-4bae-9e53-27a9e03e1911",
  "eventTime":          "2023-03-28T14:29:23.876+0000",
  "eventTarget":        "client-topic"
  "eventSource":        "dmi-plugin:enm-1"(dmi service name)
  "eventType":          "org.onap.cps.ncmp.event.model.BulkResponseEvent",
  "eventSchema":        "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",
  "schemaVersion":      "1.0.0",

}

{
  "$schema": "https://json-schema.org/draft/2019-09/schema",
  "$id": "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",
  "$ref": "#/definitions/BatchDataResponseEvent",
  "definitions": {
    "BatchDataResponseEvent": {
      "description": "The payload of batch event.",
      "type": "object",
      "javaType" : "org.onap.cps.ncmp.events.async.v1.BatchEvent",
      "properties": {
        "event": {
          "description": "The content of Batch event.",
          "type": "object",
          "existingJavaType": "java.lang.Object",
          "additionalProperties": false
        }
      },
      "required": [
        "event"
      ],
      "additionalProperties": false
    }
  }
}  

{     
  "event": {
     "payload": "response of batch cm handles"
  } 
}

...

...

Not keeping the 'eventTarget' which comes from the (BulkResponseEvent(In progress of the structure agreement)).

Note:  Will consider  'eventTarget', If it finalized the schema from

...

...

...

• correlationid
• target

...

...

...

Create 'bulk' version for org.onap.cps.ncmp.rest.controller.NetworkCmProxyController#getResourceDataForCmHandle to support multiple cm-handles (~ multiple anchors in in CPS-Core)

References

1. Image AddedCPS-1515 - Spike: Support multiple CM-Handles for NCMP Get Operation
2. CPS-NCMP ↔ DMI-Plugin Interface Details Jakarta-R10
3. CPS Events Structure

Requirements

Functional

Error Handling

Capabilities

Out-of-scope

1. Support for multiple resource identifiers in one batch operation
2. Support cached data batch requests: only passthrough datastores will be supported (see decision #2)
3. NCMP does NOT keep track of request status to see if  it is completed, amalgamate responses or anything like that. It wil simply forward responses from the internal topic to the client topic.
4. Access control

Assumptions

ncmp-datastore:passthrough-running and

ncmp-datastore:passthrough-operational

Issues & Decisions

Event Format Definitions

Event Headers

Event Body (data)

• The current proposal is to use the CNFC Event structure and library as detail in CPS Events Structure#CNCFCloudEventalignment
• To prevent problems with large data request multiple response event will be sent for a successful read for each cm handle (id)
• Error messages can contain information about single cm handle ids

data

data.responses[0, 1, 2, ...]

data.responses[0].operationId

data.responses[0].ids

data.responses[0].statusCode

data.responses[0].statusMessage

Successfully applied changes.

100

cm handle id(s) is(are) not found.

cm handle id(s) is(are) in non ready state.

Implementation

Bulk Request Message Flow

Message Flow details

curl --location 'http: //localhost:8080/ncmp/v1/data?topic=my-topic-name' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic Y3BzdXNlcjpjcHNyMGNrcyE=' \
--data '{
    "operations": [
        {
            "operation": "read",
            "operationId": "12",
            "datastore": "ncmp-datastore:passthrough-operational",
            "options": "some 

option",
            "resourceIdentifier": "parent/child",
            "targetIds": [
                "836bb62201f34a7aa056a47bd95a81ed",
                "202acb75b4a54e43bb1ff8c0c17a8e08"
            ]
        },
        {
            "operation": "read",
            "operationId": "14",
            "datastore": "ncmp-datastore:passthrough-running",
            "targetIds": [
                "ec2e9495679a43c58659c07d87025e72",
                "0df4d39af6514d99b816758148389cfd"
            ]
        }
    ]
}'

{
    "requestId": "4753fc1f-7de2-449a-b306-a6204b5370b3"
}

DMI Service 1 (POST): http://172.26.202.25:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> 

[
    {
        "operationType": "read",
        "operationId": "14",
        "datastore": "ncmp-datastore:passthrough-running",
        "cmHandles": [
            {
                "id": "ec2e9495679a43c58659c07d87025e72",
                "cmHandleProperties": {
                    "neType": "RadioNode"
                }
            },
            {
                "id": "0df4d39af6514d99b816758148389cfd",
                "cmHandleProperties": {
                    "neType": "RadioNode"
                }
            }
        ]
    }
]

DMI Service 2 (POST) : http://172.26.202.26:8783/dmi/v1/data?topic=my-topic-name&requestId=4753fc1f-7de2-449a-b306-a6204b5370b3 -> 

   [
    {
        "operationType": "read",
        "operationId": "12",
        "datastore": "ncmp-datastore:passthrough-operational",
        "options": "

some option",
        "resourceIdentifier": "parent/child",
        "cmHandles": [

 {

Event Format Definitions

Event Headers

...

Event Body (data)

• The current proposal is to use the CNFC Event structure and library as detail in CPS Events Structure#CNCFCloudEventalignment
• To prevent problems with large data request a single response event will be sent for a successful read for each cm handle (id)
• Error message can contain information about multiple cm handle ids

...

Field 

...

Type

...

Description

...

Mandatory?

...

Notes

...

event

...

Mandatory

...

event.responses[0, 1, 2, ...]

...

Mandatory

...

event.responses[0].operationId

...

Mandatory

...

event.responses[0].ids

...

Mandatory

...

Example : ["0df4d39af6514d99b816758148389cfd"]
Note: Ids array should contain only a single element in the array in case of success messages and In case of error it can have any number of elements.

...

event.responses[0].status-code

...

Mandatory

...

Common NCMP defined error codes:

• status-code 0-99 is reserved for any success response
• status-code from 100 to 199 is reserved for any failed response. 

...

event.responses[0].status-message

...

Mandatory

Examples for code & message :

...

Optional

...

1. In case of success :
   • Optional,  for write operations then no need to return configurations                    application/yang-patch+json | application/yang-data+json
2. In case of failure :
   • Optional, any supplementary error data matching the error status-code

Implementation

Bulk Request Message Flow

Message Flow details

body:
["cm-1",...,"cm-n"]

'http://localhost:8080/ncmp/v1/batch/data/ds/ncmp-datastore:passthrough-running?resourceIdentifier=parent/child%26options=(a=1,b=2)&topic=my-topic-name&options=(fields=schemas/schema)' \
--header 'Authorization: Basic Y3BzdXNlcjpjcHNyMGNrcyE=' \
--header 'Content-Type: application/json' \
--data '[ "40137a9771f84459affa795fa1d633ab", "f5a92ec7a7db4d6fbb0e0ce2803a86cc" ]'

{"requestId":"123"}

body: {"Cmhandles":["cm-1",...,"cm-n"],"requestId":123}

      "id": "836bb62201f34a7aa056a47bd95a81ed",
                "cmHandleProperties": {
                    "neType": "RadioNode"
                }
            },
            {
                "id": "202acb75b4a54e43bb1ff8c0c17a8e08",
                "cmHandleProperties": {
                    "neType": "RadioNode"
                }
            }
        ]
    }
]

{
  "eventId$schema":            "4cb32729-85e3-44d1-aa6e-c923b9b059a5",
  "eventCorrelationId": "68f15800-8ed4-4bae-9e53-27a9e03e1911"https://json-schema.org/draft/2019-09/schema",
  "eventTime$id":          "2023-03-28T14:29:23.876+0000"urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",
  "eventTarget$ref": "#/definitions/BatchDataResponseEvent",
  "definitions": {
  "client-topic"   "eventSourceBatchDataResponseEvent": {
      "description"dmi-plugin:enm-1"(dmi service name): "The payload of batch event.",
      "eventTypetype": "object",
      "javaType" : "org.onap.cps.ncmp.events.eventasync.modelv1.BulkResponseEventBatchEvent",
      "eventSchemaproperties": {
       "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",
  "schemaVersion": "event": {
          "1.0.0",

}

description": "The content of Batch event.",
   

      "

type": "object"

,

   "

existingJavaType": "java.lang.Object",
          "

additionalProperties": false
        }
      

},
      "

required": [
        "

event"
    

  ],
      

"additionalProperties": false
    }
  }
}  

{     
  "urn:cps:org.onap.cps.ncmp.events.async:batch-event-schema:1.0.0",event": {
     "schemaVersionpayload": "response of batch cm handles"
  "1.0.0",} 
}

{     
  "event": {
     "payload": "response of batch cm handles"
  } 
}

{
  "timestamp":"2023-03-01T23:00:00.345-0400", 
  "requestId":123,
  "error": "DMI Service Unavailable, {service-name}<error-message>",
  "Cmhandles":["cm-1",...,"cm-n"]
  }

{
	"timestamp":"2023-03-01T23:00:00.345-0400",
    "requestId":123,
	"error":"Cm-Handle not found",
	"Cmhandles":["cm-1",...,"cm-n"]
}

{
	"timestamp":"2023-03-01T23:00:00.345-0400",
    "requestId":123,
	"error":"Cm-Handle not in READY state.",
	"Cmhandles":["cm-1",...,"cm-n"]
}

Proposed JIRAs :

Proposed JIRAs :

Planning :

• Allow for 2 more user stories each may take 1 weeks
• Based on 2 resource working  working parallelly may take àpproxapprox.. 8 weeks from 29 March 2023  2023  
• (updated after additional items where added) The estimated date for completion is 14 28 Jun 2023  

• The release for cps-and-ncmp 3.3.3 version is completed on Friday, 30 Jun 2023 

  Release tag : https://gerrit.onap.org/r/gitweb?p=cps.git;a=tag;h=refs%2Ftags%2F3.3.3

• Demo recording can be referred here : https://lf-onap.atlassian.net/wiki/display/DW/CPS+User+Story+Demos