Impl. Proposal CM Subscriptions : Creating/Merging( Positive Scenarios )

References

CPS-1870 - Getting issue details... STATUS

CPS-1812 - Getting issue details... STATUS

Assumptions

#	Issue	Notes	Decisions
1

Issues & Decisions

#	Issue	Notes	Decisions
1	Whether we need to persist the raw request for auditing purpose?
2	Keep track of the correlation id	Request to and from DMI Plugin has to be tracked hence we need to have a correlation id for this. ( to match the request and response )

Overview

This page only contains the details used for starting the development on the Merging of Subscriptions use case. Just the straight forward scenarios will be covered and the overall solution will be build on top of it.

Schema Definitions

DME to NCMP Subscription Request

	Name	Parent	Type	Example	Mandatory ?
1	id	header	String		Y
2	version	header	String	1.0	Y
3	source	header	String	DME	Y
4	type	header	String	subscriptionCreateRequest	Y
5	dataschema	header	URI	org.onap.ncmp.cm.subscription:1.0.0	Y
6	data		Object		Y
7	subscriptionId	data	String		Y
8	predicates	data	Array		N
9	targetFilter	predicates	Array of String		Y
10	scopeFilter	predicates	Object		N
11	datastore	scopeFilter	String		Y
12	xpathFilter	scopeFilter	Array of String		Y

Example

DME to NCMP Subscription Request

id : random-UUID
version : "1.0"
source : "DME"
type : "subscriptionCreateRequest",
dataschema : org.onap.ncmp.cm.subscription:1.0.0  {
  "data": {
    "subscriptionId": "unique subscription id", // mandatory
    "predicates": [
	{
        “targetFilter” :  list of cmhandles, // mandatory
		"scopeFilter"  :  { 
	       "datastore": “ncmp-datastore:passthrough-operational or ncmp-datastore:passthrough-running", // optional. default is passthrough-operational
           "xpath-filter": list of valid xpaths  // mandatory
      }
    }
	]
}

NCMP to DMI-Plugin Subscription Request

	Name	Parent	Type	Example	Mandatory ?	Notes
1	id	header	String		Y
2	version	header	String	1.0	Y
3	source	header	String	DME	Y
4	type	header	String	subscriptionCreateRequest	Y
5	dataschema	header	URI	org.onap.ncmp.dmi.cm.subscription:1.0.0	Y
6	correlationid	header	String	subscriptionId#dmiPluginName	Y	Concatenation of subscriptionId and dmi plugin name
7	data		Object		Y
8	cmHandles	data	Array		Y
9	cmhandleId	cmHandles	String		Y
10	privateProperties	cmHandles	Map		N
11	predicates	data	Array		N
12	targetFilter	predicates	Array of String		Y	list of cm handle ids
13	scopeFilter	predicates	Object		N
14	datastore	scopeFilter	String		Y	ncmp-datastore:passthrough-operational OR ncmp-datastore:passthrough-running
15	xpathFilter	scopeFilter	Array of String		Y

Example

NCMP to DMI-Plugin Subscription Request

id : random UUID
version : 1.0
source : "NCMP"
type : "subscriptionCreateRequest"
dataschema : org.onap.ncmp.dmi.cm.subscription:1.0.0
correlationId ( concatenation of subscriptionId and dmi-plugin-name with a separator)

{
  "data": {
   
	"cmhandles" : [ // mandatory
	{
		"cmHandleId": "cmhandle1",
		"privateProperties": {...} 
	},
	{
		"cmHandleId": "cmhandle2",
		"privateProperties": {...}
	},
	{
		"cmHandleId": "cmhandle3",
		"privateProperties": {...}
	}
	]
    "predicates": [
	{
        “targetFilter” :  [“cmHandle1”, “cmHandle2”, "cmhandle3"], // mandatory
		"scopeFilter" : { 
	       "datastore": “ncmp-datastore:passthrough-operational or ncmp-datastore:passthrough-running", // optional. default is passthrough-operational
           "xpathFilter": list of valid xpaths  // mandatory
      }
    },  
	{
        “targetFilter” :  [“cmHandle1”, “cmHandle2”, "cmhandle3"],
		"scopeFilter" : { 
	       "datastore": “ncmp-datastore:passthrough-operational or ncmp-datastore:passthrough-running", // optional. default is passthrough-operational
           "xpathFilter": list of valid xpaths  *
      }
    }
  ]
}

DMI-Plugin to NCMP Subscription Response

	Name	Parent	Type	Example	Mandatory ?	Notes
1	id	header	String		Y
2	version	header	String	1.0	Y
3	source	header	String	DME	Y
4	type	header	String	subscriptionCreateResponse	Y
5	dataschema	header	URI	org.onap.ncmp.dmi.cm.subscription:1.0.0	Y
6	correlationid	header	String	subscriptionId#dmiPluginName	Y	Concatenation of subscriptionId and dmi plugin name
7	data		Object		Y
8	statusCode	data	String		Y	1 - Accept the subscription request 104 - Reject the subscription request
9	statusMessage	data	String		Y	ACCEPTED REJECTED

Example

DMI-Plugin to NCMP Subscription Response

id : random UUID
version : "1.0"
source : <dmi-plugin-name>
type : "subcriptionCreateResponse"
dataschema : org.onap.ncmp.dmi.cm.subscription:1.0.0
correlationid : ( concatenation of subscriptionId and dmi-plugin-name with a separator) 
{ 
"data" : {
   "statusCode": "1/104",  // mandatory
   "statusMessage" : "ACCEPTED/REJECTED" // mandatory
 }
}

NCMP to DME Subscription Response

	Name	Parent	Type	Example	Mandatory ?
1	id	header	String		Y
2	version	header	String	1.0	Y
3	source	header	String	DME	Y
4	type	header	String	subscriptionDeleteResponse	Y
5	dataschema	header	URI	org.onap.ncmp.cm.subscription:1.0.0	Y
6	correlationid	header	String	subscriptionId	Y
7	data		Object		Y
8	subscriptionId	data	String		Y
9	acceptedTargets	data	Array of String		N
10	rejectedTargets	data	Array of String		N
11	pendingTargets	data	Array of String		N

Example

NCMP to DME Subscription Response

id : random UUID
version : "1.0"
source : "NCMP"
type : "subcriptionCreateResponse",
dataschema : org.onap.ncmp.cm.subscription:1.0.0,
correlationId : <subscriptionId> {
  "data": {
		"subscriptionId": "sample-subscription-id", // mandatory
	    "accepted-targets" : ["ch-1", ...], // optional
		"rejected-targets" : ["ch-1", ...], // optional
		"pending-targets" : ["ch-1", ...],  // optional
  }
}

Storage Solutions

Proposed Data Storage

Use a combination of

Database (yang modelled) for permanent subscription details, subscribers, predicates
In-Memory (hazelcast) for transient subscription details: state (acceptep, rejected, pending)

Example Flow 1: First Subscription

We have the below configuration managed by CPS-NCMP

#
cmhandle
dmi-plugin
1 ch-1 dmi-1
2 ch-2 dmi-1
3 ch-3 dmi-2
4 ch-4 dmi-2
5 ch-5 non-responding-dmi

We get a Subscription Create Request with unique subscription-id and a list of predicates(list of targets , datastore and list of filters)

Subscription Create Message

{
"subscriptionId" : "A-10",
"predicates" : [
  {
    "targets": [ch-1,ch-2],
    "datastore" : "ncmp-datastore:passthrough-operational",
    "datastore-xpath-filter" : ["p1/c1","p2/c2"]
    },
  { "targets": [ch-3],
    "datastore" : "ncmp-datastore:passthrough-operational",
    "datastore-xpath-filter" : ["p3/c3"]
    }
] 
}

We maintain a distributed datastructure in Hazelcast to keep track of the request and response to the client.

SubscriptionId
dmi-plugin
affectedCmHandles
status
A-10 dmi-1 ["ch-1","ch-2"] PENDING
A-10 dmi-2 ["ch-3"] PENDING
Read the requests from the distributed data structure and form the request for the DMI-Plugin and publish it to the internal kafka channel.
If the DMI-Plugin responds back within the configured time , so the request will either be ACCEPTED or REJECTED.
1. if whole subscription request is ACCEPTED , update the distributed data structure and then store the subscription to the database.
2. if whole subscription request is REJECTED , update the distributed data structure and send back the response to the client and no need to store anything to database.
3. if the DMI Plugin fails to respond then the status would remain PENDING only in the distributed datastructure as there is no change.
  
  SubscriptionId
  dmi-plugin
  affectedCmHandles
  status
  A-10 dmi-1 ["ch-1","ch-2"] ACCEPTED
  A-10 dmi-2 ["ch-3"] REJECTED
  
  If there are no more PENDING requests , then we can right away send the response without updating the cache.

We store the "ActiveSubscriptions" using a new model which will look like below.

SubscriptionId	cmHandle	filter	datastore
A-10	ch-1	p1/c1	ncmp-datastore:passthrough-operational
A-10	ch-1	p2/c2	ncmp-datastore:passthrough-operational
A-10	ch-2	p1/c1	ncmp-datastore:passthrough-operational
A-10	ch-2	p2/c2	ncmp-datastore:passthrough-operational

Now based on whatever we have in the distributed map , we will send the response to the clients and may be clear the in-memory structure if required.

Example Flow 2: Second Subscription (merge)

We get a new subscription create request.

2nd Subscription Create Message

{
"subscriptionId" : "B-52",
"predicates" : [
  {
    "targets": [ch-2],
    "datastore" : "ncmp-datastore:passthrough-operational",
    "datastore-xpath-filter" : ["p2/c2" , "p3/c3"]
    }
]
}

We now look at the "ActiveSubscriptions" in the database if we already have an ongoing subscription either for a combination of "cmhandle and xpath filter and the datastore".

Current view of ActiveSubscriptions

SubscriptionId	cmHandle	filter	datastore
A-10	ch-1	p1/c1	ncmp-datastore:passthrough-operational
A-10	ch-1	p2/c2	ncmp-datastore:passthrough-operational
A-10	ch-2	p1/c1	ncmp-datastore:passthrough-operational
A-10	ch-2	p2/c2	ncmp-datastore:passthrough-operational

Check if we have ch-2 , p2/c2 and ncmp-datastore:passthrough-operational then directly add to the ActiveSubscription. Similarly check for ch-3 , p3/c3 and ncmp-datastore:passthrough-operational ( which in this case will not be directly to the active subscription)

Now the state of ActiveSubscription is :

SubscriptionId	cmHandle	filter	datastore
A-10	ch-1	p1/c1	ncmp-datastore:passthrough-operational
A-10	ch-1	p2/c2	ncmp-datastore:passthrough-operational
A-10	ch-2	p1/c1	ncmp-datastore:passthrough-operational
A-10	ch-2	p2/c2	ncmp-datastore:passthrough-operational
B-52	ch-2	p2/c2	ncmp-datastore:passthrough-operational

and state of the in-memory data structure is as follows.

SubscriptionId	dmi-plugin	affectedCmHandles	status
B-52	dmi-1	["ch-2"]	PENDING

Now we send the request to DMI plugin (dmi-1) with just the private properties of the ch-2 with the delta xpath.
1. DMI Plugin could either apply the whole subscription request or reject the whole subscription. i.e respond with accepted/rejected
Based on the response from DMI plugin the in-memory structure is updated and database also updated if the request is ACCEPTED and response is sent back to the clients.

if DMI says it is accepted then :
SubscriptionId
dmi-plugin
affectedCmHandles
status
B-52 dmi-1 ["ch-2"] ACCEPTED

if DMI says it is rejected then :
SubscriptionId
dmi-plugin
affectedCmHandles
status
B-52 dmi-1 ["ch-2"] REJECTED

if we get no response within the configured time then :
SubscriptionId
dmi-plugin
affectedCmHandles
status
B-52 dmi-1 ["ch-2"] PENDING

But we still send some response within the pre-configured time.

if the above request was accepted then we should add it to the list of "ActiveSubscriptions" as well in the database.

SubscriptionId	cmHandle	filter	datastore
A-10	ch-1	p1/c1	ncmp-datastore:passthrough-operational
A-10	ch-1	p2/c2	ncmp-datastore:passthrough-operational
A-10	ch-2	p1/c1	ncmp-datastore:passthrough-operational
A-10	ch-2	p2/c2	ncmp-datastore:passthrough-operational
B-52	ch-2	p2/c2	ncmp-datastore:passthrough-operational
B-52	ch-2	p3/c3	ncmp-datastore:passthrough-operational

If the status was accepted/rejected then after sending the response to the client we can clear the in-memory structure for subscriptionId B-52.

Storage Solution Discussion

20 Sep 2023 We had an internal meeting and concluded to do a POC with the below YANG schema to efficiently store the subscriptions.

Summary for Grooming

Note : This is the latest approach we are following. (Priyank Maheshwari as of 31 Jan 2024 )

Tracking Subscriptions in memory

#	cmhandle	dmi-plugin
1	ch-1	dmi-1
2	ch-2	dmi-1
3	ch-3	dmi-2
4	ch-4	dmi-2
5	ch-5	non-responding-dmi

Impl. Proposal CM Subscriptions : Creating/Merging( Positive Scenarios )

References

Assumptions

Issues & Decisions

Overview

Schema Definitions

Storage Solutions

Proposed Data Storage

Example Flow 1: First Subscription

Example Flow 2: Second Subscription (merge)

Storage Solution Discussion

Summary for Grooming