...
| Table of Contents |
|---|
Introduction
- CPS Temporal is a dedicated service, distinct and decoupled from CPS Core (separated option #4 from CPS-78: Deployment View).
- Integration between Core and Temporal is message event notification based, asynchronous, send and forget. By doing this, we are avoiding the dependency from CPS Core on CPS Temporal and its API. Actually, it reverses the dependency, which makes more sense from a conceptual point of view.
- For each data modification requesthandled by CPS Core,
- CPS Core is publishing, to a dedicated topic, a message an event representing the current data configuration or state.
- CPS Temporal is listening to the same topic for the event and is responsible to keep track of all data over time.
- In the future, some other services can be created to listen to the same topic in order to implement additional functionalities or storage forms.
- The event messaging system for this integration is either DMAAP or Kafka (to be deployed independently of CPS). The choice between one of the two is made by configuration when deploying CPS services.
- Messages Events published by CPS Core contains following information:
- Timestamp
- Dataspace
- Schema set
- Anchor
- Data (complete json data payload)
- A contract is defined to make the expected information explicit for the publisher and all subscribers. This contract can be provided by a structured message event schema format using JSON, Protobuf , or Avro or JSON.
Architecture Diagrams
Following diagrams are C4 model diagrams (context, containers and components) for CPS Temporal.
...
| Drawio | ||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Data Updated Event Schema
The structure to represent a Data Updated Event for CPS point of view need needs to to be defined without any ambiguity. This the interface contract to be rely on for:
- CPS Core to publish data updates
- CPS Temporal to listen to data updates
- Any other system listening to data updates
Here is an example of a CPS Data Updated Event is a JSON document that adheres to following JSON Schema:
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"schema": "urn:cps:org.onap.cps:data-updated-event-schema:1.0.0-SNAPSHOT",
"id": "38aa6cc6-264d-4ede-b534-18f5c1f403ea",
"source": "urn:cps:org.onap.cps",
"type": "org.onap.cps.data-updated-event",
"correlationId": "7515984f-064a-464f-8849-4c60f886ea4f",
"content": {
"timestamp": "2020-12-01T00:00:00.000+0000",
"dataspaceName": "my-dataspace",
"schemaSetName": "my-schema-set",
"anchorName": "my-anchor",
"data": {
"interface": {
"name": "itf-1",
"status": "up"
}
}
}
}
|
To be more formal, here is the JSON Schema that is representing any CPS Data Event:
| Code Block | ||||
|---|---|---|---|---|
| ||||
{
"$schema": "https://json-schema.org/draft/2019-09/schema",
"$id": "urn:cps:org.onap.cps:data-updated-event-schema:1.0.0-SNAPSHOT",
"$ref": "#/definitions/CpsDataUpdatedEvent",
"definitions": {
"CpsDataUpdatedEvent": {
"titledescription": "CpsDataUpdatedEvent",The "description": "Apayload for CPS data updated event.",
"type": "object",
"properties": {
"schema": {
"description": "The schema, including its version, that this event adheres to.",
"const": "urn:cps:org.onap.cps:data-updated-event-schema:1.0.0-SNAPSHOT"
},
"id": {
"description": "The unique schema,id includingidentifying itsthe version,event thatfor the event adheres to."
} specified source. Producer must ensure that source + id is unique for each distinct event.",
"idtype": { "string"
},
"typesource": "string",{
"description": "The uniquesource id identifyingof the event for the specified source. ProducersProducer must ensure that source + id is unique for each distinct event."
,
}, "sourcetype": {"string",
"typeformat": "string",uri"
},
"formattype": "uri",{
"description": "The sourcetype of the event.",
Producers must ensure that source + id is unique for each distinct event."type": "string"
},
"typecorrelationId": {
"typedescription": "stringA unique identifier that may be used to correlate and trace information between several systems.",
"descriptiontype": "The type of the event.string"
},
"content": {
"$ref": "#/definitions/Content"
}
},
"required": [
"schema",
"id",
"source",
"type",
"content"
],
"additionalProperties": false
},
"Content": {
"title": "Content",
"description": "The event content.",
"type": "object",
"properties": {
"timestamp": {
"description": "The timestamp when the data has been observed.",
"type": "string"
},
"coreReferencesdataspaceName": {
"description": "The name of CPS Core dataspace the data belongs to.",
"$reftype": "#/definitions/CoreReferenceListstring"
},
"dataschemaSetName": {
"$refdescription": "#/definitions/Data"
}
The name of CPS Core schema set the data adheres to.",
}, "additionalPropertiestype": "string"
false },
"CoreReferenceListanchorName": {
"title": "CoreReferenceList", "description": "AllThe name of CPS Core referencesanchor forthe data is attached to.",
"type": "objectstring",
"properties": { },
"dataspacedata": {
"$ref": "#/definitions/CoreReferenceData"
},
},
"schemaSetrequired": {
[
"$ref": "#/definitions/CoreReference"timestamp",
}"dataspaceName",
"anchor": {
schemaSetName",
"$ref": "#/definitions/CoreReference"anchorName",
}"data"
}],
"additionalProperties": false
},
"CoreReferenceData": {
"titledescription": "CoreReferenceData as json object.",
"descriptiontype": "A CPS Core reference for data.","object"
}
}
}
|
CPS Event API
Having CPS Data Updated Event formalized, we could be able to use at AsyncAPI specification to define CPS Event Driven APIs in similar way we are using OpenAPI to define REST APIs:
| Code Block | ||||
|---|---|---|---|---|
| ||||
asyncapi: 2.0.0 info: title: CPS Event API version: 1.0.0-SNAPSHOT description: This specification describes CPS event driven interfaces. channels: cps/data/updated: description: This channel is the one on witch CPS Core is publishing notifications when its data is updated. subscribe: message: $ref: '#/components/messages/DataUpdated' components: messages: DataUpdated: payload: description: The payload for CPS data updated event. "type": "object",object properties: schema: description: The schema, including its version, that this event adheres to. "properties": { const: urn:cps:org.onap.cps:data-updated-event-schema:1.0.0-SNAPSHOT "id": {id: description: The unique id identifying the event for the specified source. Producer must ensure that source + id is unique for each distinct event. "type": "string",string source: "description": "The unique id identifying the reference." : The source of the event. Producer must ensure that source + id is unique for each distinct event. type: string format: uri type: description: The type of the event. type: string correlationId: description: A unique identifier that may be used to correlate and trace information between several systems. type: string content: type: object properties: timestamp: }, "name": {description: The timestamp when the data has been observed. "type": "string",string dataspaceName: "description": "The name of CPS Core dataspace the data belongs reference."to. type: string schemaSetName: } }, "additionalProperties": falsedescription: The name of CPS Core schema set the data adheres to. }, "Data": {type: string "title": "Data",anchorName: "description": "The json data content payload",The name of CPS Core anchor the data is attached to. type: string data: description: Data as json object. "type": "object"object required: - timestamp - dataspaceName - schemaSetName - anchorName - data additionalProperties: false required: - schema - id - source - type - content } additionalProperties: } }false |
AsyncAPI Playground can be used to visualize the specification. Then, it could be worth looking at documentation and code generation features.
Binary Format Alternatives
Instead of JSON, another option to exchange event over the network would be to use a serialized binary format such as Protocol Buffers or Apache Avro.
Option 1: Plain JSON Text
Observing message data is straightforward.
No additional step to serialize and de-serialize data.
If the code and classes to handle the event is not generated, either it needs to be provided or it will be left to producers and consumers responsibility.
Option 2: Binary
Producers and consumers benefit from code generation tools to handle event data. This is efficient and reliable for consistency between producers and consumers.
Binary formats are optimized for payload size and efficient for network transport (might not be relevant for CPS as most of the data payload belongs to an unique field).
Additional step to serialize and de-serialize data.
As current producer and consumer are part of the same team, the proposition is to start with JSON Text format and put in place a reliable process to manage and use the schema within the team.
CPS Core Changes
- As being the event producer, CPS Core is taking ownership of the event schema definition. Also, if the code and the classes to handle the event are not automatically generated, CPS Core could provide them to avoid code duplication. These need to be provided in a separated Maven module, delivered to Nexus and then imported as dependency by subscribers.
- The event publication is CPSDataService responsibility. After its call to DataPersistenceService is completed, it sends a Data Updated Event using a Data Updated Event Producer component. The event contains some DataNode information and JSON Data.
- CPS Core event publications are controlled by an overall general application sytem environment variable that enable or disable them. By default they are disabled.
Data Ownership and Access
CPS Temporal is part of and internal to overall CPS system.
CPS Temporal service is following CPS data ownership and access rules:
- CPS Temporal querying clients are authenticated and authorized.
- CPS Temporal querying clients only has access to dataspaces and models they own or they have been granted to in CPS Core.
- DMaaP / Kafka event topic is CPS system internal, It should be accessible to CPS components only.
Going Further
- Other types of events: Data Deleted, Data Leaf Updated (partial node). Support for these types of events within the same schema.
- Additional levels of filtering to send notifications or not are seen on top of the overall general application one:
- Anchor level
- Yang specific path level (using extension ?)
- DMaaP / Kafka authentication and authorization.
- Timestamp format on the event (ISO)