CPS-251 [WIP] Define REST Query interface for Temporal service

CPS-251 - Getting issue details... STATUS

Overview

CPS temporal data query allows the user to fetch data based on multiple filtering criteria. This data can be used by an analytical system to take decisions.

Filtering Criteria

As it is time-series data, it can be filtered based on three main criteria:

Datetime

Data in last X hours
Data after a particular DateTime (To be supported in Istanbul Release)
Data between a date range
Last X records

DataTypes in CPS Core

Dataspace & schema-set (To be supported in Istanbul Release)
Dataspace & anchor (To be supported in Istanbul Release)
Dataspace & multiple anchors - To improve performance, if there is a need to fetch data for multiple anchors.

Payload (To be partially supported in Istanbul Release)

Based on a field in the payload. These criteria need schema-set to be fixed to search in the same set of anchors. The details of the format are defined here.

We will implement only the basic APIs first and will add complex ones when required.

Other Parameters

There are other parameters, listed below, that can have an impact on the response.

No	Name	Type	Default	Purpose
1	index	int	0	The query output can have many rows so it is important to limit the fetched data. To limit the number of records and to provide pagination, these two parameters can be used. "index" parameter represents the page number, starts with 0, and maxSize is the page size.
2	maxSize	int	1000
3	pointInTime	Timestamp (ISO Format)	CurrentDateTime	When a user uses the APIs with pagination (by running several stateless requests), the result set can be impacted by new states stored while navigating. This can be a challenge to retrieve a consistent result set. This parameter must be used to avoid this issue.
4	sortBy	string	LATEST_FIRST	Possible values - OLDEST_FIRST & LATEST_FIRST

Assumptions

Temporal database stores full information in the payload (CPS-192: Design data store for Temporal Service).
Data in the response body will always contain the entire payload data.

Payload Filter Format

The data can be filtered, using JSON format, on one field at any level, but it is important to provide the entire path from the root. The below JSON is used as an example to explain the filtering format

{
  "idNearRTRIC": "1",
  "labels" : [ "cps", "ran"],
  "nestedObject": {
    "id": "4",
    "attributes": {
      "gNBDUId": 4.0,
      "gNBDUName": "gnduserver4",
      "gNBIdLength": 32.0
    }
  },
  "GNBDUFunction": [
    {
      "idGNBDUFunction": "5",
      "NRCellDU": [
        {
          "idNRCellDU": "13905",
          "attributes": {
            "nRPCI": 12.0,
            "nRTAC": 310.0,
            "cellLocalId": 13905.0
          }
        },
        {
          "idNRCellDU": "14655",
          "attributes": {
            "nRPCI": 12.0,
            "nRTAC": 310.0,
            "cellLocalId": 14655.0
          }
        }
      ],
      "attributes": {
        "gNBDUId": 5.0,
        "gNBDUName": "gnduserver5",
        "gNBIdLength": 32.0
      }
    }
  ]
}

The filtering scenarios and the format are defined below:

No	Filter Condition	To be search	Format
1	Top-level Field	idNearRTRIC	{ "idNearRTRIC": "1"}
2	Nested Field	nestedObject → id	{ "nestedObject" : {"attributes" : {"gNBDUId": 4.0 }} }
3	Value(s) in a collection	labels	{ "labels": ["cps"] }
4	Field of an object inside a collection	GNBDUFunction → idGNBDUFunction	{ "GNBDUFunction" : [ {"idGNBDUFunction" : 5 }] }

Proposed APIs

The GET Endpoints URI contains information about the anchors. But, the format of the anchor data is different in both CPS Core and CPS-temporal, which can cause some confusion. Hence, two endpoint format have been proposed.

The filtering criteria are passed as query parameter. The below mentioned query parameters are supported by all endpoints.

after - timestamp ( ISO Format) to consider data created after the specified Timestamp
payload - to define payload filtering criteria
index - to define the page number (required for pagination)
maxSize - to limit the response size
pointInTime - to consider data changed or added before this time.
sortBy - to define the order of data in the result set

The response format for endpoints is the same and defined here

By Dataspace and Anchor

Endpoint Format A

URL: GET /dataspaces/{dataspace-name}/anchors/{anchor-name}

Example: GET /dataspaces/{dataspace-name}/anchors/{anchor-name}?after=2021-03-21T00:00:00-0:00&payload={"idNearRTRIC":"1"}&maxSize=500

Endpoint Format B

URL: GET /dataspaces/{dataspace-name}/anchor-data?anchor={anchor-name}

Example: GET /dataspaces/{dataspace-name}/anchor-data?anchor={anchor-name}&after=2021-03-21T00:00:00-0:00&payload={"idNearRTRIC":"1"}&maxSize=500

By Dataspace and Schema-set

Endpoint Format A

URL: GET /dataspaces/{dataspace-name}/anchors?schemaSet={schemaset-name}

Example: GET /dataspaces/{dataspace-name}/anchors?schemaSet={schemaset-name}&after=2021-03-21T00:00:00-0:00&payload={"idNearRTRIC":"1"}

Endpoint Format B

URL: GET /dataspaces/{dataspace-name}/anchors?schemaSet={schemaset-name}

Example: GET /dataspaces/{dataspace-name}/anchor-data?schemaSet={schemaset-name}&after=2021-03-21T00:00:00-0:00&payload={"idNearRTRIC":"1"}

Response

name	type
nextRecordsLink	string	added only if there are remaining records to be fetched for the query.
previousRecordsLink	string	added only if it is not the first set of records.
records	list	contains one record for each timestamp that meets filtering criteria. It contains header information along with data.

{
  "nextRecordsLink": "cps-temporal/api/v1/dataspaces/{dataspace-name}/anchors/{anchor-name}?after=2021-03-21T00:00:00-0:00&payload={"idNearRTRIC":"1"}&maxSize=500&pointInTime=2021-04-21T00:00:00-0:00&index=2",
  "previousRecordsLink": "cps-temporal/api/v1/dataspaces/{dataspace-name}/anchors/{anchor-name}?after=2021-03-21T00:00:00-0:00&payload={"idNearRTRIC":"1"}&maxSize=500&pointInTime=2021-04-21T00:00:00-0:00&index=0",
  "records": [
    {
      "timestamp": "2021-03-21T00:00:00-0:00",
      "dataspace": "my-dataspace",
      "schemaSet": "my-schema-set",
      "anchor": "my-anchor",
      "data": {
        "status" : "UP"
      }
    }
  ]
}