Overview

The second API proposed under the Delta feature is to generate a delta between configuration stored under an Anchor and JSON payload provided by the user. All the responses and exceptions thrown by this API will be similar to the API to generate delta between 2 anchors to maintain consistency between the API's.

Issues & Decisions

HTTP response codes for Delta between Anchor and Payload API

Proposed API

The proposed API will be part of the CPS Data Interface. The following response codes will be returned by the API:

...

HTTP Response codes

to be implemented

...

Proposed API:

POST- /v2/dataspaces/{dataspace-name}/anchors/{anchor-name}/delta?xpath={xpath}

Proposed method name:  CpsDataApi.getDeltaByDataspaceAnchorAndPayload()

...

.

...

                                            CpsDataApi.createDeltaByDataspaceAnchorAndPayload()

                                            CpsDataApi.fetchDeltaByDataspaceAnchorAndPayload()

...

Generate a delta report between an anchor and JSON payload

...

• 200 (OK)
  
  • success
• 400
  • dataspace not found
    DataspaceNotFoundException
  • anchor not found
    AnchorNotFoundException
  • Data node not found
    DataNodeNotFoundException
  • invalid xpath
    CpsPathException
• 500
  • unexpected error

Request parameters:

...

Alternative proposed API

This alternative API takes in an additional query parameter as an input, where in multipart form data as request body, using this the user can provide the JSON payload and its corresponding Schema Context in form of a Yang or ZIP file. This yang file will only be used to generate the data nodes from the json JSON payload and will not be persisted in CPS DB. The following response codes will be returned by the API:

Request parameters:

Request body:

Content-Type:

...

multipart/form-data

Response Body/Delta Report Format

[
  {
    "action": "ADDcreate",
    "xpath": "/bookstore/categories/[@code=3]",
    "target-data": {
      "code": 3,
      "name": "kidz"
    }
  },
  {
    "action": "DELETEremove",
    "xpath": "/bookstore/categories/[@code=1]",
    "source-data": {
      "code": 1,
      "name": "Fiction"
    }
  },
  {
    "action": "UPDATEreplace",
    "xpath": "/bookstore/categories/[@code=2]",
    "source-data": {
      "name": "Funny"
    },
    "target-data": {
      "name": "Comic"
    }
  }
]

Alternative approach for schema free delta report.

Problem Statement

The problem with 2nd Delta API is that it needs to compare any JSON payload to data stored under an anchor. This JSON payload can be from any source and regardless of its schema a delta should be generated between the payload and the data under the anchor.

The preferable approach here would be to parse the JSON payload into DataNodes and then compare the generated DataNodes against the DataNodes under the anchor. This would allow us to have a delta report which is in line with CPS and would enable us to provide crucial detail about the data nodes that is their xpath.

But since the second API accepts any JSON string as an input, one cannot be always sure that the payload will have the same schema as the anchor. And hence the approach of parsing to DataNodes can only happen when the schemas are identical.

In cases where schemas are not identical, there as soon as we attempt to parse the payload to DataNode, CPS will throw an exception terminating the parsing.

The most feasible alternative is to make use of JSON Nodes along with JSON path. Where JSON Nodes and JSON Path are used as a replacement for DataNodes and xPaths respectively.

Proposed Solutions

Use of JSON Nodes and JSON Path

Another alternative approach can be use of JSON Nodes and JSON Path as a replacement for DataNodes and xPaths respectively. In this approach, we can use JSON Paths to identify if a JSON Node has been added or deleted and if two identical JSON Paths exist in source anchor and the JSON payload then a comparison of JSON Nodes will be performed.

In order to achieve this the overall approach will be divided into two parts,

First, we determine if the JSON payload has the same schema as the data within the anchor. To do so the payload will be parsed against the schema information retrieved from the anchor and DataNodes will be formulated using this schema. If the parsing of JSON payload to DataNodes fails, then this would mean the schema differs and we switch to the alternative approach. If the parsing is successful we can go ahead with generating delta between the DataNodes.

If the parsing fails then, the data from source anchor needs to be fetched as a JSON string. And then both the JSON payload and the data from the anchor will be parsed to JSON Nodes and their respective JSON paths.

Once this is done a separate algorithm will be used to find the delta between JSON Nodes.

JSON Path

The most suitable candidate to be used as replacement for xpath are JSON path, a sample JSON path looks like this: 

$.store.book[0].title

Here:

• $ symbol refers to the root object or element.
• . operator is the dot-child operator, which you use to denote a child element of the current element.
• [ ] is the subscript operator, which you use to denote a child element of the current element (by name or index).

In order to utilize the JSON Path a new algorithm will need to be implemented to generate JSON paths from the available JSON Nodes.

Also, the existing delta algorithm can then be modified and reused for JSON path and JSON nodes.

Separation of Added Nodes from the JSON payload

Another approach to determine the delta between the anchor data and JSON payload can be implemented as follows.

• Step 1: The anchor data and JSON payload are fetched as JSON strings.
• Step 2: The JSON payload is compared to the JSON string from the anchor and all the newly added data nodes in JSON payload are extracted to a separate JSON string, while simultaneously removing this newly added data from the JSON payload.
• Step 3: After step 2 the JSON payload will be split into two JSON strings, one containing the Unmodified, Updated and Deleted data and second containing only the Added data.
• Step 4: The JSON string from anchor and the JSON string containing the Unmodified, Updated and Deleted data can then be parsed into DataNodes, as at this point, we can be sure that all the data in the particular JSON string will belong to the schema, which can then be used by the existing algorithm to find the delta. This part of delta report will contain the action, xpath and source/target data.
• Step 5: The added data can then be put in the delta report, but since this approach assumes that all the added data nodes are not part of the schema, it is not possible to determine their xpath and hence the field for xpath can be redacted for added data nodes in this approach.

Exapmle:

...

{
  "test:bookstore": {
    "bookstore-name": "Chapters/Easons",
    "categories": [
      {
        "code": "01/1",
        "name": "SciFi",
        "books": [
          {
            "authors": [
              "Iain M. Banks"
            ],
            "lang": "en/it",
            "price": "895",
            "pub_year": "1994",
            "title": "Feersum Endjinn/Endjinn Feersum"
          },{
            "authors": [
              "Ursula K. Le Guin"
            ],
            "lang": "en",
            "price": "1099",
            "pub_year": "1999",
            "title": "Far Horizons"
          }
        ]
      },{
        "name": "Horror",
        "code": "03",
        "books": [
          {
            "authors": [
              "ABC"
            ],
            "lang": "en",
            "price": "699",
            "pub_year": "1995",
            "title": "Horror Book"
          }
        ]
      }
    ],
  }
}

{
  "test:bookstore": {
    "bookstore-name": "Chapters/Easons",
    "categories": [
      {
        "code": "01/1",
        "name": "SciFi",
        "books": [
          {
            "authors": [
              "Iain M. Banks"
            ],
            "lang": "en/it",
            "price": "895",
            "pub_year": "1994",
            "title": "Feersum Endjinn/Endjinn Feersum"
          },
          {
            "authors": [
              "Ursula K. Le Guin"
            ],
            "lang": "en",
            "price": "1099",
            "pub_year": "1999",
            "title": "Far Horizons"
          }
        ]
      },
      {
        "name": "kids",
        "code": "02",
        "books": [
          {
            "authors": [
              "Philip Pullman"
            ],
            "lang": "en",
            "price": "699",
            "pub_year": "1995",
            "title": "The Golden Compass"
          }
        ]
      },
      {
        "name": "Horror",
        "code": "03",
        "books": [
          {
            "authors": [
              "ABC"
            ],
            "lang": "en",
            "price": "699",
            "pub_year": "1995",
            "title": "Horror Book"
          }
        ]
      }
    ],
    "bookstore-phone": "123",
    "bookstore-emp": { }
  },
  "test:bookstore2": {
    "bookstore-name": "Test"
  }
}

{
  "test:bookstore" : {
    "categories" : [ {
      "name" : "kids",
      "code" : "02",
      "books" : [ {
        "authors" : [ "Philip Pullman" ],
        "lang" : "en",
        "price" : "699",
        "pub_year" : "1995",
        "title" : "The Golden Compass"
      } ]
    } ],
    "bookstore-phone" : "123",
    "bookstore-emp" : { }
  },
  "test:bookstore2" : {
    "bookstore-name" : "Test"
  }
}

{
  "test:bookstore": {
    "bookstore-name": "Chapters/Easons",
    "categories": [
      {
        "code": "01/1",
        "name": "SciFi",
        "books": [
          {
            "authors": [
              "Iain M. Banks"
            ],
            "lang": "en/it",
            "price": "895",
            "pub_year": "1994",
            "title": "Feersum Endjinn/Endjinn Feersum"
          },
          {
            "authors": [
              "Ursula K. Le Guin"
            ],
            "lang": "en",
            "price": "1099",
            "pub_year": "1999",
            "title": "Far Horizons"
          }
        ]
      },
      {
        "name": "Horror",
        "code": "03",
        "books": [
          {
            "authors": [
              "ABC"
            ],
            "lang": "en",
            "price": "699",
            "pub_year": "1995",
            "title": "Horror Book"
          }
        ]
      }
    ]
  }
}

...

Functional Details about the API

The proposed API will be used to generate a delta between data fetched from an anchor against JSON payload provided by the user. The following are some key functional details about the API:

• The second Delta API utilizes the same algorithm which is used by the 1st CPS Delta API i.e. the API to generate delta between 2 anchors, by doing so we can ensure consistent results between both APIs, similar performance in delta generation and easier maintainability of code. 
• The second API fetches the data from the anchor provided as part of request and compares it to the raw JSON payload which is also sent as part of the request.
• In order to generate the delta report in the expected format, the JSON payload needs to be parsed into data nodes which would provide essential details such as xpath and leaf data.
• In order to parse the JSON payload to data nodes, the schema context of the JSON is required. The schema can be fetched using two approaches depending on the user's requirement:
  • The request body is of type multipart form data and accepts two parameters, one is the JSON payload as Text/String and the second is an optional parameter to provide the schema details in form of a yang or zip file. If the schema details are provided as part of request body, then the particular schema will be used to parse the JSON payload into data nodes.
  • If no Schema detail is provided as part of the request, then the algorithm will fetch the Schema Context using the anchor name, but here the assumption is that the JSON payload has same schema as that of the anchor.
  • If no schema is provided explicitly then the assumption is that the JSON payload has same schema as that of the anchor and it uses the anchor name to get the schema context which it, then uses to parse the JSON payload to data nodes. If in case, no schema is provided, and the schema of the anchor does not match the schema of the JSON payload then appropriate error message will be thrown.
• The descendants option is not provided as part of the request and is set to INCLUDE_ALL_DESCENDANTS by default, as one cannot be sure whether the JSON payload contains the descendants similar to the descendants in the anchor. So, the best-case scenario is to fetch all the descendants under the anchor and compare them to JSON payload.

Anchor
references references

References/Future Changes

• CPS-2310: Refactoring of CPS Delta APIs