CPS-1567: Investigate Implementation of pagination for Query API
- Rajesh Kumar
Jira Refs: CPS-1605: Implementation of pagination through anchors in query APIClosed
Background
To query large outputs or large number of fragments across all anchors, we need to provide pagination for Query API(query data nodes across all anchors). it will help client to fetch limited set of data per request. below are two possible solutions for further discussions to decide appropriate choice.
Issues and Decisions
# | Issues | Notes | Decision |
|---|---|---|---|
1 | How to support pagination. | we need to decide on how to support pagination for query API from two possible alternatives given below (pagination over fragments or pagination over anchors) | Mar 22, 2023 it was discussed in weekly meeting and decided by team members to implement pagination over anchors. |
2 | Default page size | should we go with some default page size or query all records if pagination parameter in not provided. | Apr 10, 2023 By default if pagination parameter in not provided then query all records. |
Pagination over fragments
Implementation
API : GET http://<IP>:<PORT>/cps/api/v2/dataspaces/{dataspace-name}/nodes/query?cps-path={cps-path}&descendants={descendants}&pageIndex={pageIndex}&pageSize={pageSize}
Request Parameters:
Parameter | isRequired | Description |
|---|---|---|
cps-path | yes | node path to be queried. |
descendants | no | Number of descendants to be queried. default is none. |
pageIndex | no | page index starting from 1. default is -1 for all records. |
resultSize | no | number of fragments per page. default is -1 for all fragments. |
Query
Query | Description |
|---|---|
SELECT id, anchor_id AS anchorId, xpath, parent_id AS parentId, CAST(attributes AS TEXT) AS attributes FROM FRAGMENT WHERE xpath ~ :xpathRegex ORDERED BY id LIMIT :resultSize OFFSET (:resultIndex - 1) * :resultSize | Query limited fragments in order of fragment id |
Pagination over anchors
we can provide pagination over anchors where client can choose number of anchors per request to fetch fragments from.
Implementation
API : GET http://<IP>:<PORT>/cps/api/v2/dataspaces/{dataspace-name}/nodes/query?cps-path={cps-path}&descendants={descendants}&pageIndex={pageIndex}&pageSize={pageSize}
Request Parameters:
Parameter | isRequired | Description |
|---|---|---|
cps-path | yes | node path to be queried |
descendants | no | Number of descendants to be queried. default is none |
pageIndex | no | page index starting from 1. default is -1 for all pages(all anchors) |
pageSize | no | number of records per request. It is nothing but number of anchors to be queried. Default is -1 for all pages(all anchors) |
Response Headers
Header | description |
|---|---|
totalPages | Total number of pages for given page size |
Response body
Format | Example |
|---|---|
[ {"anchorName":"anchor-01", "dataNodes":[{data-tree1},...{data-tree-n}] ... ,{"anchorName":"anchor-02", "dataNodes": [{datatree1},...,{data-tree-n}] } ] | Example[
{
"anchorName":"anchor01",
"dataNodes":[
{
"test:bookstore":{
"bookstore-name":"Chapters",
"categories":[
{
"code":1,
"name":"SciFi"
}
]
}
}
]
},
{
"anchorName":"anchor02",
"dataNodes":[
{
"test:bookstore":{
"bookstore-name":"store01",
"categories":[
{
"code":2,
"name":"SciFi"
}
]
}
}
]
}
] |
Query executed in persistence layer.
Query | Description |
|---|---|
SELECT id FROM anchor order by id LIMIT : pageSize OFFSET : (pageIndex -1) * pageSize | query list of anchors. |
SELECT id, anchor_id AS anchorId, xpath, parent_id AS parentId, CAST(attributes AS TEXT) AS attributes FROM FRAGMENT WHERE anchor_id IN :anchorList AND xpath ~ :xpathRegex | query fragments for list of anchors from above query. |