Background
To query large outputs or large number of fragments across all anchors, we need to provide pagination for Query API( query data nodes across all anchors). it will help client to fetch limited set of data per request. below is two possible solutions for further discussions to decide appropriate choice.
Pagination over fragments
Implementation
API : GET http://<IP>:<PORT>/cps/api/v2/dataspaces/{dataspace-name}/nodes/query?cps-path={cps-path}&descendants={descendants}&pageIndex={pageIndex}&pageSize={pageSize}
Request Parameters:
| Parameter | isRequired | Description |
|---|---|---|
| cps-path | yes | node path to be queried |
| descendants | no | Number of descendants to be queried. default is none |
| pageIndex | no | page index starting from 1. |
| pageSize | no | number of fragments per page starting from 1 (TBD). |
Query
| Query | Description | Decision |
|---|---|---|
| SELECT id, anchor_id AS anchorId, xpath, parent_id AS parentId, CAST(attributes AS TEXT) AS attributes FROM FRAGMENT WHERE xpath ~ :xpathRegex ORDERED BY id LIMIT :pageSize OFFSET (:pageIndex - 1) * :pageSize | Query limited fragments in order of fragment id | |
Pagination over anchors
we can provide pagination over anchors where client can chose number of anchors per request to fetch fragments from.
Implementation
API : GET http://<IP>:<PORT>/cps/api/v2/dataspaces/{dataspace-name}/nodes/query?cps-path={cps-path}&descendants={descendants}&pageIndex={pageIndex}&anchorSize={anchorSize}
Request Parameters:
| Parameter | isRequired | Description |
|---|---|---|
| cps-path | yes | node path to be queried |
| descendants | no | Number of descendants to be queried. default is none |
| pageIndex | no | page index starting from 1. |
| anchorSize | no | number of anchors per page starting from 1 (TBD). |
Query:
| Query | Description | Decision |
|---|---|---|
| SELECT id FROM anchor order by id LIMIT : anchorSize OFFSET : (pageIndex -1) * :anchorSize | query number of anchors to query from fragment table | |
| SELECT id, anchor_id AS anchorId, xpath, parent_id AS parentId, CAST(attributes AS TEXT) AS attributes FROM FRAGMENT WHERE anchor_id IN :anchorList AND xpath ~ :xpathRegex | query fragments for list of anchors from above query |