A&AI Pagination

1 What is it?
2 Why have it?
3 How is it implemented?
- 3.1 curl example
4 An Example
5 Important Additional Information

What is it?

The ability to provide queries with large response sizes and/or long running times in bucketed chunks, ie. Get all calls would be a good candidate for this

Why have it?

Pagination can help both api consumers and save our server resources
By returning results in chunks, this allows for quicker response times, less constant resource load on our server (avoids tying up a thread for an extended period of time) and provides more manageable amounts of data
The most common use case for using pagination is any UI requesting data (as speed is key, you don’t want to keep your user waiting) or any client that is concerned with performance or response size and can handle receiving results in chunks

How is it implemented?

Currently only implemented in the resources & traversal microservices
Accepts resultSize and resultIndex as query parameters
The resultSize is an integer passed in by the client as a query parameter that specifies what amount of results should be returned
The resultIndex is an integer that is passed in by a client as a query parameter that specifies which bucket the client wants back ie. for a resultSize of 10 an index of 1 would return 1-10, index of 2 would return 11-20 etc
The results are returned with the below information in the header, the parameters passed in, total amount of results, and total amount of pages
Optionally we could choose to include a flag to say has more results for ease of use, but this could also be derived by checking if resultIndex == total-pages

curl example

curl --verbose --silent --insecure --user AAI:AAI --header 'Accept: application/json' \
 --header 'X-TransactionId: testaai' --header 'Content-Type: application/json' --header 'X-FromAppId: AAI' \
 'https://<AAI-IP>:30233/aai/v14/nodes/pnfs?resultIndex=1&resultSize=10'


> GET /aai/v14/nodes/pnfs?resultIndex=1&resultSize=10 HTTP/1.1^M
> Authorization: Basic QUFJOkFBSQ==^M
> User-Agent: curl/7.29.0^M
> Host: <AAI-IP>:30233^M
> Accept: application/json^M
> X-TransactionId: testaai^M
> Content-Type: application/json^M
> X-FromAppId: AAI^M
> ^M
< HTTP/1.1 200 OK^M
< Date: Mon, 18 Feb 2019 23:40:49 GMT^M
< vertex-id: 82300928^M
< total-results: 1^M
< total-pages: 1^M
< Content-Type: application/json^M
< X-AAI-TXID: 2-aai-resources-190218-23:40:49:594-65208^M
< Content-Length: 158^M
< Strict-Transport-Security: max-age=16000000; includeSubDomains; preload;^M
< ^M
{ [data not shown]
* Connection #0 to host <AAI-IP> left intact
{
  "pnf": [
    {
      "pnf-name": "pnf-test-3",
      "pnf-name2": "name 2-3",
      "pnf-id": "test-id-3",
      "equip-type": "test-type-3",
      "in-maint": false,
      "resource-version": "1550533245515"
    }
  ]
}

In order for pagination to work properly the order must remain consistent
The way this is achieved is the vertices are pre sorted on disk by vertex-id (this is done by default)
When a call is made to retrieve all vertices we get them sorted by vertex-id (since they are indexed this is very quick)
We then take a sublist of those vertices based on the pagination parameters and gather all information associated to them (this step and the network transfer of less data is where the time/resources is saved)

An Example

GET ALL https://{{hostname}}:{{port}}/aai/{{version}}/cloud-infrastructure/pservers 11208 ms 3.54mb 8983 results
GET Page 1 of 1000 results https://{{hostname}}:{{port}}/aai/{{version}}/cloud-infrastructure/pservers?resultIndex=1&resultSize=1000 911 ms 408kb 1000 results
GET Page 2 of 1000 results https://{{hostname}}:{{port}}/aai/{{version}}/cloud-infrastructure/pservers?resultIndex=2&resultSize=1000 901 ms 416kb 1000 results
GET Page 3 of 1000 results https://{{hostname}}:{{port}}/aai/{{version}}/cloud-infrastructure/pservers?resultIndex=3&resultSize=1000 937 ms 407kb 1000 results
GET Page 4 of 1000 results https://{{hostname}}:{{port}}/aai/{{version}}/cloud-infrastructure/pservers?resultIndex=4&resultSize=1000 1009 ms 409kb 1000 results
GET Page 5 of 1000 results https://{{hostname}}:{{port}}/aai/{{version}}/cloud-infrastructure/pservers?resultIndex=5&resultSize=1000 1029 ms 397kb 1000 results
GET Page 6 of 1000 results https://{{hostname}}:{{port}}/aai/{{version}}/cloud-infrastructure/pservers?resultIndex=6&resultSize=1000 878 ms 399kb 1000 results
GET Page 7 of 1000 results https://{{hostname}}:{{port}}/aai/{{version}}/cloud-infrastructure/pservers?resultIndex=7&resultSize=1000 886 ms 394kb 1000 results
GET Page 8 of 1000 results https://{{hostname}}:{{port}}/aai/{{version}}/cloud-infrastructure/pservers?resultIndex=8&resultSize=1000 919 ms 404kb 1000 results
GET Page 9 of 983 results https://{{hostname}}:{{port}}/aai/{{version}}/cloud-infrastructure/pservers?resultIndex=9&resultSize=1000 940 ms 394kb 983 results

Important Additional Information

If you are doing processing using pagination be aware that the results are returned in chunks dynamically for the chunks present at the given time of the query, there is no concept of persistence of an original set of results.

So say you come in and ask for resultIndex=1&resultSize=1 and there are 10 entries

[Node 1], Node 2, Node 3, Node 4, Node 5, Node 6, Node 7, Node 8, Node 9, Node 10

Then you say resultIndex=2&resultSize=1 you get that Node 2

Node 1, [Node 2], Node 3, Node 4, Node 5, Node 6, Node 7, Node 8, Node 9, Node 10

Then say Node 1 gets deleted

Node 2, Node 3, Node 4, Node 5, Node 6, Node 7, Node 8, Node 9, Node 10

Then say you ask for resultIndex=3&resultSize=1 you would get Node 4 because now the entire set is different

Node 2, Node 3, [Node 4], Node 5, Node 6, Node 7, Node 8, Node 9, Node 10