AAI History Documentation

Owned by William Reehil
Last updated: May 30, 2018 by Dan Silverthorn
8 min read

Table Of Contents

Needed Stories



LIFESPAN not to be supported in Casablanca.

Chameleon/Gallifrey need to handle all deployment stories (dockerization/runbook/etc), and adhere to junit coverage - amdocs to create these stories

Historical Meta-properties

Outlined in the following ppt: historyScenarios.pptx

Historical Work Breakdown

Historical Tracking Iterations Story Assignment.xlsx

mS Pros/Cons

Why are we breaking the functionality out into these granular levels (mainly for scalability, deployment/maintenance flexibility)

  • Independent Development – All microservices can be easily developed based on their individual functionality

  • Independent Deployment – Microservices can be individually deployed in any application

  • Fault Isolation – Even if one service of the application does not work, the system still continues to function and the fault is easily detectable

  • Mixed Technology Stack – Different languages and technologies can be used to build different services of the same application

  • Granular Scaling – Individual components can scale as per need, there is no need to scale all components together

  • Ease of Unit Testing - unit testing is easier to maintain as functionality per mS is isolated

Some disadvantages of splitting out the functionality into these granular microservices are:

  • Full stack error log traceability

  • Latency introduced

  • Deployment more complicated

  • e2e testing more complicated

  • more points of failure

High Level Design of microservice flow

Resources - Client exposed endpoints

Gizmo - CRUD abstraction subsystem

Synapse - Data/request router, will handle the traffic proxying to various microservices based on its built in rules

Champ - General purpose graph database abstraction

Spike - publishes dmaap or kafka events and attempts to ensure order

Chameleon -  Subsystem that processes spike events from the real-time graph updates and formats the events, entry point to gallifrey/time machine, and enforces/formats the requests into the format needed by Gallifrey/Time Machine

Gallifrey/Time Machine - Subsystem that makes & retrieves historical assertions



PUT/POST/PATCH/DELETE to real-time flow

Resources > Gizmo > Champ > Real-time DB Cluster

this triggers the historical storage flow

Champ > Spike > Chameleon > Gallifrey > Historical Champ > Historical DB Cluster

GET of real-time data

Resources > Gizmo > Synapse > Real-time Champ > Real-time DB Cluster

GET of historical data

Resources > Gizmo > Synapse > Chameleon > Gallifrey/Time Machine > Historical Champ > Historical DB Cluster

I feel Chameleon could be bypassed for this flow, treating chameleon's functionality as just the dmaap history processor, and synapse would call gallifrey/tm directly. This way if chameleon goes down, gallifrey/tm could still service historical GET requests.

Resources > Gizmo > Synapse > Chameleon > Gallifrey/Time Machine > Historical Champ > Historical DB Cluster

Resources API Spec



Resources will be updated to accept a timestamp or a network timestamp and will trickle down through gizmo to synapse then to chameleon.

If the timestamp was sent on a non-singular node call then we would return a message stating that this functionality is not supported.

Gallifrey/Time Machine API Spec:



Type

URI

Query Params

Description

Champ Interaction

GET

relationship/<ID>

t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM

t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network



Retrieve a relationship by ID



Champ needs to handle accepting a relationship id and the timestamp to run a historical query on the graph db

A subgraph strategy will be used in champ to filter on the relative timestamp provided.

meta=[true/false] if true, payload retrieved will hold the metaproperties at t-k or t-t

GET

relationship/<ID>/lifespan

NOT in scope for Casablanca - decided on call 5/23/2018

None

Retrieve all the timestamps for create, update, delete operations against this relationship

Champ would be called to retrieve the lifespan on the relationship with meta=true and t-k=lifespan to retrieve all the metaproperties on the relationship

GET

entity/<ID>

t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM

t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network



Retrieve an entity by ID

Champ needs to handle accepting an entity id and the timestamp to run a historical query on the graph db retrieving the asserted state. Default not sending back metaproperties, if metaproperties are needed a parameter would need to be sent to champ.

A subgraph strategy will be used in champ to filter on the relative timestamp provided.

meta=[true/false] if true, payload retrieved will hold the metaproperties at t-k

GET

entity/<ID>/lifespan

NOT in scope for Casablanca - decided on call 5/23/2018

None

Retrieve all the timestamps for create, update, delete operations against this entity



Champ would be called to retrieve the lifespan on the entity with meta=true and t-k=lifespan to retrieve all the metaproperties on the entity

PUT

relationship/<ID>

actor=name of the system making the assertion

changes-only=[true|false] if true, gallifrey/tm will actually determine what has changed between the PUT payload and the most recent set of assertions for the relationship. If false, the entire PUT body will be considered as a new set of assertions whether something has changed or not.

create=[true|false] if true, Gallifrey/TM assumes that this is a create request, if false it assumes it is an update

t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network



Asserts that a relationship is to be created or updated (depending on the query parameters that are passed in). This API appends new assertions against the specified relationship.

Generated in Gallifrey/TM

t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM (why are we generating this here? if there is a maintenance issue our timings would be out of sync with when these took place in the real-time db)

create = false (changes-only true (execute diff)/false(assume everything changed))

When an assertion is being made without a network timestamp gallifrey/tm will call champ requesting the relationship with it's most current metaproperties. Gallifrey/TM would then adjust the metaproperties (of the updated properties) and would send the payload back to champ with a new current state and an updated previous state's metaproperties. Champ would override it's current metaproperty (for the updated/deleted properties) with the old and current metaproperties sent from Gallifrey/TM. For added properties, they would be added directly with the metaproperties sent from Gallifrey/TM.

When an assertion is being made with a network timestamp gallifrey/tm will call champ requesting the relationship with all of its metaproperties. Gallifrey/TM would then insert the new assertion where appropriate (and adjust neighboring metaproperties) and send the modified payload back to champ for a replace.

create = true

POST - this would be a new create and Gallifrey/TM would pass the metaproperties on each of it's property values and on the relationship itself

PUT

entity/<ID>

actor=name of the system making the assertion

changes-only=[true|false] if true, gallifrey/tm will actually determine what has changed between the PUT payload and the most recent set of assertions for the entity. If false, the entire PUT body will be considered as a new set of assertions whether something has changed or not.

create=[true|false] if true, Gallifrey/TM assumes that this is a create request, if false it assumes it is an update

t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network

Asserts that an entity is to be created or updated (depending on the query parameters that are passed in). This API appends new assertions against the specified entity.

Generated in Gallifrey/TM

t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM

create = false (changes-only true (execute diff)/false(assume everything changed))

When an assertion is being made without a network timestamp gallifrey/tm will call champ requesting the entity with it's most current metaproperties. Gallifrey/TM would then adjust the metaproperties (of the updated properties) and would send the payload back to champ with a new current state and an updated previous state's metaproperties. Champ would override it's current metaproperty (for the updated/deleted properties) with the old and current metaproperties sent from Gallifrey/TM. For added properties, they would be added directly with the metaproperties sent from Gallifrey/TM.

When an assertion is being made with a network timestamp gallifrey/tm will call champ requesting the entity with all of its metaproperties. Gallifrey/TM would then insert the new assertion where appropriate (and adjust neighboring metaproperties) and send the modified payload back to champ for a replace.

create = true

POST - this would be a new create and Gallifrey/TM would pass the metaproperties on each of it's property values and on the entity itself

DELETE

relationship/<ID>

actor=name of the system making the assertion

t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network

Asserts that a relationship has been deleted.

Generated in Gallifrey/TM

t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM

Gallifrey/TM would request the latest relationship from champ and would set all of it's properties dbEndTimes to t-k along with the dbEndTime on the relationship itself.

DELETE

entity/<ID>

actor=name of the system making the assertion

t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network

Asserts that an entity has been deleted.

Generated in Gallifrey/TM

t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM



Gallifrey/TM would request the latest entity from champ and would set all of it's properties dbEndTimes to t-k along with the dbEndTime on the entity itself.



Chameleon API Spec



Type

URI

Query Params

Description

GET

relationship/<ID>

t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM

t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network

Retrieve a relationship by ID

GET

entity/<ID>

t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM

t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network

Retrieve an entity by ID





Between Chameleon->Gallifrey/TM, the following calls are made:



PUT entity/<ID>?actor=aai&create=true

PUT relationship/<ID>?actor=aai&create=true

PUT entity/<ID>?actor=aai&changes-only=true

PUT relationship/<ID>?actor=aai&changes-only =true

DELETE entity/<ID>?actor=aai

DELETE relationship/<ID>?actor=aai

GET entity/<ID>?t-k=<timestamp>

GET relationship/<ID>? t-k=<timestamp>

Need Chameleon updated with this:

GET entity/<ID>?t-t=<timestamp>

GET relationship/<ID>? t-t=<timestamp>

Champ API Spec

Every entity/relationship/property must have a metaproperty or set of meta properties, this will be enforced in champ.

POST an entity- For History provide meta-properties in the payload

Method=POST

https://<host>:9522/services/champ-service/v1/objects

Retrieve an entity across its lifespan

Method= GET, As default,meta=false

https://<host>:9522/services/champ-service/v1/objects/<key>

https://<host>:9522/services/champ-service/v1/objects/<key>&meta=true

(called when updating with network time)

https://<host>:9522/services/champ-service/v1/objects/<key>&meta=latest

(called when updating with only db time)

Retrieve an object for a given database timestamp

Method= GET, As default,meta=false

https://<host>:9522/services/champ-service/v1/objects/<key>?t-k=t1

https://<host>:9522/services/champ-service/v1/objects/<key>?t-k=t1&meta=true

Retrieve an object for a given network timestamp

Method= GET, As default,meta=false

https://<host>:9522/services/champ-service/v1/objects/<key>?t-t=t1

https://<host>:9522/services/champ-service/v1/objects/<key>?t-t=t1&meta=true

UPDATE an object

Method=PUT - default=[patch=false] which means full replace

https://<host>:9522/services/champ-service/v1/objects/<key>

https://<host>:9522/services/champ-service/v1/objects/<key>?patch=true

DELETE an object - DELETE

Method=PUT - default=[patch=false] which means full replace

https://<host>:9522/services/champ-service/v1/objects/<key>

https://<host>:9522/services/champ-service/v1/objects/<key>?patch=true

POST a relationship - For History provide meta-properties in the payload

Method: POST

URL: https://<host>:9522/services/champ-service/v1/relationships

Retrieve a relationship across its lifespan

Method= GET, As default,meta=false

URL: https://<host>:9522/services/champ-service/v1/relationships/<key>

URL: https://<host>:9522/services/champ-service/v1/relationships/<key>&meta=true

Retrieve a relationship for a given database timestamp

Method= GET, As default,meta=false

URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-k=t1

URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-k=t1&meta=true

Retrieve a relationship for a given network timestamp

Method= GET, As default,meta=false

URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-t=t1

URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-t=t1&meta=true

UPDATE an object

Method=PUT - default=[patch=false] which means full replace

URL: https://<host>:9522/services/champ-service/v1/relationships/<key>

URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?patch=true

DELETE an object - DELETE

Method=PUT - default=[patch=false] which means full replace

URL: https://<host>:9522/services/champ-service/v1/relationships/<key>



Historical Tracking Metaproperties

dbStartTime - when an entity/property value/relationship was added to db

dbEndTime - when and entity or relationship was deleted from the db or a property's value was asserted to another state

ntStartTime - asserted by the client as to when the change took place in the network

ntEndTime - set when an assertion provided by the client make's the current state no longer true

startSOT - the source of truth that made the assertion

endSOT- the source of truth that made an assertion to make the current state no longer true

Schema

Separate db edge rules file: with all relationships many to many except parent child which could be one to many

Separate schema file: no properties on vertices are unique except for aai-uuid

Open question: what happens in the event that db edge rules change etc. (migrations)

When To Record T-K value (Live DB vs Historical DB)

Below is an explanation of the thoughts around when to record the t-k value, examples explaining outcome of storing at live vs historical DB and their pros/cons.

GUI Mocks

New integrated functionality (updates for history)





Existing GUI

Topology Current view

Updates include a history button, which pops up a date/time picker to replace the topology view with that historical topology. In the bottom right, it will designate the time period, and if not current the ability to revert back to current.

Compare pops up different date/time picker for a secondary topology. to compare against.

Compare View