Table Of Contents
Needed Stories
LIFESPAN not to be supported in Casablanca.
Chameleon/Gallifrey need to handle all deployment stories (dockerization/runbook/etc), and adhere to junit coverage - amdocs to create these stories
Historical Meta-properties
Outlined in the following ppt: historyScenarios.pptx
Historical Work Breakdown
Historical Tracking Iterations Story Assignment.xlsx
mS Pros/Cons
Why are we breaking the functionality out into these granular levels (mainly for scalability, deployment/maintenance flexibility)
- Independent Development – All microservices can be easily developed based on their individual functionality
- Independent Deployment – Microservices can be individually deployed in any application
- Fault Isolation – Even if one service of the application does not work, the system still continues to function and the fault is easily detectable
- Mixed Technology Stack – Different languages and technologies can be used to build different services of the same application
- Granular Scaling – Individual components can scale as per need, there is no need to scale all components together
- Ease of Unit Testing - unit testing is easier to maintain as functionality per mS is isolated
Some disadvantages of splitting out the functionality into these granular microservices are:
- Full stack error log traceability
- Latency introduced
- Deployment more complicated
- e2e testing more complicated
- more points of failure
High Level Design of microservice flow
Resources - Client exposed endpoints
Gizmo - CRUD abstraction subsystem
Synapse - Data/request router, will handle the traffic proxying to various microservices based on its built in rules
Champ - General purpose graph database abstraction
Spike - publishes dmaap or kafka events and attempts to ensure order
Chameleon - Subsystem that processes spike events from the real-time graph updates and formats the events, entry point to gallifrey, and enforces/formats the requests into the format needed by Gallifrey
Gallifrey - Subsystem that makes & retrieves historical assertions
PUT/POST/PATCH/DELETE to real-time flow
Resources > Gizmo > Champ > Real-time DB Cluster
this triggers the historical storage flow
Champ > Spike > Chameleon > Gallifrey > Historical Champ > Historical DB Cluster
GET of real-time data
Resources > Gizmo > Synapse > Real-time Champ > Real-time DB Cluster
GET of historical data
Resources > Gizmo > Synapse > Chameleon > Gallifrey > Historical Champ > Historical DB Cluster
I feel Chameleon could be bypassed for this flow, treating chameleon's functionality as just the dmaap history processor, and synapse would call gallifrey directly. This way if chameleon goes down, gallifrey could still service historical GET requests.
Resources > Gizmo > Synapse > Chameleon > Gallifrey > Historical Champ > Historical DB Cluster
Resources API Spec
Resources will be updated to accept a timestamp or a network timestamp and will trickle down through gizmo to synapse then to chameleon.
If the timestamp was sent on a non-singular node call then we would return a message stating that this functionality is not supported.
Gallifrey API Spec:
Type | URI | Query Params | Description | Champ Interaction |
GET | relationship/<ID> | t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Retrieve a relationship by ID | Champ needs to handle accepting a relationship id and the timestamp to run a historical query on the graph db A subgraph strategy will be used in champ to filter on the relative timestamp provided. meta=[true/false] if true, payload retrieved will hold the metaproperties at t-k or t-t |
GET | relationship/<ID>/lifespan | None | Retrieve all the timestamps for create, update, delete operations against this relationship | Champ would be called to retrieve the lifespan on the relationship with meta=true and t-k=lifespan to retrieve all the metaproperties on the relationship |
GET | entity/<ID> | t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Retrieve an entity by ID | Champ needs to handle accepting an entity id and the timestamp to run a historical query on the graph db retrieving the asserted state. Default not sending back metaproperties, if metaproperties are needed a parameter would need to be sent to champ. A subgraph strategy will be used in champ to filter on the relative timestamp provided. meta=[true/false] if true, payload retrieved will hold the metaproperties at t-k |
GET | entity/<ID>/lifespan | None | Retrieve all the timestamps for create, update, delete operations against this entity | Champ would be called to retrieve the lifespan on the entity with meta=true and t-k=lifespan to retrieve all the metaproperties on the entity |
PUT | relationship/<ID> | actor=name of the system making the assertion changes-only=[true|false] if true, gallifrey will actually determine what has changed between the PUT payload and the most recent set of assertions for the relationship. If false, the entire PUT body will be considered as a new set of assertions whether something has changed or not. create=[true|false] if true, Gallifrey assumes that this is a create request, if false it assumes it is an update t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Asserts that a relationship is to be created or updated (depending on the query parameters that are passed in). This API appends new assertions against the specified relationship. Generated in Gallifrey t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey (why are we generating this here? if there is a maintenance issue our timings would be out of sync with when these took place in the real-time db) | create = false (changes-only true (execute diff)/false(assume everything changed)) When an assertion is being made without a network timestamp gallifrey will call champ requesting the relationship with it's most current metaproperties. Gallifrey would then adjust the metaproperties (of the updated properties) and would send the payload back to champ with a new current state and an updated previous state's metaproperties. Champ would override it's current metaproperty (for the updated/deleted properties) with the old and current metaproperties sent from Gallifrey. For added properties, they would be added directly with the metaproperties sent from Gallifrey. When an assertion is being made with a network timestamp gallifrey will call champ requesting the relationship with all of its metaproperties. Gallifrey would then insert the new assertion where appropriate (and adjust neighboring metaproperties) and send the modified payload back to champ for a replace. create = true POST - this would be a new create and Gallifrey would pass the metaproperties on each of it's property values and on the relationship itself |
PUT | entity/<ID> | actor=name of the system making the assertion changes-only=[true|false] if true, gallifrey will actually determine what has changed between the PUT payload and the most recent set of assertions for the entity. If false, the entire PUT body will be considered as a new set of assertions whether something has changed or not. create=[true|false] if true, Gallifrey assumes that this is a create request, if false it assumes it is an update t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Asserts that an entity is to be created or updated (depending on the query parameters that are passed in). This API appends new assertions against the specified entity. Generated in Gallifrey t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey | create = false (changes-only true (execute diff)/false(assume everything changed)) When an assertion is being made without a network timestamp gallifrey will call champ requesting the entity with it's most current metaproperties. Gallifrey would then adjust the metaproperties (of the updated properties) and would send the payload back to champ with a new current state and an updated previous state's metaproperties. Champ would override it's current metaproperty (for the updated/deleted properties) with the old and current metaproperties sent from Gallifrey. For added properties, they would be added directly with the metaproperties sent from Gallifrey. When an assertion is being made with a network timestamp gallifrey will call champ requesting the entity with all of its metaproperties. Gallifrey would then insert the new assertion where appropriate (and adjust neighboring metaproperties) and send the modified payload back to champ for a replace. create = true POST - this would be a new create and Gallifrey would pass the metaproperties on each of it's property values and on the entity itself |
DELETE | relationship/<ID> | actor=name of the system making the assertion t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Asserts that a relationship has been deleted. Generated in Gallifrey t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey | Gallifrey would request the latest relationship from champ and would set all of it's properties dbEndTimes to t-k along with the dbEndTime on the relationship itself. |
DELETE | entity/<ID> | actor=name of the system making the assertion t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Asserts that an entity has been deleted. Generated in Gallifrey t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey | Gallifrey would request the latest entity from champ and would set all of it's properties dbEndTimes to t-k along with the dbEndTime on the entity itself. |
Chameleon API Spec
Type | URI | Query Params | Description |
GET | relationship/<ID> | t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Retrieve a relationship by ID |
GET | entity/<ID> | t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Retrieve an entity by ID |
Between Chameleon->Gallifrey, the following calls are made:
PUT entity/<ID>?actor=aai&create=true
PUT relationship/<ID>?actor=aai&create=true
PUT entity/<ID>?actor=aai&changes-only=true
PUT relationship/<ID>?actor=aai&changes-only =true
DELETE entity/<ID>?actor=aai
DELETE relationship/<ID>?actor=aai
GET entity/<ID>?t-k=<timestamp>
GET relationship/<ID>? t-k=<timestamp>
Need Chameleon updated with this:
GET entity/<ID>?t-t=<timestamp>
GET relationship/<ID>? t-t=<timestamp>
Champ API Spec
Every entity/relationship/property must have a metaproperty or set of meta properties, this will be enforced in champ.
POST an entity- For History provide meta-properties in the payload
Method=POST
https://<host>:9522/services/champ-service/v1/objects
Retrieve an entity across its lifespan
Method= GET, As default,meta=false
https://<host>:9522/services/champ-service/v1/objects/<key>
https://<host>:9522/services/champ-service/v1/objects/<key>&meta=true
(called when updating with network time)
https://<host>:9522/services/champ-service/v1/objects/<key>&meta=latest
(called when updating with only db time)
Retrieve an object for a given database timestamp
Method= GET, As default,meta=false
https://<host>:9522/services/champ-service/v1/objects/<key>?t-k=t1
https://<host>:9522/services/champ-service/v1/objects/<key>?t-k=t1&meta=true
Retrieve an object for a given network timestamp
Method= GET, As default,meta=false
https://<host>:9522/services/champ-service/v1/objects/<key>?t-t=t1
https://<host>:9522/services/champ-service/v1/objects/<key>?t-t=t1&meta=true
UPDATE an object
Method=PUT - default=[patch=false] which means full replace
https://<host>:9522/services/champ-service/v1/objects/<key>
https://<host>:9522/services/champ-service/v1/objects/<key>?patch=true
DELETE an object - DELETE
Method=PUT - default=[patch=false] which means full replace
https://<host>:9522/services/champ-service/v1/objects/<key>
https://<host>:9522/services/champ-service/v1/objects/<key>?patch=true
POST a relationship - For History provide meta-properties in the payload
Method: POST
URL: https://<host>:9522/services/champ-service/v1/relationships
Retrieve a relationship across its lifespan
Method= GET, As default,meta=false
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>&meta=true
Retrieve a relationship for a given database timestamp
Method= GET, As default,meta=false
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-k=t1
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-k=t1&meta=true
Retrieve a relationship for a given network timestamp
Method= GET, As default,meta=false
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-t=t1
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-t=t1&meta=true
UPDATE an object
Method=PUT - default=[patch=false] which means full replace
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?patch=true
DELETE an object - DELETE
Method=PUT - default=[patch=false] which means full replace
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>
Historical Tracking Metaproperties
dbStartTime - when an entity/property value/relationship was added to db
dbEndTime - when and entity or relationship was deleted from the db or a property's value was asserted to another state
ntStartTime - asserted by the client as to when the change took place in the network
ntEndTime - set when an assertion provided by the client make's the current state no longer true
startSOT - the source of truth that made the assertion
endSOT- the source of truth that made an assertion to make the current state no longer true
Schema
Separate db edge rules file: with all relationships many to many except parent child which could be one to many
Separate schema file: no properties on vertices are unique except for aai-uuid
Open question: what happens in the event that db edge rules change etc. (migrations)
GUI Mocks
New integrated functionality (updates for history)
Existing GUI
Topology Current view
Updates include a history button, which pops up a date/time picker to replace the topology view with that historical topology. In the bottom right, it will designate the time period, and if not current the ability to revert back to current.
Compare pops up different date/time picker for a secondary topology. to compare against.
Compare View
