GraphAdmin

GraphAdmin microservice is based on Spring boot and deployed using docker which is primarily responsible for handling admin level tasks such as taking snapshot of the disk and running gremlin queries.

Repository is found here - https://gerrit.onap.org/r/#/admin/projects/aai/graphadmin

The jenkins job found here - https://jenkins.onap.org/view/aai-graphadmin/

DB tools from resources and aai core are moved the new GraphAdmin MS which includes :

•     DataGrooming

•     DataSnapshot

•     DupeTool

•     ForceDeleteTool

•     GraphMLTokens

•     SchemaGenerator

•     SchemaMod

•     SchemamodInternal

•     UpdateEdgeTags

Migration code from the resources api is moved to the new GraphAdmin MS.

Gremlin queries via the "query" endpoint is moved to the graphadmin mS via a new endpoint "dbquery"

Interfaces with Hbase/Cassandra via TinkerPop APIs using JanusGraph
CreateDBSchema

Name - createDBSchema.sh

Purpose - This tool creates the graph schema for the database

Logging config - graphAdmin logback.xml

Logs - cd /opt/app/aai-graphadmin/logs/createDBSchema/error.log ,metrics.log, debug.log.

Output - on console

Usage:

For running CreateDbSchema, first take a snaphot and check the log, then run the script and again check the logs.

Data Grooming

Name - dataGrooming.sh

Purpose - This tool is built to look for data problems that need to be cleaned up for the database to operate properly.   This tool is run automatically from a cron once every few hours.   If run with the "-autoFix" option, it will automatically run once, and then, if it finds some delete candidates, it will sleep for 7 minutes and run again.  It will automatically delete the deleteCandidates found if they are found again on the second run.  Note, we use the "-autoFix" option when running from the cron.  Any time that delete-candidates are found, an exception is thrown and logged.  The error thrown is: "AAI_6123::Bad Data found by DataGrooming Tool - Investigate".   Whoever is monitoring AAI logs will need to alert Tier-4 if delete candidates are found and not cleaned up by the cron.  The list of candidate deletes (nodes or edges or both) in the file can be edited and the script can then be run with this file as an input to the script.   If it re-verifies the problem-candidates, it will delete them.

Logging config - graphAdmin logback.xml

Logs - dataGrooming/error.log and dataGrooming/debug.log

Output - (in the /opt/app/aai-graphadmin/logs/data/dataGrooming  directory): dataGrooming.FULL.YYYYMMDDHHSS.out  or  dataGrooming.PARTIAL.YYYMMDDHHSS.out

Usage:

Previous output file

For example to generate a list of delete candidates:  

To actually delete an existing generated candidates list of data:

dataGrooming.20180924.out is created from the previous command to generate list of delete candidates inside the container /opt/app/aai-graphadmin/logs/data/dataGrooming/ folder

There are quite a few command line parameters that let you control what happens with this tool.  They are all optional. They are documented as comments in the script itself.  Here is a list of them; for details on how to use them, see the comments at the top of the script itself, or the T-space write up of the tool.   "-f oldfileName", "-autoFix", "-sleepMinutes nn", "-edgesOnly", "-dontFixOrphans", "-maxFix", "skipHostCheck", "-singleCommits", "-dupeCheckOff", "-dupeFixOn", "-timeWindowMinutes", "-skipEdgeCheck", "-ghoste2CheckOff", "-ghost2FixOn".

 NOTE - if the optional timeWindowMinutes parameter is used, the output file will have the text "PARTIAL" as part of the name, if this parameter is not used, then the output file will have the text "FULL" in it.                  

Ie. output file names could look like:     dataGrooming.FULL.201801171710.out  or  dataGrooming.PARTIAL.201801152110.out


Sample error from the dataGrooming.log that tells you you'd need to investigate a data problem (and what file needs to be looked into):

2015-09-10T14:54:51-04:00|c6533401-0a33-4ea7-a039-d2fd2dfd6151||||AAIDATAGROOM:AAI:dataGrooming||INFO|||mtinjvmsdn30|135.25.186.202|DataGrooming|1762077|co=aaidbgen:ss=y:ec=0: >  Look at : [oam-network] ...:emsg=See file: [/opt/aai/logs/dataGrooming.201509101825.out] and investigate delete candidates.:ERROR=ErrorObject [errorCode=6123, errorText=Bad Data found by DataGrooming Tool - Investigate, restErrorCode=3000, httpResponseCode=Internal Server Error, severity=ERROR, disposition=5, category=4]|


Sample of what part of the output file looks like is below.  Note - besides the candidate list, there are other sections of data which should help us understand the delete-candidates:

 ============ Summary ==============

Deleted this many delete candidates from previous run =  0

Total number of nodes looked at =  2617

Ghost Nodes identified = 5

Orphan Nodes identified =  0

Missing Dependent Edge (but not orphaned) node count = 0

Duplicates count =  0

MisMatching Label/aai-node-type count =  0

 ------------- Delete Candidates ---------

DeleteCandidate: Phantom Vid = [491749424]

DeleteCandidate: Phantom Vid = [147584]

DeleteCandidate: Phantom Vid = [98672]

DeleteCandidate: Phantom Vid = [164200]

DeleteCandidate: Phantom Vid = [491798944]

Sample of what part of the output file looks like which includes detail of a pair of duplicate nodes. Note that it gives the details of the two nodes including whatever IN/OUT edges it finds.  For this example, the tool cannot determine which of the two nodes (note - some groups can have more than two nodes) is the best one to delete.  You can tell because of these two lines:

           For this group of duplicates, could not tell which one to keep.

           >>> This group needs to be taken care of with a manual/forced-delete.


 It is up to Tier support to look at the duplicates in the list and make a determination.  See notes for the "DupeTool" for pointers on how to determine which duplicate node to delete. Note also - Typically, the "forceDelete" tool must be used to delete nodes like this since they cannot be reached via the normal REST API - so the "normal" delete tool cannot be used on them.


 ------------- Duplicates:

 --- Duplicate Group # 1 Detail -----------

    >> Duplicate Group # 1  Node # 0 ----

 AAINodeType/VtxID for this Node = [tenant/34656336]

 Property Detail:

Prop: [last-mod-source-of-truth], val = [Robot]

Prop: [aai-node-type], val = [tenant]

Prop: [aai-created-ts], val = [1509635306703]

Prop: [aai-last-mod-ts], val = [1509635306705]

Prop: [source-of-truth], val = [Robot]

Prop: [aai-uri], val = [/cloud-infrastructure/cloud-regions/cloud-region/cloudowner-AAI-vm230w/cloudregion-id-AAI-vm230w/tenants/tenant/tenant-10124-vm230w]

Prop: [tenant-id], val = [tenant-10124-vm230w]

Prop: [tenant-name], val = [tenant-namevm230w]

Prop: [resource-version], val = [1509635306705]

Found an IN edge (has) to this vertex from a [cloud-region] node with VtxId = 5984488

Found an OUT edge (owns) from this vertex to a [vserver] node with VtxId = 23040080

    >> Duplicate Group # 1  Node # 1 ----

 AAINodeType/VtxID for this Node = [tenant/122888264]

 Property Detail:

Prop: [last-mod-source-of-truth], val = [Robot]

Prop: [aai-node-type], val = [tenant]

Prop: [aai-created-ts], val = [1511892771574]

Prop: [aai-last-mod-ts], val = [1511892771579]

Prop: [source-of-truth], val = [Robot]

Prop: [aai-uri], val = [/cloud-infrastructure/cloud-regions/cloud-region/testowner1/testregion1/tenants/tenant/testtenant1]

Prop: [tenant-id], val = [testtenant1]

Prop: [tenant-name], val = [testtenantname]

Prop: [resource-version], val = [1511892771579]

Found an IN edge (org.onap.relationships.inventory.BelongsTo) to this vertex from a [vserver] node with VtxId = 122892360

Found an OUT edge (org.onap.relationships.inventory.BelongsTo) from this vertex to a [cloud-region] node with VtxId = 5984488


 For this group of duplicates, could not tell which one to keep.

 >>> This group needs to be taken care of with a manual/forced-delete.
Data Snapshot

Name - dataSnapShot.sh

Purpose - This tool creates the graph schema for the database

Logging config - graphAdmin logback.xml

Logs - opt/app/aai-graphadmin/logs/dataSnapshot/error.log and /opt/app/aai-graphadmin/logs/dataSnapshot/debug.log

Output - on console

Usage:

    This will take a data snapshot and create graphson files using JanusGraph APIs. Please note there is a different snapshot taken at the storage backend using the tools that are backend specific.

    If you want to run the tool while he GraphAdmin application is up, use the execTool.sh to run the tool as below.

     /opt/app/aai-graphadmin/execTool.sh dataSnapshot.sh

    ${PROJECT_HOME}/logs/misc/run_dataSnapshot.log.$(date +\%Y-\%m-\%d) 2>&1


ls -ltr /opt/app/aai-graphadmin/logs/data/dataSnapshots/

Gremlin API

The following API should only be used by admins and should not be exposed to clients as it exposes the appilcation

PUT /aai/v14/dbquery?format=console

Returns the number of vertexes in the graph

PUT /aai/v14/dbquery?format=console

Returns all the pserver vertexes with their ids since the format chosen is console