Page Comparison

...

Previous output file

For example to generate a list of delete candidates:  

 docker-compose -f /opt/test-config/docker-compose-app.yaml aai-graphadmin dataGrooming.sh

To actually delete an existing generated candidates list of data:

 docker-compose -f /opt/test-config/docker-compose-app.yaml aai-graphadmin dataGrooming.sh dataGrooming.sh -f dataGrooming.20180924.out

dataGrooming.20180924.out is created from the previous command to generate list of delete candidates inside the container /opt/app/aai-graphadmin/logs/data/dataGrooming/ folder

There are quite a few command line parameters that let you control what happens with this tool.  They are all optional. They are documented as comments in the script itself.  Here is a list of them; for details on how to use them, see the comments at the top of the script itself, or the T-space write up of the tool.   "-f oldfileName", "-autoFix", "-sleepMinutes nn", "-edgesOnly", "-dontFixOrphans", "-maxFix", "skipHostCheck", "-singleCommits", "-dupeCheckOff", "-dupeFixOn", "-timeWindowMinutes", "-skipEdgeCheck", "-ghoste2CheckOff", "-ghost2FixOn".

 NOTE - if the optional timeWindowMinutes parameter is used, the output file will have the text "PARTIAL" as part of the name, if this parameter is not used, then the output file will have the text "FULL" in it.                  

Ie. output file names could look like:     dataGrooming.FULL.201801171710.out  or  dataGrooming.PARTIAL.201801152110.out


Sample error from the dataGrooming.log that tells you you'd need to investigate a data problem (and what file needs to be looked into):

2015-09-10T14:54:51-04:00|c6533401-0a33-4ea7-a039-d2fd2dfd6151||||AAIDATAGROOM:AAI:dataGrooming||INFO|||mtinjvmsdn30|135.25.186.202|DataGrooming|1762077|co=aaidbgen:ss=y:ec=0: >  Look at : [oam-network] ...:emsg=See file: [/opt/aai/logs/dataGrooming.201509101825.out] and investigate delete candidates.:ERROR=ErrorObject [errorCode=6123, errorText=Bad Data found by DataGrooming Tool - Investigate, restErrorCode=3000, httpResponseCode=Internal Server Error, severity=ERROR, disposition=5, category=4]|


Sample of what part of the output file looks like is below.  Note - besides the candidate list, there are other sections of data which should help us understand the delete-candidates:

 ============ Summary ==============

Deleted this many delete candidates from previous run =  0

Total number of nodes looked at =  2617

Ghost Nodes identified = 5

Orphan Nodes identified =  0

Missing Dependent Edge (but not orphaned) node count = 0

Duplicates count =  0

MisMatching Label/aai-node-type count =  0

 ------------- Delete Candidates ---------

DeleteCandidate: Phantom Vid = [491749424]

DeleteCandidate: Phantom Vid = [147584]

DeleteCandidate: Phantom Vid = [98672]

DeleteCandidate: Phantom Vid = [164200]

DeleteCandidate: Phantom Vid = [491798944]

Sample of what part of the output file looks like which includes detail of a pair of duplicate nodes. Note that it gives the details of the two nodes including whatever IN/OUT edges it finds.  For this example, the tool cannot determine which of the two nodes (note - some groups can have more than two nodes) is the best one to delete.  You can tell because of these two lines:

           For this group of duplicates, could not tell which one to keep.

           >>> This group needs to be taken care of with a manual/forced-delete.


 It is up to Tier support to look at the duplicates in the list and make a determination.  See notes for the "DupeTool" for pointers on how to determine which duplicate node to delete. Note also - Typically, the "forceDelete" tool must be used to delete nodes like this since they cannot be reached via the normal REST API - so the "normal" delete tool cannot be used on them.


 ------------- Duplicates:

 --- Duplicate Group # 1 Detail -----------

    >> Duplicate Group # 1  Node # 0 ----

 AAINodeType/VtxID for this Node = [tenant/34656336]

 Property Detail:

Prop: [last-mod-source-of-truth], val = [Robot]

Prop: [aai-node-type], val = [tenant]

Prop: [aai-created-ts], val = [1509635306703]

Prop: [aai-last-mod-ts], val = [1509635306705]

Prop: [source-of-truth], val = [Robot]

Prop: [aai-uri], val = [/cloud-infrastructure/cloud-regions/cloud-region/cloudowner-AAI-vm230w/cloudregion-id-AAI-vm230w/tenants/tenant/tenant-10124-vm230w]

Prop: [tenant-id], val = [tenant-10124-vm230w]

Prop: [tenant-name], val = [tenant-namevm230w]

Prop: [resource-version], val = [1509635306705]

Found an IN edge (has) to this vertex from a [cloud-region] node with VtxId = 5984488

Found an OUT edge (owns) from this vertex to a [vserver] node with VtxId = 23040080

    >> Duplicate Group # 1  Node # 1 ----

 AAINodeType/VtxID for this Node = [tenant/122888264]

 Property Detail:

Prop: [last-mod-source-of-truth], val = [Robot]

Prop: [aai-node-type], val = [tenant]

Prop: [aai-created-ts], val = [1511892771574]

Prop: [aai-last-mod-ts], val = [1511892771579]

Prop: [source-of-truth], val = [Robot]

Prop: [aai-uri], val = [/cloud-infrastructure/cloud-regions/cloud-region/testowner1/testregion1/tenants/tenant/testtenant1]

Prop: [tenant-id], val = [testtenant1]

Prop: [tenant-name], val = [testtenantname]

Prop: [resource-version], val = [1511892771579]

Found an IN edge (org.onap.relationships.inventory.BelongsTo) to this vertex from a [vserver] node with VtxId = 122892360

Found an OUT edge (org.onap.relationships.inventory.BelongsTo) from this vertex to a [cloud-region] node with VtxId = 5984488


 For this group of duplicates, could not tell which one to keep.

 >>> This group needs to be taken care of with a manual/forced-delete.
Data Snapshot

Name - dataSnapShot.sh

Purpose - This tool creates the graph schema for the database

Logging config - graphAdmin logback.xml

Logs - opt/app/aai-graphadmin/logs/dataSnapshot/error.log and /opt/app/aai-graphadmin/logs/dataSnapshot/debug.log

Output - on console

Usage:

    This will take a data snapshot and create graphson files using JanusGraph APIs. Please note there is a different snapshot taken at the storage backend using the tools that are backend specific.

    If you want to run the tool while he GraphAdmin application is up, use the execTool.sh to run the tool as below.

     /opt/app/aai-graphadmin/execTool.sh dataSnapshot.sh

 docker-compose -f /opt/test-config/docker-compose-app.yaml aai-graphadmin dataSnapshot.sh

    ${PROJECT_HOME}/logs/misc/run_dataSnapshot.log.$(date +\%Y-\%m-\%d) 2>&1


ls -ltr /opt/app/aai-graphadmin/logs/data/dataSnapshots/