AAF Initialization Data

Owned by Jonathan Gathman
Last updated: Mar 04, 2020
7 min read

As noted before, ONAP runs a AAF Test instance in WindRiver.  This allows ONAP groups to create and test entities within their own Namespace while running on their own Development boxes.

Data Scope:

The data in this Environment should be treated as "initialization" data, as it is intended to be the essential set of Info for starting up an out-of-the-box ONAP Environment with AAF Identities and coordinated Roles, Permissions, etc.

The issue is, of course, that AAF's Permissions, Roles and Identities are intended to cross many boundaries.  It is, however, not possible for any central team (at least without resources, funding, and large knowledge scope) to maintain full permission information about Apps and people.  This is why AAF needs Data setup FOR the Organization in mind to be of any use.

ONAP's data is about the individual apps, which can change, but not much about the reality of who is actually running at the beginning.  For example, NO ONAP developers or current ONAP leadership, developers or other are included, for instance, in the Identity Data.  To emphasize that this is Data that is to be modified, it is named "Sample".

Sample Data:

All the Sample Data is found in the core AAF repository, called "authz".  To obtain this Repo locally, see ONAP Git documentation, or simply, just pull locally with

Authz Repo
git clone "https://gerrit.onap.org/r/aaf/authz"

NOTE: ALL DIRECTORIES REFERENCED IN THIS DOC WILL START FROM THE AUTHZ ROOT.

Identities:

Identities in any Organization will be hierarchical, and they are their own.  AAF was designed in ways to allow companies to utilize their own way to dealing with Identities.  Please start here (Philosophy of Identity).

ONAP Initial identities fictitiously a euphemistic hierarchy.  Obviously, a real company would want to replace this with their own.

identity.dat

This file is store in the Repository (starting from authz, see above clone)



Location of identities.dat
auth/sample/data/sample.identities.dat



When deployed, you will find the file "identities.dat" in docker directories "/opt/app/osaaf/data", and a configured Docker Volume of "config".  (The Docker Volume keeps this file up persistently, whether Apps exist or not).

Before each AAF Docker component is launched, it is preceded by "aaf-config" init-container. The aaf-config init-container, among other things:

checks to see if the "identities.dat" file exists.

If not, it copies it from "/opt/app/aaf_config/data/sample.identities.dat" in the aaf-config Docker Image, and places it at /opt/app/osaaf/data/identities.dat, which is on the Persistent Docker Volume "config", as noted.

This ensures that identities.dat only starts new when it doesn't exist, and DOESN'T overwrite work that Companies may be doing.  Being in the "aaf-config" image avoids size implications in the other components.

* This "identities.dat" file is utilized exclusively by the "DefaultOrganization".  Companies are welcome and encouraged, if they wish, to create their own "Plugin that implements the 'Organization' interface", and connects to their own data how they please.  

** If this is too much work, they are free to update the "identities.dat" file from their own Organization information on a timely basis.

*** Companies should note that this mechanism was written for an ONAP member company with a nightly feed that included more than 1.3 million records.  It does so very efficiently, without synchronizing data.

Cassandra Data:

Cassandra data, that is interrelated to the above "Identities" and each other also need enough data to be functional with all the ONAP apps required.

In the repo, these files (ending in ".dat") are found at

Cassandra Initial Data
auth/sample/cass_data

Included here are the ESSENTIAL data files in Cassandra (other Cassandra Data files are fine to be empty at the beginning, i.e. the history data).  

Similarly to the identity file, the Cassandra Docker keeps the INIT files in a docker image, but as opposed to AAF Components, it simply keeps them in the aaf-cass docker image.

IF the container, when starting, finds that cassandra tables are not defined, THEN the container will initialize cassandra from "cql" files located in the image at "/opt/app/aaf/cass_init " (copied from the repo directory auth/auth-cass/cass_init).  The container executes "keyspace.cql", "init.cql", "temp_identity.cql" and "onap.cql" to setup the initial tables.

IF the image finds that the cassandra tables are uninitialized THEN the image will load the data from the docker image directory "/opt/app/aaf/cass_init/".  It does NOT initialize if the data exists, so as not to overwrite running systems.

Please see repo "auth/auth-cass/cass_init/cmd.sh" to see how this is accomplished.

Relationship to various ONAP Test environments:

ONAP Testing typically start off "from scratch", which means that there is no existing implementation or configuration.  In some cases, every day. This validates that all ONAP could start, as a System, "from scratch", and gives a clean environment for various tests.  

AAF utilizes the above Configurations mechanisms to ensure that each brand-new environment for ONAP starts off with existing Data.  This includes NEW ONAP components, documented in "sample.identities.dat" and the Permissions and Roles (etc) that the  Apps create within the AAF Test system.

As described above, all the Identities and Data are pushed into Containers (init or otherwise), which can start up and configure when data does not exist, or avoid overwriting when they do exist.

Processes to copy data from AAF Test systems in WindRiver to be ready in ONAP Docker Images, when required for Testing, other.

The expectation is that this process 

  • Is done from the development machine of ONAP AAF PTL (or other responsible party within ONAP)

  • This person already works with ONAP "aaf" project and has "authz" git repo setup and ready.

  • This person has VPN Access to WindRiver and a Key to the AAF WindRiver Project for K8s environment, and specifically the "aaf-test" instance running there (should never be removed)

GOALS:

  1. Cause a Remote Backup from the Containers

  2. Bring that Remote Backup locally, as added safeguard (Note: at times, people have removed the WindRiver AAF Test VM accidentally.  Extra backups made reconstruction reasonably simple)

  3. Use filtered data from backup to populate Initialization Data within AAF Containers, ready for check-in to Repo

  4. Use Backup Data to update Identity and Initialized Cassandra Data files into local "authz" file area

  5. Perform normal Checkin Procedures for authz

Steps 1-2 are accomplished with One local Shell Script, combined with Shell Scripts deployed on remote Directories and in Containers (see script to trace each one)

IT SHOULD BE NOTED, that backups can be run often, but that doesn't mean you HAVE to push the data to the REPO daily.  That is why there are two scripts.  Currently, we wait until ONAP Project leads ask for it to be pushed, or during correct Milestone.

This is done via a shell script.  Documented here is a template which the identified person can modify for their particular keys, but requires:

A) VPN Tunnel access to WindRiver, turned on

B) Private Key setup for WindRiver VM.  This is typically stored locally in "pem" format.  We'll call it "aaf-onap.pem" for the following template.

C) Your "/etc/hosts" directory has entry "10.12.5.145 aaf-onap-test.osaaf.org" (or whatever is the current IP with ONAP aaf-test VM)

D) This is being run from a subdirectory being worked by said user.  This subdirectory should NOT be within a Repo Boundary.  Here is a recommendation for this structure, assuming Unix 

/home/me

fictitious home directory

/home/me/open

Directory where I keep all Open Source work, including ONAP

/home/me/open/authz

Created by 'git clone "https://gerrit.onap.org/r/aaf/authz"'

/home/me/open/...

Don't forget all the other AAF and ONAP projects, like 'cadi', 'doc', 'oom', etc.

/home/me/open/backup

The directory we'll work with in this effort... 



WARNING:  the "DAT Date" mechanism is simple, and doesn't adjust for Global Time zones.  Just choose a time (like morning?) when WindRiver Day is the same as your location's Day

pull.sh



Results:  In the directory, after running, you should find the following (This sample is run on March 4, 2020.  OBVIOUSLY, dates should vary)

  • a "tar gz" backup for today of Test Data

  • the latest "identities.dat" file

  • a subdirectory called "dats", which has the latest data file un-tarred, ready for updating repo.



Run pull.sh



Steps 3-4:

WHEN it is appropriate to update data in the Repo (when asked by another ONAP Project PTL, or when Milestones arrive, needing a new Docker Image), the process of moving the "Golden Source" of aaf-test to repo is as follows.

  • go to "authz"... "git status" should be updated, so you don't mix up commits.  

  • Note: This script uses "dos2unix", which is typically available to Unix.  This keeps DOSisms out, and allows for a cleaner check in.

  • Either Finish checking in, stash, whatever..  Just try to make commits of new aaf-test data CLEAN.

  • cd - (go back to "backup" directory)

toSample.sh



run toSample.sh



Step 5:

You are ready to do normal AAF authz code check in procedures with "git commit" (and --amend), and git review.



SUMMARY:

This process

  • Created a Backup of aaf-test data remotely

  • Copied Backup locally

  • Filtered data that shouldn't be in Initialization data

  • Prepared this data in "authz" repo directory, ready for check in



Obviously, to be part of the containers, you have to follow ONAP Release procedures, which are not covered here.