AAF Initialization Data

As noted before, ONAP runs a AAF Test instance in WindRiver.  This allows ONAP groups to create and test entities within their own Namespace while running on their own Development boxes.

Data Scope:

The data in this Environment should be treated as "initialization" data, as it is intended to be the essential set of Info for starting up an out-of-the-box ONAP Environment with AAF Identities and coordinated Roles, Permissions, etc.

The issue is, of course, that AAF's Permissions, Roles and Identities are intended to cross many boundaries.  It is, however, not possible for any central team (at least without resources, funding, and large knowledge scope) to maintain full permission information about Apps and people.  This is why AAF needs Data setup FOR the Organization in mind to be of any use.

ONAP's data is about the individual apps, which can change, but not much about the reality of who is actually running at the beginning.  For example, NO ONAP developers or current ONAP leadership, developers or other are included, for instance, in the Identity Data.  To emphasize that this is Data that is to be modified, it is named "Sample".

Sample Data:

All the Sample Data is found in the core AAF repository, called "authz".  To obtain this Repo locally, see ONAP Git documentation, or simply, just pull locally with

git clone "https://gerrit.onap.org/r/aaf/authz"

NOTE: ALL DIRECTORIES REFERENCED IN THIS DOC WILL START FROM THE AUTHZ ROOT.

Identities:

Identities in any Organization will be hierarchical, and they are their own.  AAF was designed in ways to allow companies to utilize their own way to dealing with Identities.  Please start here (Philosophy of Identity).

ONAP Initial identities fictitiously a euphemistic hierarchy.  Obviously, a real company would want to replace this with their own.

identity.dat

This file is store in the Repository (starting from authz, see above clone)

auth/sample/data/sample.identities.dat

When deployed, you will find the file "identities.dat" in docker directories "/opt/app/osaaf/data", and a configured Docker Volume of "config".  (The Docker Volume keeps this file up persistently, whether Apps exist or not).

Before each AAF Docker component is launched, it is preceded by "aaf-config" init-container. The aaf-config init-container, among other things:

checks to see if the "identities.dat" file exists.

If not, it copies it from "/opt/app/aaf_config/data/sample.identities.dat" in the aaf-config Docker Image, and places it at /opt/app/osaaf/data/identities.dat, which is on the Persistent Docker Volume "config", as noted.

This ensures that identities.dat only starts new when it doesn't exist, and DOESN'T overwrite work that Companies may be doing.  Being in the "aaf-config" image avoids size implications in the other components.

* This "identities.dat" file is utilized exclusively by the "DefaultOrganization".  Companies are welcome and encouraged, if they wish, to create their own "Plugin that implements the 'Organization' interface", and connects to their own data how they please.  

** If this is too much work, they are free to update the "identities.dat" file from their own Organization information on a timely basis.

*** Companies should note that this mechanism was written for an ONAP member company with a nightly feed that included more than 1.3 million records.  It does so very efficiently, without synchronizing data.

Cassandra Data:

Cassandra data, that is interrelated to the above "Identities" and each other also need enough data to be functional with all the ONAP apps required.

In the repo, these files (ending in ".dat") are found at

auth/sample/cass_data

Included here are the ESSENTIAL data files in Cassandra (other Cassandra Data files are fine to be empty at the beginning, i.e. the history data).  

Similarly to the identity file, the Cassandra Docker keeps the INIT files in a docker image, but as opposed to AAF Components, it simply keeps them in the aaf-cass docker image.

IF the container, when starting, finds that cassandra tables are not defined, THEN the container will initialize cassandra from "cql" files located in the image at "/opt/app/aaf/cass_init " (copied from the repo directory auth/auth-cass/cass_init).  The container executes "keyspace.cql", "init.cql", "temp_identity.cql" and "onap.cql" to setup the initial tables.

IF the image finds that the cassandra tables are uninitialized THEN the image will load the data from the docker image directory "/opt/app/aaf/cass_init/".  It does NOT initialize if the data exists, so as not to overwrite running systems.

Please see repo "auth/auth-cass/cass_init/cmd.sh" to see how this is accomplished.

Relationship to various ONAP Test environments:

ONAP Testing typically start off "from scratch", which means that there is no existing implementation or configuration.  In some cases, every day. This validates that all ONAP could start, as a System, "from scratch", and gives a clean environment for various tests.  

AAF utilizes the above Configurations mechanisms to ensure that each brand-new environment for ONAP starts off with existing Data.  This includes NEW ONAP components, documented in "sample.identities.dat" and the Permissions and Roles (etc) that the  Apps create within the AAF Test system.

As described above, all the Identities and Data are pushed into Containers (init or otherwise), which can start up and configure when data does not exist, or avoid overwriting when they do exist.

Processes to copy data from AAF Test systems in WindRiver to be ready in ONAP Docker Images, when required for Testing, other.

The expectation is that this process 

• Is done from the development machine of ONAP AAF PTL (or other responsible party within ONAP)
• This person already works with ONAP "aaf" project and has "authz" git repo setup and ready.
• This person has VPN Access to WindRiver and a Key to the AAF WindRiver Project for K8s environment, and specifically the "aaf-test" instance running there (should never be removed)

GOALS:

1. Cause a Remote Backup
2. Bring that Remote Backup locally, as added safeguard (Note: at times, people have removed the WindRiver AAF Test VM accidentally.  Extra backups made reconstruction reasonably simple)
3. Use Backup Data to update Identity and Initialized
4. Cause a Remote Backup from the Containers
5. Use filtered data from backup to populate Initialization Data within AAF Containers, ready for check-in to Repo

Steps 1-4 are accomplished with One local Shell Script, combined with Shell Scripts deployed on remote Directories and in Containers (see script to trace each one)

This is done via a shell script.  Documented here is a template which the identified person can modify for their particular keys, but requires:

A) VPN Tunnel access to WindRiver, turned on

B) Private Key setup for WindRiver VM.  This is typically stored locally in "pem" format.  We'll call it "aaf-onap.pem" for the following template.

C) Your "/etc/hosts" directory has entry "10.12.5.145 aaf-onap-test.osaaf.org" (or whatever is the current IP with ONAP aaf-test VM)

D) This is being run from a subdirectory being worked by said user.  This subdirectory should NOT be within a Repo Boundary.  Here is a recommendation for this structure, assuming Unix 

WARNING:  the "DAT Date" mechanism is simple, and doesn't adjust for Global Time zones.  Just choose a time (like morning?) when WindRiver Day is the same as your location's Day

#!/etc/bash

# NOTE: YOU much change this to point to YOUR personal locate for PEM based ONAP VM Key
PEM=/home/me/<directory for ONAP keys>/aaf-onap.pem

ssh -i $PEM ubuntu@aaf-onap-test.osaaf.org -t 'cd ~/authz/auth/auth-cass/docker;bash backup.sh'
ssh -i $PEM ubuntu@aaf-onap-test.osaaf.org -t 'cd ~/authz/auth/sample/data; bash pull.sh'

#DAT_TODAY="dat20190222.gz"
DAT_TODAY="dat$(date +%Y%m%d).gz"

scp -i $PEM ubuntu@aaf-onap-test.osaaf.org:~/authz/auth/auth-cass/docker/$DAT_TODAY .
scp -i $PEM ubuntu@aaf-onap-test.osaaf.org:~/authz/auth/sample/data/identities.dat .
if [ -e dats ]; then
  rm dats/*
fi

tar -xvf $DAT_TODAY

Results:  In the directory, you should find the following (This sample is run on March 4, 2020.  OBVIOUSLY, dates should vary)

backup me$ bash pull.sh
Using 7 child processes

Starting copy of authz.locate with columns [name, hostname, port, latitude, longitude, major, minor, patch, pkg, port_key, protocol, subprotocol].
Processed: 16 rows; Rate:      92 rows/s; Avg. rate:      92 rows/s
16 rows exported to 1 files in 0.183 seconds.
Using 7 child processes
...
x dats/run_lock.dat
x dats/user_role.dat
x dats/x509.dat

backup me$ ls -altr
...
-rw-r--r--   1 jon  staff     533 Mar  4 08:09 pull.sh
-rw-r--r--   1 jon  staff  323591 Mar  4 08:06 dat20200304.gz
-rw-r--r--   1 jon  staff    4186 Mar  4 08:06 identities.dat
drwxr-xr-x  23 jon  staff     736 Mar  4 08:06 dats
...

Step 5:

Step 6: