2019-02-14 AAI Developers Meeting

Shared Cassandra Database

@Mahendra Raghuwanshi

@Roger Maitland

Open

Regarding https://lf-onap.atlassian.net/browse/OOM-1194 / https://lf-onap.atlassian.net/browse/OOM-1199:

• Is this something AAI team needs to be aware of? Or is it OK for OOM team to just switch it around?

• Will this introduce unexpected dependencies between AAI, AAF, OOF, Portal and SDC that will create difficulties for upgrade/downgrade/backup/restore/maintenance/schema change?

• Is it going to exacerbate the performance problems already noted?

Change PNF to use pnf-id as unique key

@Benjamin Cheung

@Chesla Wechsler

@David Perez Caparros

@Former user (Deleted)

@Matthieu Geerebaert

Open

Potentially breaking change: https://lf-onap.atlassian.net/browse/AAI-2096

See also:

• Proposal to Change AAI PNF Entity to use PNF-ID as key

• AAI R4 Use Case and Functional Requirements Impacts

Questions:

• how to minimise impact of the transition from pnf-name as unique to pnf-id as unique key?

  • would the v14 URL be different from the v15 URL? would both paths be equally supported for GET/PUT/etc?

• what forwards-compatibility or backwards-compatibility will be supported?

• how to migrate forwards or backwards database versions, ONAP versions, etc, across this transition?

• who is going to implement it? Test it?

• what is the impact of this not going ahead?

AAI-BBS for Dublin

@Keong Lim

Open

Started collating proposed changes in wiki AAI-BBS Proposals for Dublin Release

Added https://lf-onap.atlassian.net/browse/AAI-2154: discussions about options. Perhaps some of this should go into the "AAI data model principles document"?

AAI REST API

@THIRILOSHINI KRISHNAKUMAR

Open

1. I see APIs to ADD/DELETE/UPDATE a single PNF to/from AAI. Is it possible to do a bulk ADD/DELETE/UPDATE ? That is, ADD/DELETE/UPDATE multiple PNFs in one API call instead of doing multiple calls for each PNF.

The reason behind the question is that, I’ll have to get Nodes from SDNC (third party) and add it to AAI. When there are 100s of nodes, making more than 100 API calls individually to add the nodes becomes expensive.

If the bulk operation is not supported, Can you please help me understand if there is any specific reason for not supporting it ?

2. I added some p-interfaces as part of adding a pnf. Please find below a sample json that I used. I use POSTMAN for all REST operations.

The API to add the PNF: PUT https://10.1.8.105:8443/aai/v14/network/pnfs/pnf/T310-NE4

INPUT:

{

"pnf-name": "T310-NE4",

"in-maint": false,

"p-interfaces": {

        "p-interface": [

                {

                        "interface-name": "otu-1/1/0/E1.1",

                        "in-maint": false

                },

                {

                        "interface-name": "otu-1/2/0/E1.1",

                        "in-maint": false

                }

        ]

  }

}

The pnf addition was successful. When I try to fetch the pnf, I expected to see the p-interfaces as well under the pnf element as that’s how I see in the pnf model. But I had to make a separate GET API call to see the interfaces.

The API to get the pnf: GET https://10.1.8.105:8443/aai/v14/network/pnfs/pnf/T310-NE4

OUTPUT:

<?xml version="1.0" encoding="UTF-8"?>

<pnf xmlns="http://org.onap.aai.inventory/v14">

<pnf-name>T310-NE4</pnf-name>

<pnf-id>059b7753-0c29-3407-a855-11174a1c77bc</pnf-id>

<in-maint>false</in-maint>

<resource-version>1550012822891</resource-version>

</pnf>

----- I expected to see the p-interfaces in the above output

The API to get the p-interfaces: GET https://10.1.8.105:8443/aai/v14/network/pnfs/pnf/T310-NE4/p-interfaces

OUTPUT:

<?xml version="1.0" encoding="UTF-8"?>

<p-interfaces xmlns="http://org.onap.aai.inventory/v14">

<p-interface><interface-name>otu-1/2/0/E1.1</interface-name><resource-version>1550012823703</resource-version><in-maint>false</in-maint></p-interface>

<p-interface><interface-name>otu-1/1/0/E1.1</interface-name><resource-version>1550012824476</resource-version><in-maint>false</in-maint></p-interface>

</p-interfaces>

https://docs.onap.org/en/casablanca/submodules/aai/aai-common.git/docs/AAI%20REST%20API%20Documentation/nodesQuery.html#depth

range query

@Keong Lim

In Progress

7th Feb 2019

• if I had some AAI data with attributes that are strings but nominally contain date/timestamps, is there a way to query for a particular range of values?

• is there a way to do partial match? regex?
  PUT /aai/v13/query?format=raw

  {"gremlin": "g.V().has('aai-ts', org.janusgraph.core.attribute.Text.textContains('URL'))"}

  https://github.com/JanusGraph/janusgraph/blob/master/janusgraph-core/src/main/java/org/janusgraph/core/attribute/Text.java

Image footprint reduction as part of CIA project

@Paul-Ionut Vaduva

In Progress

7th Feb 2019

Reduction in size is mostly onf aai-common image as that based on ubuntu.

2/7 - Move the base image to be a part of ONAP Build, maybe aai-common repo

Move the aai-common Dockerfile RUN into the resources, traversal, graphadmin, cacher, schema service microservice

AAI Questions

@THIRILOSHINI KRISHNAKUMAR

Done

7th Feb 2019

We are trying to bring in transport network into ONAP as part of CCVPN usecase. We have some questions on AAI for the same.

- We are using a SDN controller which exposes TAPI Rest APIs. I'll have to map the TAPI model to ONAP AAI Model to add the topology to AAI. We are planning to use a mediation which does the model mapping and push the topology to AAI. Is this approach valid or Is there any other standard approach of doing it ? Is there any example that i can refer to where somebody has done mapping from one model to AAI model.

- Are there any examples of AAI Rest API to add/delete/update nodes,its interfaces and links to and from AAI inventory that i can refer to ?

- The AAI model, may i know if it follows any standards ?

- In the model under physical link and logical link, i do not see any endpoint information. May i know how the links are formed when it doesnt know what the endpoints are

- We have a onap cluster running Beijing version. Is there any standard approach of upgrading it to Casablanca.

Casablanca RESTAPI Documentation

https://onap.readthedocs.io/en/casablanca/submodules/aai/aai-common.git/docs/AAI%20REST%20API%20Documentation/AAIRESTAPI_CASABLANCA.html

https://nexus.onap.org/service/local/repositories/releases/archive/org/onap/aai/aai-common/aai-schema/1.3.4/aai-schema-1.3.4.jar/!/onap/aai_swagger_html/aai_swagger_v14.html

2019-02-06 AAI Meeting Notes Amit Sinha discussed an approach for upgrading version

A&AI Data Restore for OOM - Casablanca

Schema-service roadmap

@James Forsyth

@Former user (Deleted)

On Hold

31th Jan 2019

The schema-service is ready. Currently it provides file-sharing capabilities in terms of schema/edgerule files.

In order for GraphGraph to take advantage of the schema parsing/processing in schema-service additional abstractions have to be implemented on top of the crude file2string functionality currently in schema-service.

Open question:

1. What schema abstractions are planned in schema-service?

2. When will the abstractions (from point 1.) be implemented?

New AAF Certificates at startup

Jimmy Forsyth 

In Progress

24th Jan 2019

AAF will generate certificates to the be used by the containers at startup; AAI services should use the run-time generated certs instead of the ones that are in the repos or oom charts.

In dublin the services will mount a volume with certificates. This is on the roadmap for Dublin as a feature.

• is this for all service and/or HAProxy?

• Where are the certificates coming from (OOM/gerrit/generated by AAF)
  
  

AAI Backup and Restore

@Keong Lim

On Hold

10th Jan 2019

@FREEMAN, BRIAN D asked on Re: Backup and Restore Solution: ONAP-OOM :

what would be the approach to backup an entire ONAP instance particualarly SDC, AAI, SDNC data ? would it be a script with all the references to the helm deploy releases or something that does a helm list and then for each entry does  the ark backup ?

What is the AAI strategy for backup and restore?

What is the overall ONAP strategy for backup and restore?

Should it be unified with the data migration strategy as per "Hbase to Cassandra migration" on 2018-11-14 AAI Meeting Notes?

Jimmy didn't directly raise the topic but there was movement - @Keong Lim asked "if istio service mesh is a no-go, is there a replacement for secure onap communications?
is backup/restore/upgradability included in s3p?"

@Michael O'Brien replied that a reference tool set for backup and restore was introduced in Casablanca:  Backup and Restore Solution: ONAP-OOM

@Mike Elliott said he would look at Brian's question, AAI will provide support as needed.

aai-cassandra performance issues

@Keong Lim

On Hold

10th Jan 2019

@Michael O'Brien has documented performance issues in aai-cassandra:

• Cloud Native Deployment (search for LOG-376 to find the specific references to AAI)

• https://jira.onap.org/browse/LOG-376?focusedCommentId=29358&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-29358

hector has discovered that the stress test jar (liveness probe?) in aai-cassandra is hammering the cpu/ram/hd on the vm that aai is on - this breaks the etcd cluster (not the latency/network issues we suspected that may cause pod rescheduling)

Is there something that should be tweaked in AAI config? Or documentation on the recommended setup to run the VM?

I'll come to the next AAI meet (conflicts with pomba meet) -

https://lf-onap.atlassian.net/browse/LOG-376

20190108 work continues to find the cause - I see 7 vCore spikes on cassandra as well as a saturated logstash on that particular vm - we are no longer a DaemonSet (13 instances on a 13+1 cluster) - I will reduce the current ReplicaSet from 5 to 2 or 1 until I can label the nodes and/or find out what is causing ls to saturate - @Prudence Au and Sanjay Agraharam mentioned cassandra - I have seen cs high on several "top" sessions - will post screen caps - bottom line is correlation - I have a 2nd cluster where I can just run aai,dmaap and log

LOG Meeting Minutes 2019-01-15

https://lf-onap.atlassian.net/browse/LOG-915

On-hold for 3 weeks (end of January) - if until then no performance issues reported agenda item will be closed

get notified of AAI Cassandra issues automatically

@Keong Lim

In Progress

24th Jan 2019

@Mike Elliott wrote in OOM Meeting Notes - 2018-12-5

f. AAI team wanted to get notified of AAI Cassandra issues automatically
                i. Can we setup a Nagios or equivalent to monitor both rancher/k8 and the applications for rancher/k8 issues ?

Keep an eye out for new issues!

Question: @Keong Lim should this be part of a larger A&AI monitoring and failure prevention initiative

Purpose of fields in AAI

@Keong Lim

Open

@Dénes Németh wrote in https://lf-onap.atlassian.net/browse/AAI-1104:

In think it would be good to answer what is the meaning of the field (collection of PEMs of the CA xor URL)

Questions:

1. Is AAI intended to strictly prescribe how the fields are used and what contents are in the values?
2. Or does AAI simply reflect the wishes of all the client projects that use it to store and retrieve data?

Even if (1) is true, AAI is not really in any position to enforce how clients use the data, so really (2) is always true and we need to consult the original producers of the data and the ultimate consumers of the data to document their intended meanings.

How do we push to have documentation on the purpose and meaning of the fields in AAI?

Where does all this documentation go?

Should the documentation be backed up by validation code?

See also discussion about AAI in 2018-11-28 ExtAPI Meeting notes

29th Nov: Started on new wiki page AAI Schema Producer-Consumer Pairings

Copyright license header restricted to 1 company

@Former user (Deleted)

Open

24th Jan 2019

In the copyright

/**

 * ============LICENSE_START=======================================================

 * org.onap.aai

 * ================================================================================

 * Copyright © 2017-2018 AT&T Intellectual Property. All rights reserved.

If there is a company other than AT&T the build fails saying the license header is wrong

New cases found in gerrit reviews on aai/traversal:

• https://gerrit.onap.org/r/#/c/78286/

• https://gerrit.onap.org/r/#/c/78284/

@Keong Lim to try a license change in aai/resources to see if it also fails the same verify job.

Return codes and messages for WS

@Former user (Deleted)

Open

Is there a guide for the description of the error message and the error codes? How are new error states (message + code) added?

OOM Artifacts

@Former user (Deleted)

Open

Some of our top level OOM deployment artifacts are not unique (i.e. don’t take namespace into account as all other deployables), is that intentional?

AAI test data bootstrap

@Keong Lim

Open

Looking at AAI usage in OOF - HPA guide for integration testing by @Dileep Ranganathan, wondering whether there is a better way to bootstrap AAI test data?

Note: Required only if the Multicloud has no real cloud-regions and HPA discovery cannot happen.

If Multicloud team has data for creating the Cloud-region and doesn't have the HPA, then please update the existing data with the flavors with HPA.

1. Import the postman collection CASABLANCA_AAI_postman.json

2. To add/remove HPA Capabilities edit the flavors section in the body of PUT Cloud-Region{x}

3. Once all the necessary Use postman to add the complex and cloud regions in the order specified below
   (snip screenshot of specific sequence)

4. Use the GET requests to verify the data.
   (snip screenshot of specific sequence)

Similarly, @Scott Seabolt and @J / @Joss Armstrong wrote for APPC Sample A&AI Data Setup for vLB/vDNS for APPC Consumption and Script to load vLB into AAI:

The below put_vLB.sh script can be used to submit the vLB data to A&AI in order to run ConfigScaleOut use case. This script and referenced JSON files are used on an AAI instance where the cloud-region and tenant are already defined.

Similarly:

• https://lf-onap.atlassian.net/browse/TEST-133

• https://lf-onap.atlassian.net/browse/INT-705

• vCPE Use Case Tutorial: Design and Deploy based on ONAP

Related https://lf-onap.atlassian.net/browse/AAI-1948 on the brittleness of the ReadTheDocs links to data files.

One for VIM: How-To: Register a VIM/Cloud Instance to ONAP and https://lf-onap.atlassian.net/browse/AAI-1928

Potential issues:

• fragility of static import data file w.r.t. schema changes and version upgrades for each ONAP release?

• how "common" is this knowledge, i.e. what to load, where to get it, who else should be using it, etc?

• should it be automated/scripted, rather than manual steps to bootstrap?

• should it be a simulator program or test harness, rather than a static data file?

• should it reside within AAI CI/CD jobs for maintenance and upgrade of schema versions?

• who maintains the data itself? Is there a "data repository" which can be delegated to other teams, e.g. like documentation repository links in git?

• how many other teams have similar private stashes of AAI bootstrap data?

• does it need to be published at a stable URL to avoid linkrot?

AAI too slow for OOF/HAS

@Keong Lim

Open

Under OOF Homing and Allocation Service (HAS) section, @Dileep Ranganathan wrote about Project Specific enhancements:

Optimize - AAI cache

• Use MUSIC or any other alternative in memory caching like Redis etc?

• Optimize flavor retrieval from A&AI and Cache the information if necessary

See also https://lf-onap.atlassian.net/browse/OPTFRA-268 / https://lf-onap.atlassian.net/browse/OPTFRA-291

Similarly to the "AAI too slow for Holmes" item below, this introduction of extra caching of AAI data is a worrisome development and sad indictment of the performance of the system architecture.

What can we do about this?

Would the AAI Cacher https://lf-onap.atlassian.net/browse/AAI-1337 help to improve performance?

MultiCloud usage of AAI for HPA telemetry/time-series data to OOF

@Keong Lim

Open

@Bin Yang and @Lianhao Lu (Deactivated) wrote in https://lf-onap.atlassian.net/browse/MULTICLOUD-274:

HPA telemetry data collection and make it persistent in A&AI, from which OOF can leverage during its decision making process.

and

1.  Multi-cloud to collect the data from time-series data services like Prometheus (http://prometheus.io) or openstack Gnocchi, and push them to A&AI based on the data recording & aggregation rules.

and

The reason why we propose here is that VES mechanism doesn't store the telemetry data into A&AI. And OOF now can only get those kind of data from A&AI.

Some concerns:

• how much additional load will this place on AAI?

• will AAI cope with this load?

• is AAI suitable for "time-series data"?

• is "telemetry data" considered to be "active & available inventory"?

• should OOF access the telemetry/time-series data via other means (not AAI)?

• AAI API latency (4~6 second per request as benchmarked in CMCC lab) could be a problem

Orchestration Scenarios for VNFs

@Keong Lim

Open

Comments on Orchestration Scenarios related to AAI:

@Viswanath Kumar Skand Priya / @kspviswa said:

Thank you @Ranny Haiby & @Fernando Oliveira . I agree partly, but I still have following queries.

• I agree & acknowledge that atleast for a foreseeable future, we would need a way to specify the VNFM / NFVO as part of "Design Decision", which I believe can be reflected as part of VNFD/NSD ( using some special attribute ) or as part of internal Model that SDC might build before distributing the same. SO can then use this hint to select relevant actors. My only question is, why this has to be maintained in AAI which is exclusively for runtime record? All AAI cares about is what is running in the network irrespective of how that got orchestrated. Isn't it ?

On a broader note, I would like to understand what's the original intent of AAI ( atleast in ECOMP world ) ? Are we simply assuming that, just because AAI has "available inventory" in its name, we are expecting it to keep track of cloud inventory realtime ? Because our entire story ( including the new G-FPS proposal ) is based on this assumption. Can anyone from AAI team or ATT clarify on this ?

Because AFAIK, AAI neither has the schema to host such available inventory, nor the MC has the pub/sub or polling mechanism ( today ) to refresh the cloud inventory inplace. Ofcourse those can be scoped for further releases, but my original question is, was that the original intent behind AAI or are we now including it in the scope?

and @Fernando Oliveira replied:

For the first question: I think that A&AI needs to maintain the VNF instance ↔ VNFM instance and the NS instance ↔ NFVO instance relationship for subsequent life cycle operations, i.e. a scale or heal operation.  The path would be something like Event (VNF Instance, Busy)  → DCAE (policy for VNF instance) → Policy Evaluation (VNF instance, Scale-out)  → SO (VNF instance, Scale-out) → A&AI (find VNFM instance for the VNF instance) → SO (VNF instance, VNFM instance, Scale-out) → SOL003 Adapter (VNFM instance, VNF instance, Scale-out) → VNFM instance (VNF Instance, Scale-out).

As I understand, ESR has "esr-vnfm-list", which has an "esr-vnfm", which has "esr-system-info-list", which has "esr-system-info", which has a "relationship-list" that can contain relationships to "generic-vnf" and other AAI objects.

The "generic-vnf" object also contains "self-link", "ipv4OamAddress", "ipv4OamGatewayAddress", etc, which links the AAI object back to its "source-of-truth" external-system.

Is there some new data, new schema or new API that is required on top of this?

@Fernando Oliveira; Apologies for my lack of knowledge, but a few comments:

1. For the VNF/VF ↔ VNFM case, I think that there needs to be a reference from a VNF/VF instance record to the specific instance of the VNFM that was used to deploy the VNF/VF. If there is already such a reference from the VNF/VF through the ESR to the specific item on the esr-vnfm-list, then I think that would be sufficient. If not, I think that would be a new requirement.

2. For the Service ↔ NFVO case, Is there an equivalent NFVO/Orchestrator list in the ESR? The esr-nfvo-list would need the same set of info as the VNFM case. If the esr-nfvo-list does exist, I think that there needs to be a reference from the Service Instance record to the specific NFVO instance that deployed the Service. Is there such a reference? If not, I think that would be a new requirement.

@Bo Lv can comment more on the current ESR capabilities, but I believe there are only 3 kinds of systems so far: EMS, VNFM and third-party SDNC.

ESR could be extended to handle VNFO as another kind of system.

@Fernando Oliveira : I created JIRA stories:

• https://lf-onap.atlassian.net/browse/ONAPARC-388

• https://lf-onap.atlassian.net/browse/ONAPARC-389

• https://lf-onap.atlassian.net/browse/ONAPARC-390

for various parts of the scenario.

Modeling team R4 discussion, including extra AAI attributes in a model-driven way

@Keong Lim

In Progress

15th Nov 2018

Modelling team having Service Instance thoughts by Chesla Wechsler, which will affect AAI schema.

Also referred from comments on ONAP R4+ Service Modeling Discussion Calls

9)“vhn-portal-url”?“Bandwidth”,"QoS","SLA",etc, attribtutes that not all the services need but still need to be stored in certain service instance: stored as a schemaless field on the service-instance vertex (Chesla will follow up) (my concerns: according to the call, is that ok if we set a "global-type of service" and a "customized-type of service", then mapped it with internal descriptor, and A&AI's model only stores global type in service instance's schema, but stores the customer-faced attributes of service in a schemaless way? Chesla Wechsler  )