2019-01-17 AAI Developers Meeting

IN PROGRESS

Keong Lim

FREEMAN, BRIAN D asked on Re: Backup and Restore Solution: ONAP-OOM :

what would be the approach to backup an entire ONAP instance particualarly SDC, AAI, SDNC data ? would it be a script with all the references to the helm deploy releases or something that does a helm list and then for each entry does  the ark backup ?

What is the AAI strategy for backup and restore?

What is the overall ONAP strategy for backup and restore?

Should it be unified with the data migration strategy as per "Hbase to Cassandra migration" on 2018-11-14 AAI Meeting Notes?

• James Forsythwill raise the topic of having backups and restore functionality in ONAP - if it is feasible, on the roadmap and what others PTL think

Jimmy didn't directly raise the topic but there was movement - Keong Lim asked "if istio service mesh is a no-go, is there a replacement for secure onap communications?
is backup/restore/upgradability included in s3p?"

Michael O'Brien replied that a reference tool set for backup and restore was introduced in Casablanca:  Backup and Restore Solution: ONAP-OOM

Mike Elliott said he would look at Brian's question, AAI will provide support as needed.

Chandra Cinthala

IN PROGRESS

5G Use Case (Dublin)

BBS Broadband Service Use Case (Dublin)

CCVPN Use Case (Dublin)

Change Management Dublin Extensions

Edge Automation Functional Requirements for Dublin

OpenSource Access Manager (OSAM) Use Case

K8S based Cloud Region Support

• Chandra Cinthala will go over all these use cases and will write a 1 page report per use-case where he lays out the impact and needed high-level description of changes in order to support the use-cases in A&AI. He will send this reports to James Forsyth so he can plan accordingly. The 1 pagers should be discussed with the modelling comitee before sending them to James Forsyth . Chandra Cinthala should start here https://wiki.onap.org/display/DW/Joining+the+ONAP+Technical+Community when integrating to the ONAP community and the mailing list https://lists.onap.org/g/main. The deadline for Chandra Cinthala to send the reports to James Forsyth is 15 Jan 2019 

The Dublin use-cases for review are Release 4 (Dublin) Use Cases and functional requirements

On AAI weekly call on Wednesday Chandra Cinthala  will report on his progress in analyzing the use-cases.

Davide Cherubini

Susana Sabater

Emmanuel Sarris

Christopher Sheng

Tony Noori

IN PROGRESS

During the VF2F we have presented the CCVPN Use Case extension for Dublin Release.

In particular, we have defined the E-LAN Service (EP-LAN. EVP-LAN) sub-Use Case. More details can be found at this page (the presentation given to the Use Case SubCommittee and during the VF2F can also be found in that page).

We have also created 2 additional pages where we describe the:

1. Implementation Options
2. Initial Impact assessment

In the former page, we have identified Option 2 (for Service Creation) and Option A (for Service Modification) as the “ideal” implementation. The Impacts in the latter page are based on such an assumption.

We would be grateful if you could help us with the assessment for A&AI and if you could confirm the project commitment to CCVPN.

14th Dec: Regarding the Options and Impact assessment linked above:

AAI

1. Support Composite Service Instances (multi-operator)

As far as I know, the ServiceComposite pattern hasn't even exited the Modelling team's discussions for R3 or R4 workshops Hui Deng :

• Composite vs Recursive Discussion Page
• Composite Pattern UML diagram

This would seem like a stretch goal given the impacts on all the downstream clients of AAI, if it would require reworking any existing schema elements.

However, if you are looking for a zero-impact method of implementation, the EdgeRules already contain service-instance-to-service-instance relationship with label "org.onap.relationships.inventory.ComposedOf", so AAI can do the composite data with this relationship.

• James Forsythwill send a reminder e-mail to Tony and Christopher to engage them on this task, the plan is to discuss this issues and modeling in general in the first 2 weeks in January

Update 3rd January 2019 from CFL comment:

LIN MENG

Hi，I think there might be some confusions in CCVPN use case and the current service IM in the sheet.

I don't think CCVPN has extension demands for A&AI in Dublin release.

What we want to do in D release is to provide a CCVPN service by using one service template to includes the SOTN and SD-WAN resources, which will be a requirement for SDC and modeling DM ( In C release, we have to design each  resource as a service based on SDC's ability)

We've actually talked about composite pattern at the beginning of R3 and the modeling team accepts this idea but have no implementation in this concepts.

Also, I think the composite pattern is the same as nested service from service IM perspective. And I think it would be better if we unify them as one concept in ONAP for less confusion, which I will propose to next week's service IM call. Back to the point, we're in redesigning the new service IM of CCVPN and  currently, CCVPN doesn't have this need.

Former user (Deleted)  James Forsyth  Hui Deng  Davide Cherubini jianguo zeng Chuanyu Chen

Keong Lim

Just raised AAI-2082 - Getting issue details... STATUS , apart from getting a fix, if possible, may need some help with recovery actions.

Moved from AAI Weekly Status Meeting.

Michael O'Brien has documented performance issues in aai-cassandra:

• Cloud Native Deployment (search for LOG-376 to find the specific references to AAI)
• https://jira.onap.org/browse/LOG-376?focusedCommentId=29358&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-29358

hector has discovered that the stress test jar (liveness probe?) in aai-cassandra is hammering the cpu/ram/hd on the vm that aai is on - this breaks the etcd cluster (not the latency/network issues we suspected that may cause pod rescheduling)

Is there something that should be tweaked in AAI config? Or documentation on the recommended setup to run the VM?

I'll come to the next AAI meet (conflicts with pomba meet) -

LOG-376 - Getting issue details... STATUS

20190108 work continues to find the cause - I see 7 vCore spikes on cassandra as well as a saturated logstash on that particular vm - we are no longer a DaemonSet (13 instances on a 13+1 cluster) - I will reduce the current ReplicaSet from 5 to 2 or 1 until I can label the nodes and/or find out what is causing ls to saturate - Prudence Au and Sanjay Agraharam mentioned cassandra - I have seen cs high on several "top" sessions - will post screen caps - bottom line is correlation - I have a 2nd cluster where I can just run aai,dmaap and log

LOG Meeting Minutes 2019-01-15

LOG-915 - Getting issue details... STATUS

• ask Michael O'Brien about performance problems - if they persist and what the problem exactly is.
• Venkata Harish Kajur will inform Michael about the schema performance fix - he should test with the casablanca maintenance release.

On-hold for 3 weeks (end of January) - if until then no performance issues reported agenda item will be closed

Keong Lim

OPEN

Mike Elliott wrote in OOM Meeting Notes - 2018-12-5

f. AAI team wanted to get notified of AAI Casandra issues automatically
                i. Can we setup a Nagios or equivalent to monitor both rancher/k8 and the applications for rancher/k8 issues ?

Keep an eye out for new issues!

OPEN

Dénes Németh wrote in AAI-1104 - Getting issue details... STATUS :

In think it would be good to answer what is the meaning of the field (collection of PEMs of the CA xor URL)

Questions:

1. Is AAI intended to strictly prescribe how the fields are used and what contents are in the values?
2. Or does AAI simply reflect the wishes of all the client projects that use it to store and retrieve data?

Even if (1) is true, AAI is not really in any position to enforce how clients use the data, so really (2) is always true and we need to consult the original producers of the data and the ultimate consumers of the data to document their intended meanings.

How do we push to have documentation on the purpose and meaning of the fields in AAI?

Where does all this documentation go?

Should the documentation be backed up by validation code?

See also discussion about AAI in 2018-11-28 ExtAPI Meeting notes

29th Nov: Started on new wiki page AAI Schema Producer-Consumer Pairings

OPEN

While helping UUI team to debug a family of related issues such as AAI-1915 - Getting issue details... STATUS , found that their JSON request bodies had inserted extra lines that were not acceptable to AAI, e.g.

{
  ext-aai-network: {
   "aai-id": "VDF",
    "schema-version": "version-1",
    "resource-version": "1542082337501"
  }
}

vs

{
  "aai-id": "VDF",
  "schema-version": "version-1",
  "resource-version": "1542082337501"
}

Initially, I thought this could be due to XML-to-JSON translation error, but it could also have been a copy-paste from the output of a GET, e.g.

> GET /aai/v14/network/ext-aai-networks HTTP/1.1
{
  "ext-aai-network": [
    {
      "aai-id": "createAndDelete",
      "schema-version": "version-1",
      "resource-version": "1542247826990"
    },
    {
      "aai-id": "aaiId-2",
      "schema-version": "version-2",
      "resource-version": "1542029867153"
    }
  ]
}

In the spirit of Postel's Law ("be liberal in what you accept"), could/should AAI accept both variations of the input data above?

There could be additional validation that the element name inside the request body matches the element name in the URL of the API (similar validation is already performed for the key ID value).

It would also allow for simple methods of data transfer/migration, where AAI output is directly accepted as AAI input.

7th Dec: As per Jimmy's comment below, the difference between the "Not Accepted Input" and "Expected Input Format" is the behaviour controlled by "SerializationFeature.WRAP_ROOT_VALUE" as described in:

• https://stackoverflow.com/questions/37882478/serializationfeature-wrap-root-value-as-annotation-in-jackson-json
• https://fasterxml.github.io/jackson-databind/javadoc/2.6/com/fasterxml/jackson/databind/SerializationFeature.html

Former user (Deleted)

OPEN

Former user (Deleted)

OPEN

OPEN

OPEN

Looking at AAI usage in OOF - HPA guide for integration testing by Dileep Ranganathan, wondering whether there is a better way to bootstrap AAI test data?

Generating AAI data

Note: Required only if the Multicloud has no real cloud-regions and HPA discovery cannot happen.

If Multicloud team has data for creating the Cloud-region and doesn't have the HPA, then please update the existing data with the flavors with HPA.

1. Import the postman collection CASABLANCA_AAI_postman.json
2. To add/remove HPA Capabilities edit the flavors section in the body of PUT Cloud-Region{x}
3. Once all the necessary Use postman to add the complex and cloud regions in the order specified below
   (snip screenshot of specific sequence)
4. Use the GET requests to verify the data.
   (snip screenshot of specific sequence)

Similarly, Scott Seabolt and J / Joss Armstrong wrote for APPC Sample A&AI Data Setup for vLB/vDNS for APPC Consumption and Script to load vLB into AAI:

The below put_vLB.sh script can be used to submit the vLB data to A&AI in order to run ConfigScaleOut use case. This script and referenced JSON files are used on an AAI instance where the cloud-region and tenant are already defined.

Similarly:

• TEST-133 - Getting issue details... STATUS
• INT-705 - Getting issue details... STATUS
• vCPE Use Case Tutorial: Design and Deploy based on ONAP

Related AAI-1948 - Getting issue details... STATUS on the brittleness of the ReadTheDocs links to data files.

One for VIM: How-To: Register a VIM/Cloud Instance to ONAP and AAI-1928 - Getting issue details... STATUS

Potential issues:

• fragility of static import data file w.r.t. schema changes and version upgrades for each ONAP release?
• how "common" is this knowledge, i.e. what to load, where to get it, who else should be using it, etc?
• should it be automated/scripted, rather than manual steps to bootstrap?
• should it be a simulator program or test harness, rather than a static data file?
• should it reside within AAI CI/CD jobs for maintenance and upgrade of schema versions?
• who maintains the data itself? Is there a "data repository" which can be delegated to other teams, e.g. like documentation repository links in git?
• how many other teams have similar private stashes of AAI bootstrap data?
• does it need to be published at a stable URL to avoid linkrot?

OPEN

Under OOF Homing and Allocation Service (HAS) section, Dileep Ranganathan wrote about Project Specific enhancements:

Optimize - AAI cache

• Use MUSIC or any other alternative in memory caching like Redis etc?
• Optimize flavor retrieval from A&AI and Cache the information if necessary

See also OPTFRA-268 - Getting issue details... STATUS / OPTFRA-291 - Getting issue details... STATUS

Similarly to the "AAI too slow for Holmes" item below, this introduction of extra caching of AAI data is a worrisome development and sad indictment of the performance of the system architecture.

What can we do about this?

Would the AAI Cacher AAI-1337 - Getting issue details... STATUS help to improve performance?

OPEN

Bin Yang and Lianhao Lu (Deactivated) wrote in MULTICLOUD-274 - Getting issue details... STATUS :

HPA telemetry data collection and make it persistent in A&AI, from which OOF can leverage during its decision making process.

1.  Multi-cloud to collect the data from time-series data services like Prometheus (http://prometheus.io) or openstack Gnocchi, and push them to A&AI based on the data recording & aggregation rules.

and

The reason why we propose here is that VES mechanism doesn't store the telemetry data into A&AI. And OOF now can only get those kind of data from A&AI.

Some concerns:

• how much additional load will this place on AAI?
• will AAI cope with this load?
• is AAI suitable for "time-series data"?
• is "telemetry data" considered to be "active & available inventory"?
• should OOF access the telemetry/time-series data via other means (not AAI)?
• AAI API latency (4~6 second per request as benchmarked in CMCC lab) could be a problem

Keong Lim

OPEN

Keong Lim

OPEN

Comments on Orchestration Scenarios related to AAI:

Viswanath Kumar Skand Priya / kspviswa said:

Thank you Ranny Haiby & Fernando Oliveira . I agree partly, but I still have following queries.

• I agree & acknowledge that atleast for a foreseeable future, we would need a way to specify the VNFM / NFVO as part of "Design Decision", which I believe can be reflected as part of VNFD/NSD ( using some special attribute ) or as part of internal Model that SDC might build before distributing the same. SO can then use this hint to select relevant actors. My only question is, why this has to be maintained in AAI which is exclusively for runtime record? All AAI cares about is what is running in the network irrespective of how that got orchestrated. Isn't it ?

On a broader note, I would like to understand what's the original intent of AAI ( atleast in ECOMP world ) ? Are we simply assuming that, just because AAI has "available inventory" in its name, we are expecting it to keep track of cloud inventory realtime ? Because our entire story ( including the new G-FPS proposal ) is based on this assumption. Can anyone from AAI team or ATT clarify on this ?

Because AFAIK, AAI neither has the schema to host such available inventory, nor the MC has the pub/sub or polling mechanism ( today ) to refresh the cloud inventory inplace. Ofcourse those can be scoped for further releases, but my original question is, was that the original intent behind AAI or are we now including it in the scope?

and Fernando Oliveira replied:

For the first question: I think that A&AI needs to maintain the VNF instance ↔ VNFM instance and the NS instance ↔ NFVO instance relationship for subsequent life cycle operations, i.e. a scale or heal operation.  The path would be something like Event (VNF Instance, Busy)  → DCAE (policy for VNF instance) → Policy Evaluation (VNF instance, Scale-out)  → SO (VNF instance, Scale-out) → A&AI (find VNFM instance for the VNF instance) → SO (VNF instance, VNFM instance, Scale-out) → SOL003 Adapter (VNFM instance, VNF instance, Scale-out) → VNFM instance (VNF Instance, Scale-out).

As I understand, ESR has "esr-vnfm-list", which has an "esr-vnfm", which has "esr-system-info-list", which has "esr-system-info", which has a "relationship-list" that can contain relationships to "generic-vnf" and other AAI objects.

The "generic-vnf" object also contains "self-link", "ipv4OamAddress", "ipv4OamGatewayAddress", etc, which links the AAI object back to its "source-of-truth" external-system.

Is there some new data, new schema or new API that is required on top of this?

Fernando Oliveira; Apologies for my lack of knowledge, but a few comments:

1. For the VNF/VF ↔ VNFM case, I think that there needs to be a reference from a VNF/VF instance record to the specific instance of the VNFM that was used to deploy the VNF/VF. If there is already such a reference from the VNF/VF through the ESR to the specific item on the esr-vnfm-list, then I think that would be sufficient. If not, I think that would be a new requirement.
2. For the Service ↔ NFVO case, Is there an equivalent NFVO/Orchestrator list in the ESR? The esr-nfvo-list would need the same set of info as the VNFM case. If the esr-nfvo-list does exist, I think that there needs to be a reference from the Service Instance record to the specific NFVO instance that deployed the Service. Is there such a reference? If not, I think that would be a new requirement.

Bo Lv can comment more on the current ESR capabilities, but I believe there are only 3 kinds of systems so far: EMS, VNFM and third-party SDNC.

ESR could be extended to handle VNFO as another kind of system.

Fernando Oliveira : I created JIRA stories:

• ONAPARC-388 - Getting issue details... STATUS
• ONAPARC-389 - Getting issue details... STATUS
• ONAPARC-390 - Getting issue details... STATUS

for various parts of the scenario.

Keong Lim

IN PROGRESS

Modelling team having Service Instance thoughts by Chesla Wechsler, which will affect AAI schema.

Also referred from comments on ONAP R4+ Service Modeling Discussion Calls

9)“vhn-portal-url”?“Bandwidth”,"QoS","SLA",etc, attribtutes that not all the services need but still need to be stored in certain service instance: stored as a schemaless field on the service-instance vertex (Chesla will follow up) (my concerns: according to the call, is that ok if we set a "global-type of service" and a "customized-type of service", then mapped it with internal descriptor, and A&AI's model only stores global type in service instance's schema, but stores the customer-faced attributes of service in a schemaless way? Chesla Wechsler Kevin Scaggs Andy Mayer)

See also Modeling 2018-11-13

The service-instance already uses a "metadata" relationship, which can store an arbitrary list of key-value pairs, but perhaps AAI should extend the use of the "properties" element, which is also an arbitrary list of name-value pairs or the "extra-properties" element, which is also an arbitrary list of name-value pairs.

15th Nov: Having seen Chesla's presentation, it should be called "Model-driven schema" rather than "schemaless" behaviour, since the idea is that the changes are controlled by SDC modelling. Seems aligned to the eventual goal in AAI Schema Service Use Case Proposals and AAI Schema Service.

Keong Lim

CT Paterson

IN PROGRESS

Guangrong Fu mentioned AAI in Baseline Measurements based on Testing Results:

1. Cache the AAI data and refresh them periodically so that Holmes won't have to make an HTTP call to AAI every time it tries to correlate one alarm to another.

The problem for caching is how to know when to update the cached data. Even though the access time may be fast for Holmes, the risk is using out-of-date data, so the correlations will be wrong anyway. Also, duplicating the AAI data outside of AAI is probably a bad architectural decision. Making AAI faster for these use cases would be better.

Has there been a performance analysis of where the time is spent? Could it help to use ElasticSearch (e.g. as in sparky)? Should Holmes have a batch interface to get more AAI data in fewer calls? Or a better correlation API that results in fewer calls?

31st Oct: https://lists.onap.org/g/onap-discuss/topic/27805753

1st Nov:

• Guangrong Fu will try custom queries for queries that took to long to return
• The hardware (mainly storage) influences the query speed - need to find out what hardware was the speed test conducted on (Guangrong Fu will provide HW specs)
• HOLMES-186 - Getting issue details... STATUS

Would the AAI Cacher AAI-1337 - Getting issue details... STATUS help to improve performance?

Former user (Deleted)

ON HOLD

There are 2 types of logging in the services

• one read from EELFManager
• the other Logger log = Logger.getLogger( ...

Is that correct? Shouldn't there be just 1 type?

1st Nov:

After Casablanca release investigate logging guidelines and figure out what library to use in order to unify logging within A&AI

26th Nov: See also ONAP Application Logging Specification - Post Dublin

29th Nov: how does this fit with LOG-877 - Getting issue details... STATUS ?

Former user (Deleted)

IN PROGRESS

Could we disable unused (i.e. not integrated) A&AI web services, so that the deployment is faster and the resource footprint is smaller? e.g. Champ (any other ws?)

Motivation: Decrease the resource footprint for A&AI (ONAP) deployments

Idea: we could support 2 different deployments 1. full (normal) deployment and 2. barebones deployment. The point of the "barebone" deployment would be to deploy only the essential services necessary for proper functioning of A&AI (leaving out services like cacher, sparky, graphadmin, having 1 cassandra node instead of 3 or 5 etc).

In order to reduce hardware/cloud costs (mainly the memory footprint) it could be beneficial to support a minimalistic A&AI deployment.

1st Nov:

Venkata Harish Kajur Former user (Deleted) - investigate how to disable/enable charts in A&AI so we can create a core group of pods which handle the use-cases and than extended group will all the services. Consider a group of unused/unintegrated services (like Champ). Consider other possible groups (like GUI?)

• James Forsythcreates a JIRA ticket to define the list of AAI subprojects and create the categories (essential, full "experience") for the OOM deployment AAI-2025 - Getting issue details... STATUS

IN PROGRESS

Keong Lim

Following updates for AAI Developer Environment Setup - Casablanca need to update the link in step 14:

Tutorial: Making and Testing a Schema Change in A&AI

• Need separate pages for Beijing, Casablanca, Dublin, etc.
• AAI-1973 - Getting issue details... STATUS

 Harish will update the documentation - the section Tutorial: Making and Testing a Schema Change in A&AI

Need to distinguish on the wiki tutorials for:

• running A&AI (just installing for non-devs)
• seting up a dev environment to develop AAI ("Developer Onboarding" section on wiki)

James Forsyth will check if we still need to support HEAT deployment for Dublin

• James Forsyth will create a JIRA task for documenting the process of testing schema changes using OOM (instead of HEAT) 31 Jan 2019 

IN PROGRESS

Technical solution to either decommission the proxy or make design changes to AAF to enable client side certificates.

After VF2F we will know if this is a requirement in Dublin. We discuss after this date.

question raised: MSB - would client authentication be supported?

15th Dec: https://wiki.onap.org/display/DW/Pluggable+Security#PluggableSecurity-7.10Identifiedandsupportedpatternsandfeatures

• James Forsythcreates a task for encryption of communication between A&AI services and Cassandra 24 Jan 2019
  
• Tian Lee and Steve Blimkie to report on the Amdocs managed A&AI microservice wheter they support criteria from the Dublin S3P requirements 31 Jan 2019 
  

Keong Lim

Yang Xu wrote in AAF Component:

Issues and Workaround

If AAF stops working, e.g. AAI returns authorization error, bounce AAF aaf-locate and aaf-service containers. 

This looks like some problem for resiliency and healthchecks. Is there a JIRA case for this already? An earlier case mentioned the "fix" in passing AAI-1762 - Getting issue details... STATUS . Also in AAI-1772 - Getting issue details... STATUS

Helen Chen, is there a scenario for this in Casablanca Release Resiliency Testing Status?

Jonathan Gathman, Mike Elliott, how can this remedy be automated?

See also AAF-599 - Getting issue details... STATUS

• James Forsyth will ask Jonathan Gathman about this item.

Conclusion: Jonathan will improve the stability of his service.

IN PROGRESS

AAI-1989 - Getting issue details... STATUS

APPC-1245 - Getting issue details... STATUS

POLICY-1278 - Getting issue details... STATUS

VID-355 - Getting issue details... STATUS

Need to replace custom queries currently in use by these systems (and others?) in Dublin toward the retirement of the named-query API in Dublin

1. Coordination with the ROBOT team needed for data population
2. Coordinate with each team (POLICY, VID...) to have the specific data for each named-query

Christopher Shang is leading the effort to deprecate named queries in e-Comp.

Next steps: get data from teams in order to prepare for testing the change from named queries to custom queries.

• Keong Limwill ask the teams about the data mentioned above.
  
• James Forsyth will remind Christopher to create a LF account 24 Jan 2019 

AAI-1940 - Getting issue details... STATUS

• James Forsyth will apply the changes he outlined in AAI-1940

Discuss a move to janusgraph 0.2.2 or 0.3.1 in Dublin.

Benefits ?

Breaking change + migration needed ?

Analysis needed on benefits of 0.3.1 until 10th Jan 2019 (decision should be made on the meeting on 10th).

If no significant benefit in 0.3.X present and migration would be needed then upgrade only within 0.2.X version (assuming upgrading within 0.2.X is not a breaking change)

Conclusion: We will update to 0.2.2 and try it as a drop-in replacement. Testing should be done in a special environment with existing data - maybe the Windriver A&AI env with data.

• James Forsyth creates a JIRA task for the JanusGraph update 24 Jan 2019 

James Forsyth

IN PROGRESS

https://spring.io/blog/2018/11/29/spring-boot-1-5-18-available-now - support for 1.5.x spring boot will end in Aug 2019 so we should consider upgrading.

We should update to spring 2.X.X.

On the meeting on the 10th Jan discuss if anyone has anything against it, if not then James Forsyth creates a JIRA item for Dublin for upgrading the spring in all A&AI repositories.

Conclusion:

• Venkata Harish Kajurwill check with Manisha Aggarwal if the update to 2XX version of spring boot will happen in Dublin 24 Jan 2019