2019-03-21 AAI Developers Meeting
Attachments (including meeting recording)
Chat Server
private group aai-dev on Rocketchat server: http://onap-integration.eastus.cloudapp.azure.com:3000/group/aai-dev
Agenda Items
START RECORDING
Title | Responsible | Status | Last discussed | Notes | |
---|---|---|---|---|---|
1 | Integration Blockers | OPEN | |||
2 | Schema-service roadmap | ON HOLD | 31st Jan 2019 | 31st Jan 2019: The schema-service is ready. Currently it provides file-sharing capabilities in terms of schema/edgerule files. In order for GraphGraph to take advantage of the schema parsing/processing in schema-service additional abstractions have to be implemented on top of the crude file2string functionality currently in schema-service.
GraphGraph needs the following functionality: Venkata Harish Kajur and Manisha Aggarwal What is missing in schema service that is needed in graphgraph is the following:
20. Mar 2019: Open questions for schema-service:
21st Mar 2019: Based on William Reehil comments https://lf-onap.atlassian.net/wiki/display/DW/AAI+Schema+Service?focusedCommentId=16325457 what is "our future proposed functionality"? | |
3 | Shared Cassandra Database | IN PROGRESS | 14th March 2019 | Regarding - OOM-1652Getting issue details... STATUS / - OOM-1676Getting issue details... STATUS / - OOM-1186Getting issue details... STATUS / - OOM-1187Getting issue details... STATUS / - OOM-1194Getting issue details... STATUS / - OOM-1199Getting issue details... STATUS :
A number of gerrit review issues raised: https://gerrit.onap.org/r/#/c/79425/ The "rolling upgrade" change has been combined with the "shared cassandra" change. The "shared cassandra" change has been combined with the "AAF shared cassandra" change, which means it's also combined with the "AAI shared cassandra" change. This sounds like a recipe for disaster. 18th Mar: New patchsets to address our concerns https://gerrit.onap.org/r/#/c/82418/ | |
4 | CSIT / CSIT on OOM | IN PROGRESS | 14th March 2019 | New requirement to have our csit run as part of oom test environment in windriver Long term goal: one robot test case per endpoint that can run as part of e2e test AAI R4 Integration Sanity Test Plans See also Contributing To AAI Best Practise 4th March: - AAI-2208Getting issue details... STATUS / https://gerrit.onap.org/r/#/c/79583/ 13th March: Success! First commit merged, more to follow! 20th March: Second commit merged https://gerrit.onap.org/r/#/c/82623/ | |
5 | AAI too slow for Holmes | IN PROGRESS | 7th March 2019 | Guangrong Fu mentioned AAI in Baseline Measurements based on Testing Results:
The problem for caching is how to know when to update the cached data. Even though the access time may be fast for Holmes, the risk is using out-of-date data, so the correlations will be wrong anyway. Also, duplicating the AAI data outside of AAI is probably a bad architectural decision. Making AAI faster for these use cases would be better. Has there been a performance analysis of where the time is spent? Could it help to use ElasticSearch (e.g. as in sparky)? Should Holmes have a batch interface to get more AAI data in fewer calls? Or a better correlation API that results in fewer calls? 31st Oct: https://lists.onap.org/g/onap-discuss/topic/27805753 1st Nov:
Would the AAI Cacher - AAI-1337Getting issue details... STATUS help to improve performance? 5th Mar: Guangrong Fu Hi, Sorry for my late response. It took me a long time to set up AAI in my own env. For Item 10, here's some information: Main APIs invoked in Holmes for different use cases: VoLTE
CCVPN
Performance We set up an AAI env on a VM (8 cores, 16GB memory, 160GB storage) following the guidance https://lf-onap.atlassian.net/wiki/display/DW/How+to+Docker+setup+on+Single+VM+HEAT+Deployment and tried to run a VNF query using "/aai/v11/cloud-infrastructure/cloud-regions/cloud-region/example-cloud-owner-val-45051/example-cloud-region-id-val-56689/tenants/tenant/example-tenant-id-val-51834/vservers/vserver/example-vserver-id-val-51834" (which is returned by "/search/nodes-query?search-node-type=vserver&filter=vserver-name:EQUALS:") for 1000 times. It took ~95ms per query. Also, we tried to query a VNF for 1000 times via "/aai/v11/network/generic-vnfs/generic-vnf/example-vnf-id-val-92494" and the average time is ~86ms. From the result, we know that even for a single request, the time cost reaches around 100ms. Let alone there will be several requests sent to AAI when an alarm is processed by Holmes. Taking CCVPN for example, for each alarm, there are up to 7 requests made. That means it'll take around 600-700 ms for Holmes to interact with AAI. In case of alarm storms, it is hard for AAI to support such intensive queries. 6th March: Guangrong Fu In my opinion, the performance of AAI queries is not only impacted by the computation inside AAI, but also impacted by the HTTP request itself. I've done another test. I tried to send requests to the health check API (which does nothing but return immediately after it receives a request ) of Holmes. The average time cost is also ~ 70ms. So it seems to be a problem with the time cost caused by setting up and releasing HTTP connections. 6th March: Keong Regarding these queries:
What depth is used on these GET calls? If the defaulting to depth=0, then perhaps some improvement can be made by using "depth=1" or "depth=2"? Fewer calls returning more data could improve overall performance. Same could be achieved by changing to Nodes query, e.g. GET /aai/v14/nodes/p-interfaces?interface-name=nodeId-{pnfName}-ltpId-{ifName} Question1: Can the Bulk API be used with GET calls? Documentation does not show any examples of GET actions. https://onap.readthedocs.io/en/casablanca/submodules/aai/aai-common.git/docs/AAI%20REST%20API%20Documentation/bulkApi.html Question2: Would it help to have the Holmes pod co-located with the AAI haproxy and AAI resources pods? Reduced network latency could improve overall performance. Guangrong: Holmes is acutally deployed by DCAE. I'm not sure whether your proposal is feasible. What's more, the performance data I got was based on the fact that Holmes and AAI were deployed on the same VM, sharing the same docker env. | |
6 | Change PNF to use pnf-id as unique key | IN PROGRESS | 7th March 2019 | Potentially breaking change: - AAI-2096Getting issue details... STATUS / blocked by - ONAPARC-409Getting issue details... STATUS See also:
Questions:
| |
7 | Maven plugin for consistent code style | IN PROGRESS | 28th Feb 2019 | Raised JIRA - AAI-2198Getting issue details... STATUS with commits submitted for aai-common, cacher, gizmo, and more to come. Also added to Contributing To AAI Best Practise Please test and review. | |
8 | Return codes and messages for WS | IN PROGRESS | 28th Feb 2019 | Is there a guide for the description of the error message and the error codes? How are new error states (message + code) added?
| |
9 | OOM Artifacts | OPEN | Some of our top level OOM deployment artifacts are not unique (i.e. don’t take namespace into account as all other deployables), is that intentional? | ||
10 | Purpose of fields in AAI | OPEN | Dénes Németh wrote in - AAI-1104Getting issue details... STATUS : In think it would be good to answer what is the meaning of the field (collection of PEMs of the CA xor URL) Questions: 1. Is AAI intended to strictly prescribe how the fields are used and what contents are in the values? Even if (1) is true, AAI is not really in any position to enforce how clients use the data, so really (2) is always true and we need to consult the original producers of the data and the ultimate consumers of the data to document their intended meanings. How do we push to have documentation on the purpose and meaning of the fields in AAI? Where does all this documentation go? Should the documentation be backed up by validation code? See also discussion about AAI in 2018-11-28 ExtAPI Meeting notes 29th Nov: Started on new wiki page AAI Schema Producer-Consumer Pairings | ||
11 | AAI test data bootstrap | Keong Lim | OPEN | Looking at AAI usage in OOF - HPA guide for integration testing by Dileep Ranganathan, wondering whether there is a better way to bootstrap AAI test data?
Similarly, Scott Seabolt and J / Joss Armstrong wrote for APPC Sample A&AI Data Setup for vLB/vDNS for APPC Consumption and Script to load vLB into AAI:
Similarly:
Related - AAI-1948Getting issue details... STATUS on the brittleness of the ReadTheDocs links to data files. One for VIM: How-To: Register a VIM/Cloud Instance to ONAP and - AAI-1928Getting issue details... STATUS Potential issues:
| |
12 | AAI too slow for OOF/HAS | Keong Lim | OPEN | Under OOF Homing and Allocation Service (HAS) section, Dileep Ranganathan wrote about Project Specific enhancements:
See also - OPTFRA-268Getting issue details... STATUS / - OPTFRA-291Getting issue details... STATUS Similarly to the "AAI too slow for Holmes" item below, this introduction of extra caching of AAI data is a worrisome development and sad indictment of the performance of the system architecture. What can we do about this? Would the AAI Cacher - AAI-1337Getting issue details... STATUS help to improve performance? | |
13 | MultiCloud usage of AAI for HPA telemetry/time-series data to OOF | OPEN | Bin Yang and Lianhao Lu (Deactivated) wrote in - MULTICLOUD-274Getting issue details... STATUS : HPA telemetry data collection and make it persistent in A&AI, from which OOF can leverage during its decision making process.and
and The reason why we propose here is that VES mechanism doesn't store the telemetry data into A&AI. And OOF now can only get those kind of data from A&AI. Some concerns:
| ||
14 | Orchestration Scenarios for VNFs | OPEN | Comments on Orchestration Scenarios related to AAI: Viswanath Kumar Skand Priya / kspviswa said:
and Fernando Oliveira replied: For the first question: I think that A&AI needs to maintain the VNF instance ↔ VNFM instance and the NS instance ↔ NFVO instance relationship for subsequent life cycle operations, i.e. a scale or heal operation. The path would be something like Event (VNF Instance, Busy) → DCAE (policy for VNF instance) → Policy Evaluation (VNF instance, Scale-out) → SO (VNF instance, Scale-out) → A&AI (find VNFM instance for the VNF instance) → SO (VNF instance, VNFM instance, Scale-out) → SOL003 Adapter (VNFM instance, VNF instance, Scale-out) → VNFM instance (VNF Instance, Scale-out). As I understand, ESR has "esr-vnfm-list", which has an "esr-vnfm", which has "esr-system-info-list", which has "esr-system-info", which has a "relationship-list" that can contain relationships to "generic-vnf" and other AAI objects. The "generic-vnf" object also contains "self-link", "ipv4OamAddress", "ipv4OamGatewayAddress", etc, which links the AAI object back to its "source-of-truth" external-system. Is there some new data, new schema or new API that is required on top of this? Fernando Oliveira; Apologies for my lack of knowledge, but a few comments:
Bo Lv can comment more on the current ESR capabilities, but I believe there are only 3 kinds of systems so far: EMS, VNFM and third-party SDNC. ESR could be extended to handle VNFO as another kind of system. Fernando Oliveira : I created JIRA stories:
for various parts of the scenario. t is this item related to your question for Support ETSI NFV-SOL 005 (Os-Ma-Nfvo ref point ) between SO & VF-C/NFVO? Keong Lim it is related to the question | ||
15 | range query | IN PROGRESS | 7th Feb 2019 |
| |
16 | Image footprint reduction as part of CIA project | IN PROGRESS | 7th Feb 2019 | Reduction in size is mostly onf aai-common image as that based on ubuntu. 2/7 - Move the base image to be a part of ONAP Build, maybe aai-common repo
| |
17 | New AAF Certificates at startup | Jimmy Forsyth | IN PROGRESS | 24th Jan 2019 | AAF will generate certificates to the be used by the containers at startup; AAI services should use the run-time generated certs instead of the ones that are in the repos or oom charts. In dublin the services will mount a volume with certificates. This is on the roadmap for Dublin as a feature.
|
18 | AAI Backup and Restore | ON HOLD | 10th Jan 2019 | FREEMAN, BRIAN D asked on Re: Backup and Restore Solution: ONAP-OOM : what would be the approach to backup an entire ONAP instance particualarly SDC, AAI, SDNC data ? would it be a script with all the references to the helm deploy releases or something that does a helm list and then for each entry does the ark backup ? What is the AAI strategy for backup and restore? What is the overall ONAP strategy for backup and restore? Should it be unified with the data migration strategy as per "Hbase to Cassandra migration" on 2018-11-14 AAI Meeting Notes?
Jimmy didn't directly raise the topic but there was movement - Keong Lim asked "if istio service mesh is a no-go, is there a replacement for secure onap communications? is backup/restore/upgradability included in s3p?" Michael O'Brien replied that a reference tool set for backup and restore was introduced in Casablanca: Backup and Restore Solution: ONAP-OOM Mike Elliott said he would look at Brian's question, AAI will provide support as needed. | |
19 | aai-cassandra performance issues | Keong Lim | ON HOLD | 10th Jan 2019 | Michael O'Brien has documented performance issues in aai-cassandra:
hector has discovered that the stress test jar (liveness probe?) in aai-cassandra is hammering the cpu/ram/hd on the vm that aai is on - this breaks the etcd cluster (not the latency/network issues we suspected that may cause pod rescheduling) Is there something that should be tweaked in AAI config? Or documentation on the recommended setup to run the VM? I'll come to the next AAI meet (conflicts with pomba meet) - - LOG-376Getting issue details... STATUS 20190108 work continues to find the cause - I see 7 vCore spikes on cassandra as well as a saturated logstash on that particular vm - we are no longer a DaemonSet (13 instances on a 13+1 cluster) - I will reduce the current ReplicaSet from 5 to 2 or 1 until I can label the nodes and/or find out what is causing ls to saturate - Prudence Au and Sanjay Agraharam mentioned cassandra - I have seen cs high on several "top" sessions - will post screen caps - bottom line is correlation - I have a 2nd cluster where I can just run aai,dmaap and log LOG Meeting Minutes 2019-01-15 - LOG-915Getting issue details... STATUS
On-hold for 3 weeks (end of January) - if until then no performance issues reported agenda item will be closed |
20 | get notified of AAI Cassandra issues automatically | IN PROGRESS | 24th Jan 2019 | Mike Elliott wrote in OOM Meeting Notes - 2018-12-5 f. AAI team wanted to get notified of AAI Cassandra issues automatically Keep an eye out for new issues! Question: Keong Lim should this be part of a larger A&AI monitoring and failure prevention initiative | |
21 | Modeling team R4 discussion, including extra AAI attributes in a model-driven way | IN PROGRESS | 15th Nov 2018 | Modelling team having Service Instance thoughts by Chesla Wechsler, which will affect AAI schema. Also referred from comments on ONAP R4+ Service Modeling Discussion Calls 9)“vhn-portal-url”?“Bandwidth”,"QoS","SLA",etc, attribtutes that not all the services need but still need to be stored in certain service instance: stored as a schemaless field on the service-instance vertex (Chesla will follow up) (my concerns: according to the call, is that ok if we set a "global-type of service" and a "customized-type of service", then mapped it with internal descriptor, and A&AI's model only stores global type in service instance's schema, but stores the customer-faced attributes of service in a schemaless way? Chesla Wechsler Kevin Scaggs Andy Mayer) See also Modeling 2018-11-13 The service-instance already uses a "metadata" relationship, which can store an arbitrary list of key-value pairs, but perhaps AAI should extend the use of the "properties" element, which is also an arbitrary list of name-value pairs or the "extra-properties" element, which is also an arbitrary list of name-value pairs. 15th Nov: Having seen Chesla's presentation, it should be called "Model-driven schema" rather than "schemaless" behaviour, since the idea is that the changes are controlled by SDC modelling. Seems aligned to the eventual goal in AAI Schema Service Use Case Proposals and AAI Schema Service. | |
22 | 2 Types of logging in A&AI WS | ON HOLD | 1st Nov 2018 | There are 2 types of logging in the services
Is that correct? Shouldn't there be just 1 type? 1st Nov: After Casablanca release investigate logging guidelines and figure out what library to use in order to unify logging within A&AI 26th Nov: See also ONAP Application Logging Specification - Post Dublin 29th Nov: how does this fit with - LOG-877Getting issue details... STATUS ? | |
23 | Disable unused web services (see also Helm chart requested values) | IN PROGRESS | 20th Dec 2018 | Could we disable unused (i.e. not integrated) A&AI web services, so that the deployment is faster and the resource footprint is smaller? e.g. Champ (any other ws?) Motivation: Decrease the resource footprint for A&AI (ONAP) deployments Idea: we could support 2 different deployments 1. full (normal) deployment and 2. barebones deployment. The point of the "barebone" deployment would be to deploy only the essential services necessary for proper functioning of A&AI (leaving out services like cacher, sparky, graphadmin, having 1 cassandra node instead of 3 or 5 etc). In order to reduce hardware/cloud costs (mainly the memory footprint) it could be beneficial to support a minimalistic A&AI deployment. 1st Nov: Venkata Harish Kajur Former user (Deleted) - investigate how to disable/enable charts in A&AI so we can create a core group of pods which handle the use-cases and than extended group will all the services. Consider a group of unused/unintegrated services (like Champ). Consider other possible groups (like GUI?)
| |
24 | AAI Champ | IN PROGRESS | 1st Nov 2018 |
| |
25 | AAI HAProxy and 2-way-TLS | IN PROGRESS | 29th Nov | Technical solution to either decommission the proxy or make design changes to AAF to enable client side certificates. After VF2F we will know if this is a requirement in Dublin. We discuss after this date. question raised: MSB - would client authentication be supported?
| |
26 | named-query replacements | James Forsyth | IN PROGRESS | 31th Jan 2019 | - AAI-1989Getting issue details... STATUS - APPC-1245Getting issue details... STATUS : Takamune Cho pointed to inherited CCSDK functions, so added a new case to follow-up there too - POLICY-1278Getting issue details... STATUS - VID-355Getting issue details... STATUS : ittay has provided an update with 2 known queries used. - CCSDK-962Getting issue details... STATUS : Raised to Dan Timoney based on info about APPC component. Need to replace custom queries currently in use by these systems (and others?) in Dublin toward the retirement of the named-query API in Dublin
Christopher Shang is leading the effort to deprecate named queries in e-Comp. Next steps: get data from teams in order to prepare for testing the change from named queries to custom queries. See also AAI Named Queries
|
27 |