Page Comparison

The Cassandra problem was a combination of:
- node-0 had Out of Memory error in JVM
- node-1 had error in commit log processing
- node-2 had Out of Disk Space error on filesystem

However, Kubernetes showed node-0 and node-2 still "Running" and apparently healthy (zero restarts), with node-1 in CrashLoopBackOff (many restarts).
The failed Cassandra nodes caused the graphadmin-create-db-schema job to fail (many restarts), which causes all other AAI pods to wait in Init state.

The Cassandra problems have been manually fixed now, but should ONAP have a monitoring solution built-in to detect it?

Question 2. How can we re-run that kubernetes job?

Deleting the failed job pods cleaned up the list but did not prompt any new job pods to be created.
Deleting the graphadmin pod caused it to restart and wait for the job completion, but did not trigger a new job to start.

I think the job has now hit its backoff limit, so it no longer runs, even though we have since fixed the Cassandra problem.
Can we reset some parameter so that it will finally run to completion?
Is there an appropriate helm command or kubectl command to re-run the job?

The rest of AAI pods are obviously waiting for that job to complete before they can progress out of Init state.

(done!)
fixing the readiness-check that is waiting for cassandra

Otherwise, we can discuss in further detail on 2019-08-15 AAI Developers Meeting

Put a message onto the rocketchat also, hoping that FREEMAN, BRIAN D or another expert on ready.py script in readiness-check pod can assist.

Also discussed at OOM Meeting Notes - 2019-08-14

2

AAI Build Problem - Dublin

Ranvijay.s

Problem with cloning from gerrit and github.

Assisted with the command shown to clone several repositories in a single for loop.

3

Docker images not pushing

James Forsyth

docker pull nexus3.onap.org:10003/onap/babel:1.5.0
docker pull nexus3.onap.org:10003/onap/aai-champ:1.5.1

docker pull nexus3.onap.org:10003/onap/aai-gizmo:1.5.0

docker pull nexus3.onap.org:10003/onap/aai-data-router:1.5.0

docker pull nexus3.onap.org:10003/onap/aai-search-data-service:1.5.0

docker pull nexus3.onap.org:10003/onap/elasticsearch-sg:1.5.0

docker pull nexus3.onap.org:10003/onap/aai-model-loader:1.5.0

docker pull nexus3.onap.org:10003/onap/aai-spike:1.5.0

docker pull nexus3.onap.org:10003/onap/aai-sparky-be:1.5.0

4

Alpine images

Dmitry Puzikov (Deactivated)

Jenkins jobs needed to create / push the common alpine image that the AAI containers will use, still need this completed before we merge the outstanding commits.

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	AAI-2574

Jira Legacy

server	System Jira
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	AAI-2580

5

Microservices enabled by default in aai/oom

James Forsyth

The ones in bold will be turned "off" by default:

aai-babel aai-elasticsearch aai-modelloader aai-search-data aai-traversal

aai-champ aai-gizmo aai-resources aai-sparky-be

aai-data-router aai-graphadmin aai-schema-service aai-spike

We will add are release note about how to turn off the ones in italics for users that want a true minimum set without the UI

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	AAI-2548

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	AAI-2549

46

El Alto Review

James Forsyth

JIRA Epic/Story count:

Jira Legacy

server	System Jira
jqlQuery	project = AAI AND issuetype in (Epic,Story) AND fixVersion = "El Alto Release" ORDER BY created DESC
count	true
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176

57

Janusgraph 0.2.3

James Forsyth

Jira Legacy

server	System Jira
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	AAI-2420

James Forsyth is going to merge updates to JanusGraph, 0.2.0 → 0.2.3 in master, should be a drop in replacement.

68

Performance Issue

James Forsyth

FREEMAN, BRIAN D reported that when there are ~2400 vservers returned in a closed loop custom query, AAI takes minutes to respond. We've discussed paging previously, any other ways to optimize this?

PUT 'https://aai.onap:8443/aai/v16/query?format=resource'

{"query":"query/closed-loop","start":"/cloud-infrastructure/cloud-regions/cloud-region/CloudOwner/RegionOne/tenants/tenant/28481f6939614cfd83e6767a0e039bcc/vservers/vserver/6ad9eb6b-d7fc-46d6-8617-4c9b46e7308b"}

79

Spring Boot 2

James Forsyth

Jira Legacy

server	System Jira
columns	key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	AAI-2111

aai-common:1.5.2 (not released yet) will be the el-alto library for springboot 1.5.x microservices. There needs to be some urgency to migrate the microservices to spring boot 2, since 1.5.3+ will not be backward compatible and so dependent applications might not have security updates if they stay on 1.5.2. 1.6.0 (Frankfort, currently master branch) aai-common will be springboot 2 so microservices must be updated by Frankfurt. Springboot 1.5.21 is the latest (and last) version in the 1.5.x spring boot train.

810

Upcoming R6 Functional Requirements

Benjamin Cheung

Release 6 (Frankfurt) proposed use cases and functional requirements

Use Case Realization Call: July 24, 2019 

911

Round trip reverse engineering of the AAI data model

James Forsyth

Moved to new meeting:

AAI Information Model Reverse Engineering Meeting

1012

Vetting DSL Queries

Manisha Aggarwal

William Reehil

Will discuss strategies AAI will employ to protect itself from rogue queries.

1113

AAI History

Philip Blackwood

Discuss and review Scenarios for using historical inventory and topology

1214

Casablanca to Dublin Migration

Ankit Bhatt

3rd July: raised

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	AAI-2507

based on Ankit's analysis that found the change in EdgeRules.

1315

Build styles

James Forsyth

We have too many ways to build and deploy artifacts, which makes it difficult and time-consuming. Some of our artifacts are not backward compatible and we need to move everything up to released versions.

1416

Alternative meeting system

Just in case Zoom problems continue, try this instead:

https://meet.jit.si/onapaai

1517

New UI Features / Historical Tracking

William Reehil

Manisha Aggarwal

The AT&T team has done an exciting POC at a sprint-a-thon event that they would like to share with the community.

ATT wants to contribute additional UI views to ONAP, want to discuss path forward

1618

Run AAI containers as non-root user

James Forsyth

https://wiki.onap.org/display/DW/Best+Practices

The Docker and Kubernetes engines may run as root until such time as the products support non-root execution.
Applications may run as root within a container.
The process ID of a container must not run as the root ID with the exception of containers supporting ONAP features that require the container to run as the root ID.
Containers may run with root privileges.
Project containers that run as the root ID must document this in the release notes along with the functionality that requires the container to run as the root ID.

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	AAI-2172

1719

Container Images

James Forsyth

Container Image Minimization Guidelines

James Forsyth will ask integration team if we can use the AAI tenant space in windriver environment

1820

Bug Review

JIRA Issue Count (in AAI project):

Jira Legacy

server	System Jira
jqlQuery	project = AAI AND issuetype = Bug and status not in (Closed, Delivered) ORDER BY fixVersion ASC, priority DESC, updated ASC
count	true
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176

JIRA Issue Count (talking about AAI):

Jira Legacy

server	System Jira
jqlQuery	project != AAI AND (text ~ "AAI" OR text ~ "A?AI" OR text ~ "A??AI" OR text ~ "A???AI") AND issuetype = Bug and status not in (Closed,Delivered,Done) ORDER BY priority DESC, updated ASC
count	true
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176

1921

Gerrit Review

https://gerrit.onap.org/r/#/q/projects:aai

2022

Sonar Review

https://sonar.onap.org/projects?search=aai&sort=coverage

2123

Jenkins CLM Review

https://jenkins.onap.org/view/CLM/

2224

Code Coverage Status

James Forsyth

code coverage to 55% for Dublin
Jira Legacy
server System Jira
serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176
key AAI-2219

Champ slipped below 55% with latest commit.

Tian Lee will do babel, model-loader, champ, gizmo, event-client, spike, rest-client, validation
Arul Nambi will do sparky-be, search-data-service, data-router, logging-service, router-core
James Forsyth create ticket for El Alto for sparky, rolling back because of sonar failure
Venkata Harish Kajur will check aai-common, resources, traversal, graphadmin, cacher, schema-service (which probably already have the setting)
Add thresholds to all repos that are above 55% so it doesn't happen again
For setting the threshold on each repo, set it to the current percentage, so if it's is at 62.7%, set the threshold to 0.62

<jacoco.line.coverage.limit>0.68</jacoco.line.coverage.limit>

Sonar job for Dublin has been changed to using line coverage.

Code Block

  <execution>
                        <id>default-check</id>
                        <goals>
                            <goal>check</goal>
                        </goals>
                        <configuration>
                            <dataFile>${project.build.directory}/coverage-reports/jacoco.exec</dataFile>
                            <rules>
                                <!--  implementation is needed only for Maven 2  -->
                                <rule implementation="org.jacoco.maven.RuleConfiguration">
                                    <element>BUNDLE</element>
                                    <limits>
                                        <limit implementation="org.jacoco.report.check.Limit">
                                            <counter>LINE</counter>
                                            <value>COVEREDRATIO</value>
                                            <minimum>${jacoco.line.coverage.limit}</minimum>
                                        </limit>
                                    </limits>
                                </rule>
                            </rules>
                        </configuration>
                    </execution>

2325

Open Action Items

New Action items

...

Versions Compared

Old Version 5

New Version Current

Key

Attendees:

Open Action Items

New Action items