DMaaP Meeting Minutes

Ref

Status

Description/Notes

1

Closed

Need Jira and Gerrit ids for Team Fortress.  Some members of Fortress have LF ids.  No problems yet.

2

Closed

Need SSL certs from AAF team.  How to initiate the process for acquisition?  Waiting for instructions from AAF (common to all teams).

• Demo planned for 7/24. Meeting info distributed to our team.
• 7/24: New documentation/video link: http://onap.readthedocs.io/en/latest/submodules/aaf/authz.git/docs/sections/development/
• need to initiate the cert request process.
  8/2: Work ongoing, mail sent to to request ONAP certificate
• 8/8: Dom to investigate generation of cert for DR until Jonathan returns. That failed due to lack of perms.
• 8/8: Ram requested assistance from Sai for Ronan's certificate request
• 8/15: still no response from Jonathan
• 8/17: AAF provisioning completed by Jonathan. Need to download certs now. Dom will download the certs until Fortress has access to Windriver.
• 8/22: Dom provided certs, but Fortress needs some overview.
• 8/24: replaced self-signed certs and are into testing

Can Ram request Committer privilege for a designated Team Fortress member?

7/25: Ram says first start Contributing, and then send requesting email to all existing DMaaP Committers. Process at: Committer Promotion Request

7/31: Ok with status quo even if Fortress contributors are not approved.

Can't see Sonar results until project CLM jobs are working.

8/2: nexus-iq links fixed, awaiting docker image push before we run CLM job

8/7: java release job still failing so fixes aren't visible yet

8/8: New Sonar report generated

Ref

Status

Description/Notes

1

Closed

Any additional documentation that can help?

Dom has identified some internal AT&T deployment docs but needs to scrub them before releasing.

Dom distributed 2 docs on installation and configuration

2

Closed

Identification of committers for relevant ONAP projects. Need to identify committer per gerrit repo:

dmaap/datarouter: Ram Koya

integration: Gary Wu

ci-management: Jessica Wagontall

oom: Borislav Glozman, Michael O'Brien, Mike Elliot

Closed

Need to review ONAP milestones – highlight high risk items.

7/25: might need a level of discretion / exemption required for what we submit on next milestone this week(M3). Ram: Is there a way to limit the functional commitment of DR to just what is needed for RAN Use Cases? e.g. existing code base, single Node, existing pub/sub API, simplest pub and sub routing path, no heat template for Itegration-Stable

Review RAN Use Case and identify DR features which will be exercised, then focus all efforts on this. Leah will follow up with RAN team to identify Use Cases.

7/26: reviewed RAN Use Case and confirmed DR functionality is adequate. And may have clarified even more function to exclude from code coverage.

4

Closed

Tie into CI process.  JJB templates.  What is repo?  What are good examples?

7/24: Sunil shared existing JJB templates

7/25: Dom provided overview of CSIT process

5

Closed

Automated testing framework.  Show some examples/repos.

1. Integration test team using ONAP test framework (aka CSIT). See item 4. Closed.
2. Unit test  (aka Code Coverage). See item 12 details
3. OOM healthcheck. ...leave till OOM work. Part of DMAAP-107. healthcheck is NOT in the Helm chart - Sunil to send link with info
   8/10: Sunil shared email from Roger Maitland (Subject: DMaaP Message Router Health Check). Very good explanation that should be captured on the wiki someplace.
   8/17: understand what is needed. need to get containers up first.
   8/22: containers working. focus on sub testing first
   8/29: target healthcheck for next week. initial oom delivery is waiting OOM team review and merge.
   9/5: healthcheck included in oom delivery - waiting to be merged
   9/10: target for merge is today

6

Closed

Different from AAI DR?

Yes, aai/datarouter seems to be a service that listens to some Message Router topic for certain events for the purpose of discovery.

7

Closed

Introduce ONAP required documentation.  Architecture, APIs, etc.

7/25: need to assign JIRA tickets

7/26: Brian needs to enumerate what documentation is really needed

7/27: Conner provided a list of documentation requirements which will be added to Jira

7/24: Ronan asks:

We have started looking into Jenkins Job Builder.

-          We have started with the Using Standard Jenkins Job (JJB) Templates page – is this the best place to start? A: Yes, but also look to existing project templates as working examples.

-          Do you have any sample template files that we could start with? A: Yes, Sunil to share. Sent.

-          Will we need access to run jobs in https://jenkins.onap.org/ ? A: No. The commit of the template into the ci-management/jjb repo is sufficient to run the job.

Be sure not to introduce any OpenSource security vulnerabilities. DR code has not been scanned yet. Expectation is that mySQL driver has some known vulnerability. Other projects have switched to PostgreSQL, or may need to claim an exception. Strategy: get a version committed ASAP so scan gets done.

7/26: get a container up and running and committed so we get a scan. Using 3/23 scan results, found few violations - Dom to send that report. Also mysql issues were NOT severe (red).

7/27: known vulnerability: new version of com.thoughtworks.xstream. Also, mySQL has a license violation, so may need to use mariadb

8/2: See Section 4, in blocking issues

8/7: code changes in progress, but still can't see because of blocking issue #4

8/8: DMAAP-557 has fixes

8/9: complicated by Jetty 9 API changes. But mySQL and xstream changes are finished.

8/17: improved results but remaining changes: xstream 1.4.10,, jetty 9.4.12.RC2 (Sunil used 9.3.8)

8/22: 1 violation left: License not approved: CDDL-1.1 or GPL-2.0, CDDL-1.1 or GPL-2.0-CPE. In  javax.websocket : javax.websocket-client-api : 1.0. See onap-discuss email thread.

8/27: Warning from jetty logs stating that the RC2 version is unstable and shouldn't be used in production. Possibly need to downgrade??

8/27: Agreed that all dmaap components should stay in sync so use 9.3.8RC0

8:28: Initiated run-clm on last merge (which uses 9.3.8.RC0). Results: https://jenkins.onap.org/job/dmaap-datarouter-maven-clm-master/2/ Summary: 0 Critical, 24 Severe. Many of the Severe license issues are due to missing license declarations.

8/28: Dom: Tried 9.3.8.RC0 for buscontroller. It did clear the old violations, but introduced 2 new Severe license issues. So, I switched back to 9.4.12RC0 which only has 1 Critical security violation for a configuration option that doesn't apply to us. I requested a "not applicable" designation from help@onap.

8/30: Dom escalated to Steve Winslow for guidance on procedure. NOTE: related to point 30

8/31: Winslow cleared 9.3.8.RC0 license exceptions.

9/5: team will work on remaining violations

9/14: down to 8 severe. will pursue with Steve Winslow

9/19: add dom4j explanation (restricted to unit test) on vulnerability page

9/24: no critical. 2 severe security only used for unit test. 5 severe license waiver requested to Steve Winslow.

Do we need to amend the M1 checklist to include DR? Or do we want to get a bit more confidence?

Is there risk for NOT amending the checklist? Anne notes that Functionality Freeze is 7/26 anyway, so maybe that is the place to re-introduce DR.

Anne to review with Ram.

7/30: Moot point now!

Fiachra requests help with code testing error resolution. Dom to meet to review who to contact next.

7/26: Fiachra demonstrated unit testing progress against DR in docker containers. Many tests (of provisioning API) are failing with 403. Attempts to manually configure for proper authorization were not successful. Also, test code seems to be missing a critical tests.properties file. Fiachra took a good stab at creating this, but uncertain if we are missing any assumptions. Dom to investigate further, and request help from prior DR contributors.

7/27: Progress! only 11 of 96 tests failing. Still concerned that code coverage metric seems not to reflect properly after these tests are run (i.e. under 5%). Will continue on current path but may need to adjust methodology.

7/30: existing tests are functional (tests the API entry points), and assume server is running in a sep container, so doesn't match the code coverage methodology. Will be a major work item. Sunil shared MR pom settings:

<jacoco.version>0.7.7.201606060606</jacoco.version> <sonar-jacoco-listeners.version>3.2</sonar-jacoco-listeners.version> <sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin> <!-- Default Sonar configuration --> <sonar.jacoco.reportPath>target/code-coverage/jacoco-ut.exec</sonar.jacoco.reportPath> <sonar.jacoco.itReportPath>target/code-coverage/jacoco-it.exec</sonar.jacoco.itReportPath> <!-- Note: This list should match jacoco-maven-plugin's exclusion list below --> <sonar.exclusions>**/gen/**,**/generated-sources/**,**/yang-gen**,**/pax/**</sonar.exclusions> <sitePath>/content/sites/site/org/onap/dmaap/messagerouter/msgrtr/${project.artifactId}/${project.version}</sitePath>

7/31: will need to mock api for unit test coverage

8/2 : No change in status, mocking API the process to increase code coverage

8/7: getting tips from test team, expects 50% to be achievable.

8/8: Target is 8/15

8/15: Sonar is not picking up code coverage (missing pom entry). Prov at 8% (Feed Servlet 95%. Subscribe Servlet 70%.) Node not started.

8/17: Prov 12%. Sunil: Be sure to add Sonar properties and surefire plugin to pom.xml

8/21: Prov 19% Sonar is still not picking up coverage - concentrating on increasing code coverage instead of resolving this

8/22: Overall > 30%. Still messing around with pom.

8/27: Results showing up in Sonar! Still just over 30% coverage. Continue covering under DMAAP-101 (Kyle)

Desire to sync with Anne about JIRA management.

7/30: need to wait for assignment of new PM

08/01: I created the next two sprints (Leah/Ericsson) so we should be able to populate DMAAP 2018-11 now, also contacted RAM about getting admin access to update the quick filters.

08/02: New sprint created - need to ensure that we also handle Casablanca Release Platform Maturity requirements

Keep this line in Copyright section of code?

ECOMP is a trademark and service mark of AT&T Intellectual Property.

A: Yes, please.

Problem with jenkins job creating docker container.

7/31: successfully building images locally. turn on boolean for push during jenkins build

8/2 : troubleshooting ongoing - expect resolution shortly

8/7: image on nexus3! seeing both SNAPSHOT and finished image, but Sunil says that is correct.

OOM updates. App team makes delivery to oom project. No special approval, but nice to let them know.

OOM NodePort List

NOTE: port reservation coordination recorded on wiki page....a good item to discuss with oom team

8/2: 2 ports reserved, a good starting place, may need a third?

Acquire original DR Prov DDL scripts.

8/3: provided by Dom

Backlog grooming. Fortress to review list and bring up anything needing discussion.

8/9: cover in JIRA discussion

Ronan:

1. Q on DMAAP-51 re docker tagging. necessary? A: Sunil: not used yet. But suggestion for creating separate jjb file for docker. see email in onap-discuss from a few weeks ago
2. See released version in Nexus3? A: Sunil advises to see if they are released by 8/9. in POM use version-SNAPSHOT. When satisfied, enter LF Ticket to release docker image. If not, then enter LF ticket.

8/9: Sunil: jar in staging until the end, then released. To release, send email to LF w/ CC to Ram for approval. Be sure image works with integration tests because tests will be repeated every day.

8/31: images on staging nexus. so lets complete some CSIT before updating manifest.

8/31: tested successfully with Buscontroller CSIT

CSIT testing: publish feed request gets redirected and auth header gets stripped.

Workarounds: publish directly to Node. Sleep before publish to allow for provisioning propagation delay.

Fortress demo of DR to RAN collector possible for this week.

8/9: probably next week. NOTE: Dom unavail on 8/20 and 8/21.

813: no reply from client group....so close till they are ready

Nobody on Dmaap team has JIRA Admin access. Gildas is on vac. Mail sent to Ram for privileges.

8/13: will provide details for filters. Ram will get that implemented in Jira. Sunil now has admin access

8/15: Rachel has admin access too

CSIT tests failing.

/setup.sh: line 16: /usr/local/bin/docker-compose: Permission denied

Sunil suggests Conor send request to helpdesk@onap.org for inquiry.

8/13: Jessica will investigate, but it cleared up in the meantime.

M3 Readiness includes API review with Arch team. Sunil opinion: Arch will request review if needed. But DR is not really new so no email necessary.

8/15: Ram: "I don’t think we need to review as its an existing component."

If DR is not supporting heat deployment, do we need a waiver of some sort? Sunil: bring up in next TSC review for official decision. Look for advice for Ram on wed.

8/15: Ram: "Should we check with deployment team if they still insist us to create heat-style deployment?" Sunil: Project is Gildas, Integration is ???

Target next thurs TSC mtg where they review M3 checklist (last thursday of release cycle)

8/17: Heat deployments contact is Marco Platania (ooo till 9/3).

8/22: Ram to inquire at Arch sub-committee mtg today

8/23: Ram has sent email to Gildas and Brian Freeman.

Answer: We are good with OOM deployment only. If time permits, we’ll create HEAT.

DR API Authorization model doesn't work well with Kubernetes. i.e. client POD names aren't known in advance, and may change, so they aren't good for DR authorized lists. For Casablanca, Fortress will introduce a config param to disable the host authorization checks. Also, will create a backlog item to re-think the API authorization model as it relates to K8S.

8/17: disabling auth checks by IP. Added story: DMAAP-597

Reviewed feed provisioning and publishing sequence with Team Frontrunner (RAN File Collector).  Initially, will assume static pre-provisioned single feed that Frontrunner can assume will exist in ONAP. Later phase would support dynamic provisioning of feed via Buscontroller, but we need all the pieces (AAF, certs, DR) in place to demonstrate how to do this, so we don't want it to block Frontrunner.

Henrik to provide Name, Description, publisher credentials for the static provisioning.

8/27: Fiachre gave Henrik the info on provisioning.

Emmett asks “Release Planning Platform Maturity Table” due date? 

Release Planning Template was due 6/28/2018  but that seems to be a different definition.

There are Platform Maturity Requirements (S3P)  with Casablanca specifics  but no date. 

8/22: see

https://wiki.onap.org/display/DW/DMaaP+R3+-+M1+Release+Planning#DMaaPR3-M1ReleasePlanning-PlatformMaturity update by RC0 (10/10)

8/31: discussed questions from Emmett's email. In general, we establish metrics and goals, and self-report back. There may be some details which could be problematic if taken literally. e.g. "single logging system" or "80% code coverage"

Sunil warns to keep an eye on "High" Security Violations for M4. May effect all projects.

8/23: In TSC call, Gildas confirmed "Severe" and "Critical" will need to be addressed.

8/24: Fortress needs access to reports so there is no delay, esp in addressing the orange violations. Need to request from Ram.

If can't fix a license issue, need to document why in project page. (Other projects have similar concerns) questions to Steve Winslow. (

swinslow@linuxfoundation.org)

Steve will do manual scans using Fossology.

9/5: Ram's advice: focus on Critical and then we'll deal with Severe

9/19: Steve will review waivers for license issues

9/27: Refer to 10

Requirement to include port number in "links" attribute of json payload/response?

eg:

"links": {
"feed": "https://dmaap-dr-prov/feed/1",
"log": "https://dmaap-dr-prov/sublog/1",
"self": "https://dmaap-dr-prov/subs/1"
}

Curl will fail without the required port so maybe we should include it?

8/27: keep an eye on this when we integrate with Buscontroller and end client pubs

8/29: another consideration, OOM deployments are short on node ports

9/10: For Casablanca, keep 8443 in place. Revisit in Dublin

Is there an ONAP container repo on public docker hub? Decide whether ONAP delivery or public delivery is best for DR client code samples.

8/29: will build docker container as part of project and deliver to ONAP Nexus

For Ram....are there logging consolidation tools requirements in Casablanca (e.g. filebeat and kabana )? as it relates to Platform Maturity.

9/5: Ram: probably kabana, but will confirm.

What is the sequence for releasing a docker container? Sunil described it. Tom was going to reply with notes so we all know the steps. see DMAAP-764 for versioning of docker containers.

Independent Versioning and Release Process.

9/24: finally its understood! details coming...

9/26: Release Version of Component

needs step 5 for oom manifest...

CII Badging. Questions about TLS certificates.

1. Certificate verification by TLS?
2. What version of TLS?
   How do we answer these?
   9/19: Tony Hansen is working this same issue for DCAE team and will share ONAP general answers.
   9/26: Conor is working this and making progress
   10/1: up to 60%

Dom to review code to remove IP addresses in source

review completed.

DR not coming up in OOM deployments. Under investigation.

Rebuilt environment, and deployment succeeded.

Sonar issues:

9/26: down to 32 sonar vulnerabilities, more coming...

10/1: 0 sonar vulnerabilities.