Page Comparison

DR transaction logs approach

Mail sent form Dom

Just wanted to give you heads up on a change to DR made to AT&T version: transaction log archiving moved from MySQL to ELK.

Reasons:

• Mysql couldn’t keep up with daily volume
• Offload transaction archives where space can be managed independently
• Common GUI (kabana) for examining logs
• DR LogAPI can still interface to ElasticSearch API

I recall that you already were doing something with filebeat in the ONAP version, but I have to admit that I haven’t followed it closely.

So, perhaps you are already applying this approach to DR transaction logs?

But if not, I think we should consider the approach chosen by AT&T so that the DB is only used for provisioning data.

04/24:

why?

Data up to 24 hrs max

Solution: Create ticket to track/implement this (EL Alto)

Ubuntu vs Alpine Image footprint reduction

02/01: Adolfo Perez-Duran

The CIA team is preparing to submit contributions to DMAAP to migrate the base images to ONAP Normative Container Base Images.

This migration is expected to reduce the image footprint and to enable multi-cpu architecture support for Dublin

We expect the work to be minimally disruptive and to coordinate changes with tour team.

ubuntu v alpine the talk continues

02/13: Lots of chat on discuss chat - need it to settle down before we implement

02/20: Frank Sandoval [mailto:frank.sandoval@oamtechnologies.com] to provide an update !

02/20: Dom, Conor & Sunil to assist here

Mandar to raise at TSC to see if the right course of action

02/22: Change image &

 Impacts: Hold off - will these reduce the container footprint? -Dosen't look likely, it is more to be uniform across ONAP

1) suggested we change our Maven approach to use "fabric" plugin instead of spotify (support for this has discontinued) plugin

2) use docker file instead of POM is recommended from spotify (currently we don't use except CSIT environment)

02/27: lets pose question to frank

02/27: Alpine is the decision for Dublin

03/01: Questions mailed to Frank - awaiting response

see confluence spac

testing changes - and need to send gerrit review to Dom

03/08: Frank made changes in relation to Alpine images only

(unit test seems ok, built in jenkins - images not pushed to docker repo - investigation ongoing?

Dom to check CSIT tests,

Gerry to see if DR have similar issue)

03/13: Need to chase up with Frank in relation to jenkins build not pushing up images

new image name is bc - oom changes pending

03/15: Frank merged 4 commits today, Dom & Sunil to be put on review

03/20: Worse - images not getting pushed, old images are no longer there

CI-MGMT - where is the review? Dom to reach out

03/22: Switch to Alpine broke 2 things in BC

• Script to install cert auth to trust store
• Curl commands missing
  Jira number is DMAAP-1120

DR our daily builds are failing - Frank investigating

New CI - MGMT push has impacts on Casablanca jobs ( not using Alpine ) - is a bug required here? Potentially only effects DR

03/29: MR having issues, CIA team looking into

04/03: Frank from CIA team still working on this

04/08 Just waiting on MR (still working on, CIA story (there responsible)

4/19:  completed update and now testing, but not committed yet

DR Staging Healthcheck issues

04/10: OOM Staging Healthcheck which was failing should be ok, just waiting for that to be merged

04/15: OOM not merging stuff - Brian Freeman has listed containers with issues (staging only?)

Ability to run containers as non-root user

No more running as root on containers

Proposed Updates to Release Templates (Dublin) - Security Questions

Is it required for M2 ?

02/11 Doesn't need to be implemented by M2, nice to have for M4. (Try to plan for M4).

Here's the link to the Jira where applications are being asked to update their impact and concerns regarding this item.

02/13: Dublin timeframe - Not mandatory, if possible complete, if not have a backlog item for El Alto

Mandar to update ticket

02/15: ticket updated to work on this in El Alto - Mandar to ask reporter how do we interpret this ticket?

  created ticket in backlog

02/27: Is this related to Alpine solution - not clear

New requirement : Wanted position to be part of Dublin

03/04: Action to start investigating - how will this be verified - Mandar

03/08: krzysztof to write script to enable verifcation -mandar to chase up

"Actually the script can be merged into oneliner:

kubectl --namespace=onap get pods | tail -n +2 | awk '{print $1}' |\ xargs -d \\n -I % kubectl --namespace=onap exec '%' -- /bin/sh -c \ 'ps aux | sed "s/^/%\t/"' | tee ps-all.txt

What it does it just exec into every pod and list processes that runs as a root. In general, if you run container using docker you can use user namespaces and match a root user inside the container to any uid on the host that you want but according kubernetes doc user namespace are not supported which effectively means that if sth runs as a root inside the container it is also a root on a host (just certain capabilities may be

dropped)

Best regards,

--

Krzysztof Opasiak"

03/13: Mail forward onto team

03/20: what is required, when is it achievable

03/29: Required by M4 (think its delayed to 11/4?)

04/01: work in progress for DR

Code is Merged for DR - https://gerrit.onap.org/r/#/c/83879/

MR - code is ready (images need to be released)

04/12 - Dom working on BC to have this functionality

04/17: Done for BC. Need to check if there are implications in oom deployment

04/24: This has been released

Component Creation - Jira

01/22: Component creation in Jira for DMaaP project:

I do not posses "Project" admin rights in JIRA, I only posses DMaaP board rights

04/17: Backlog complete with prefixes per component

• [BC]
• [MR]
• [DR]
• [DMAAP] - Encapsulates all components as effected

Only outstanding thing is

• sub-tasks
• We need to continue to implement the prefix going forward

M4 Checklist

This is a reminder that we need to review/discuss about the M4 checklist here

04/17:

04/26: Patch into Casablanca, level of difficulty to be determined if this is a possibility

An alternative is if there is a way to get workarounds/troubleshooting into the docs

04/26: Committer status for Fiachra & Mandar

04/29: Forwarded on to Catherine, has to go through Ram

El alto - 3 month cycle, further details needed

no more details on this

Global jjb vs nexus staging for El Alto - more details to follow

PTL 2019-04-29

5/01: Need to discuss for El Alto

Default feed/sub creation

post install - default feed and default subscription (to be reviewed at Monday call)

04/15: Dependency on AAF being up and running (if so all is good)

03/15: All to review and report back

ARC DMaaP Component Description - Dublin

03/15: 55% code coverage target is for M4

03/20: code and line coverage to be > 55%

04/01: DR approx at 60%

Mandar working on code coverage for MR

04/03: DmaaP client  and BC (52%) is only outstanding concern

04/10: Dom going to work on

04/12: Overall coverage is > 55%

Casablanca release notes - only points towards MR

Randa to get back to Mandar

04/15: All is good here and merged

CII Badging, DR at 89% silver, mail sent to BC and MR

04/15: All components over 80%

04/01 - Helm deploy fail when DMaaP deploy on own - only deploying bc post install

Sunil has seen timeout issue - DOM looking into how best to facilitate this

recommend to use commit shared above in https://gerrit.onap.org/r/#/c/83671/

Tested the above patch (WITH AAF ENABLED) and deployed successfully.

04/03: All is ok once tested with Patch above

if one job doesn't finish others might not run

BC will allow others to provisioning topics/feed during run time

Mirror Maker crash loop - 500 error

04/08: Issue not reproduced. If item has not been reproduced item will be closed

04/10: Bug in topic provisioner - fixed and merged

Latest jar has been tested with no issue

Anyone wishing to make changes to OOM in relation to DMaaP should discuss with DMaaP team in advance.

Similar tasks been worked on in parallel (duplication of effort) - COLLABORATION is the key

Sunil has his commit up, PM mapper guys have OOM changes

04/08 ITEM CLOSED

3/18: New ask from security team. HTTP ports should not be exposed for any of the components. Only HTTPS ports should be exposed. Action for all to remove exposure of component's HTTP port by M4-Dublin

03/20: Conflicts with OOM request to have ability to disable TLS

Need Mandar to get clarity on which way to turn - seems like without a good certificate solution we are shooting ourselves in the foot - Dom to ask his security contact

3/26: clarification from Security Team:

"Dom, I had a conversation with Amy this morning on this topic. Here are some key points from our conversation.

• • TLS must be enabled by default out of the box, but can be disabled for testing purposes.
  • Any port exposed outside of a POD should be protected by default by TLS. This includes communication between PODs.
  • Communications between containers within a POD do not need to be protected by TLS. How else would ISTIO work?
  • This isn’t actually a new ask, as the issue was raised in the Casablanca and Beijing releases. It’s just become more important.

Amy, care to add anything?

                Tony"

3/27: DMaaP team thinks we should be compliant with this because:

• TLS is supported by all components
• non-TLS ports could be disabled by an OOM flag. TBD
  Mandar to follow up w TSC

03/29: Mandar to ask Amy directly

04/01: Awaiting response from Amy - Can we use the overwrite file (flag bullet point above)

is the proposal to have "non-TLS ports could be disabled by an OOM flag. TBD" by default and have overwrite file to enable it

Mandar to inquire about DMaaP ports and downstream apps that use DMaaP

04/03: still no response - mandar will inquire later

04/08 Amy from security is fine with the flag (Not required for Dublin)

flag is to be named "allow_http" set to "false" by default.

Fiarchra to created Jira and close this point.

02/27: might have a solution for this part - mariadb sub requirement under dr-prov, issue in relation to helm with regard this

OOM meeting - 3-4 to discuss further

03/01: Mike to attend on Monday

03/04: Mike is holidays week 11-12, need to ensure Conor commit gets merged

Dom & Sunil have changes to follow suit after Conor's merge

03/06: James Mckinder (OOM team) having issues - potentially - not cleaned correctly - to be discussed at OOM meeting

03/08: Code has been merged to facilitate component deployment in diff sites

release name has release and component name in them - this is causing an issue for configMap at DMAAP level 

helm install vs helm deploy work differently - potential a bug in helm deploy

• can disable components to only deploy individual components (aids testing greatly)
• component changes: helm variables expand are getting long, e.g. filebeat configmap for datarouter at dmapp resource level - component expands out to include component name - potentially move filebeat down into relevant level (keep all dmaap components independent - )

03/15: Dom has done some great work on post install script - plan is to get the components and discuss plan going forward

03/20: Patch set 4, awaiting a MERGE once merged Sunil has follow up changes - mirror maker

03/22: Need to confirm if BC is working after merge (aaf permissions?)

03/29: Commit up for review to resolve 03/22 issue

Dom trying to release 1.0.5 -

04/01 - Image released and updated -https://gerrit.onap.org/r/#/c/83671/ should resolve OOM issue (Code Merged)

04/08: ITEM CLOSED

11/21: How to handle registration / provisioning of a new dr-node instance to an existing and/or "edge" DR deployment.

Dominic Lunanuova recommended to use bus-controller api. Existing script in OOM BusController deploy.

Is there a k8s way to do this "registration / unregistration"?

12/5: Deploy DR node, thinking on how you register to the API of BusController -

12/12: Similar issues across ONAP teams - continue to discuss with OOM team

DMAAP-534 

01/28: Mail thread in relation to this - should DR move to CADI to incorporate AAF roles (good idea!)

Sunil might run through on Fri

02/01: CADI - Steps sent onto Mariusz

Role based access will remain under AAF

02/04: DR looking into integrating AAF for provisioning via BC

02/06: Fiachra in discussions with Sunil to resolve and need to contact AAF but we already have certificates

02/11: Dom to ask Roman to attend to explain ECOMP strategy for Publisher api access. These changes could be ported back to DR code base.

02/13: Internally Romans team have done it for both prov and publish API - Roman willing to forward on the solution it back to ONAP

DR Team will implement it

02/15: Prashant - need to structure -walkthrough from very high level

02/20: Dom to track down files as per requested

02/22: All files sent - team to continue implementation

Dom to see if he can get Prashant to have chat with Fiachra

02/27: Meetings proceeded - keeping open until implementation complete

03/01: Work ongoing -

03/08: testing on windriver environment - take a call offline

03/13: Mail chain - Still blocked (Sunil to assist after call)

03/15: Issue resolved, so progress being made - mail chain

03/15 Fiachra to request port for dr-node

and Sunil to request AAF team to take snapshot of permissions/roles

03/20: blocked - cannot add prov id to dr-admin role- similar issues to meeting call last week (identity itself is missing)

(use aaf deployed within helm charts - )

What do we need to do now??? Sunil to send mail need AAF team to reset up test environment

03/22: Plan is to have this disabled by default

Can be enabled for integration testing - then whenever we satisfied we can set it to enabled by default

windriver image to be taken next week sometime!!

Impact on BC to be conveyed early next week

How to document this  - DR API and equivalent in BC API

03/25: Cadi enabled for MR by default, use same variable name across all DMaaP.

03/29: Code in for review - Plan is to have it disabled by default for this release (Ability to turn on if desired -aafID passed will dictate its on) - Impacts on Bus controller

Danger: Provisoner tries to use AAF, DR off, BC on - ?

Risk: AAF environment req'd (More stable now with ability to deploy aaf locally)

04/01: Code review in process and docs in progress ( Bc updates also reqd)

04/03: Code merged/doc up for review/OOM code not merged

04/08 ITEM CLOSED.

What is the purpose of the "groups" attribute in a subscriber?

10/22: Matt/Kim to investigate

10/24: mechanism to associate many subs to a single group. Matt to send description.

11/21: Fiachra observes that this looks more like some additional authorization method. Good to get confirmation from Matt/Kim on this.

11/28: Roman - confirms auth for sub/feed modification.

Use of HTTPS in dev/test environments. See Fiacha's email.

"Quick question on how we are testing the DR flows.

This only applies to local deploy & test. Robot CSIT would require investigation on how to implement TLS.

At present we default to "--insecure" curl option using the "-k" flag.

Is this valid or should we be using some sort of TLS auth?

We can export the IntermediateCA from the cert chain and pass it via the "--cacert" flag for curl.

Should this be the default way of testing DR if all end users will be required to use TLS?

//Fiachra"

10/31: Dom: http is convenient for dev/test, but at least CSIT should confirm that TLS is enabled on all API endpoints.

11/21: Plan to implement TLS in csit. Investigate how DR clients are using CA certs at present.

12/10: DR working on this to date - will follow up with issue, publish upon resolution

01/23 : Code up for review to test in next few days

02/13: Code up for review - remove CSIT test from Casablanca

11/12: As we consider scaling, are we defining any true load testing scenarios? And what are the tools we are using?

11/26: Sunil - JMeter used as test tool.

11/28: Work ongoing for k8s scaling in Dublin.

11/21: Look out for email about participating in Dublin Release for M0 milestone. Bhanu to inquire w PTL team since she might be on vacation.

12/5: Bhanu to follow up and send mail to register interest in participation

12/10: Projects already involved - assumed that they want to continue

11/21: Follow up on committer promotion requests: Sunil, Dom, Connor

Committer Promotion Request - Dominic Lunanuova

Committer Promotion Request - Conor Ward

Sunil Unnava Committer Promotion Request for [DMaaP]

Committer Promotion Request - Migdal Marcin

12/5: Need to make existing committers inactive

12/10: Follow up with Ram

12/12: Ram working on solution

01/07: Bhanu to follow up

01/14: Voting process - Ram to remove some old committers but still will be available

01/16: Working through some open items - Approval for some

01/21: No change in gerritt status - can we follow up on this

01/23: This might be sorted for Conor/Dom/Sunil - others have request in

01/28: Mandar to follow up with Ram

02/13: Election is complete, Mandar to follow up with TSC

11/21: Toby: PM Mapper desires a subscriber delivery function similar to kafka consumer groups. i.e. multiple subs to same feed - only 1 gets delivery. How to implement this? (Is group attribute useful?)

11/28:

DR statefulness

11/21: Reviewing requirements for statefulness of multiple components. For DR, files in transit need to persist.

11/26: Preliminary plan is to use StatefulSet in K8s.

11/28 StatefulSet with Headless service for DR in Dublin

11/26: Confirm OOM K8s deployment strategy. Single cluster?

11/28: Single K8s cluster deployment confirmed.

11/28: Check with ONAP sprint timings and should we sync with that?

12/10: Tom to chase up

12/03: PM Mapper acknowledgment - mail from DOM (a hack - can we recommend alternative)

DR not originally designed for this functionality

12/10: New API ("Privileged") - Private
Document proposal !! Mark to send around pictorial

Whose impacted by this !

12/12: Alternative soln proposed - need to investigate and get apps on board

01/09: assumption dcae ms shall have not persistence - false assumption DCAE team

01/14: individual DCAE components have some persistence, but state/persistence across components  it will not have

Option B is the agreed plan going forward

12/10: New bridge needed asap for DMaaP to open up discussions to all community

01/07: Follow up with Catherin lefevre

01/14: Kenny Paul Zoom Bridge working on it

01/16: 1 slot - available (Bhanu to update time slot)

01/21: Mon/Wed/Fri - see above logisitcs for times or Community Meetings & Calendar

Compressed/decompressed - not a new service ? Whose impacted by this

extend provisioning API (optional parameter) -  only support gzip decompression

12/12: more value outboard the service going forward

17/12: data quality checks - trying to avoid

solution to think long term

01/14: For now, Implement as part of DR functionality whilst aligning with AT&T and to ensure it is extensible & ease of extraction

Heat vs Helm? can we remove the heat discussion altogether

Tom to get back

02/13: Catherine confirmed OOM is only way forward in Dublin

Remove any tagged stories/tasks in casablanca/beijing release - if important enough we can tag with Dublin

status != Closed AND status != Done AND project != "Sandbox Project" AND project != CI-Management AND project != "ONAP TSC" AND project = DMAAP and fixVersion != "Dublin Release"  ORDER BY priority DESC, updated DESC

Sunil & Bhanu to take off line for now - in relation to MR

Windriver environment - need to inform to Stephen Gooch if we require access to these environments prior to end of week

01/28: Mandar sent mail

01/22: No permission to view security vulnerabilities in relation to

https://jenkins.onap.org/view/CLM/job/dmaap-datarouter-maven-clm-master/

Solution in Casablanca was to get Dom/Sunil to go walk us through each vulnerability detected by CLM scan

01/23: Conor now has permission to access these scans - so no longer an issue

Should we create a ticket - for BC to sync with MR/DR (an enhancement)

DR AAF Certificates expired - DR broken in Casablanca now.

Possibly generate new certs.

02/22: Johnaton to supply fresh ones

02/27: Certs updated and merged for Dublin & Casablanca

Need to get new artifacts released per release - mail sent to Ram/Mandar

03/01: Artifacts released - Code up and awaiting merge and release documents updates needed

03/04: Master - code needs to be merged in OOM

Casablanca - Process (Post maintenance release process to be determined) to get change into this release - 3.0.1 tag is already created - due to be discussed at PTL meeting 11 Mar 2019 

----------

Dmaap 1066 follow up to see if related - replica DB failed to come up (Closed)

Dmaap 1076

03/15: An overall ticket - to bundle all changes in together - Mandar to source ticket (power point in relation to release?)

03:18: updates from Mandar:
PPT: (TSC policy for maintenance releases)
https://jira.onap.org/secure/attachment/13434/Prosal%20for%20a%20TSC%20Policy%20on%20ONAP%20Release%20Maintenance-pa5.pptx

do we need to support the previous releases - Mandar to follow up

01/30: Catherine to bring up with TSC - Upgrade tickets to Casablanca (NO is the answer for now)

02/13: Docker Hub - Paul to send on mail

02/15: PTL to create own id, Mandar has created

ONAP - discuss mail from Jessica

Need to change Docker tags - all

02/22: Gerry in talks with Sunil - might need to alter approach if fabric plugin is way forward

02/22 Gerry to stick with spofity plugin

02/27: Parent pom mightn't be possible with multi component modules - still investigating

03/01: Work ongoing

03/04: Commit up for review

03/06: verification ongoing

03/08: closing as been tracked by aplin vs ubuntu discussion with CIA team

03/04: Need to try and arrange a meeting with Mike Elliot and Sunil to represent from a MR point, Mandar to see if Friday is suitable -  see what happens at today's meeting

03/13: Meeting happened - operational reqs - doesn't make much sense

they will introduce Jira tickets - we will review and if we agree then we will proceed *El Alto timeframe

All changes related to this cert expired issue should be committed before 3/25. Casablanca 3.0.2 will be released on 3/25.

03/22: awaiting for merge to solution

03/29: Bulk PM use case verified on Casablanca wind river environment

04/01: Release new image with cert of 12 month duration - working on this

04/03: Waiting on OOM review to sign off on this

04/05: Code merged in OOM.

New image is needed to be released, Dom to secure via Jessica with PTL +2