Page Comparison

DR transaction logs approach

Mail sent form Dom

Just wanted to give you heads up on a change to DR made to AT&T version: transaction log archiving moved from MySQL to ELK.

Reasons:

• Mysql couldn’t keep up with daily volume
• Offload transaction archives where space can be managed independently
• Common GUI (kabana) for examining logs
• DR LogAPI can still interface to ElasticSearch API

I recall that you already were doing something with filebeat in the ONAP version, but I have to admit that I haven’t followed it closely.

So, perhaps you are already applying this approach to DR transaction logs?

But if not, I think we should consider the approach chosen by AT&T so that the DB is only used for provisioning data.

04/24:

why?

Data up to 24 hrs max

Solution: Create ticket to track/implement this (EL Alto)

M4 Checklist

This is a reminder that we need to review/discuss about the M4 checklist here

04/17:

• Mandar has updated the M4 freeze doc with non-root and CSIT showing complete.
• Need a plan to complete swagger docs before RC2.

  Jira Legacy
  server System Jira columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key DMAAP-922

• What other docs are a requirement for release (like readthedocs etc)?

3/18: New ask from security team. HTTP ports should not be exposed for any of the components. Only HTTPS ports should be exposed. Action for all to remove exposure of component's HTTP port by M4-Dublin

03/20: Conflicts with OOM request to have ability to disable TLS

Need Mandar to get clarity on which way to turn - seems like without a good certificate solution we are shooting ourselves in the foot - Dom to ask his security contact

3/26: clarification from Security Team:

"Dom, I had a conversation with Amy this morning on this topic. Here are some key points from our conversation.

• • TLS must be enabled by default out of the box, but can be disabled for testing purposes.
  • Any port exposed outside of a POD should be protected by default by TLS. This includes communication between PODs.
  • Communications between containers within a POD do not need to be protected by TLS. How else would ISTIO work?
  • This isn’t actually a new ask, as the issue was raised in the Casablanca and Beijing releases. It’s just become more important.

Amy, care to add anything?

                Tony"

3/27: DMaaP team thinks we should be compliant with this because:

• TLS is supported by all components
• non-TLS ports could be disabled by an OOM flag. TBD
  Mandar to follow up w TSC

03/29: Mandar to ask Amy directly

04/01: Awaiting response from Amy - Can we use the overwrite file (flag bullet point above)

is the proposal to have "non-TLS ports could be disabled by an OOM flag. TBD" by default and have overwrite file to enable it

Mandar to inquire about DMaaP ports and downstream apps that use DMaaP

04/03: still no response - mandar will inquire later

04/08 Amy from security is fine with the flag (Not required for Dublin)

flag is to be named "allow_http" set to "false" by default.

Fiarchra to created Jira and close this point.

04/08: ITEM CLOSED

DR AAF Certificates expired - DR broken in Casablanca now.

Possibly generate new certs.

02/22: Johnaton to supply fresh ones

02/27: Certs updated and merged for Dublin & Casablanca

Need to get new artifacts released per release - mail sent to Ram/Mandar

03/01: Artifacts released - Code up and awaiting merge and release documents updates needed

03/04: Master - code needs to be merged in OOM

Casablanca - Process (Post maintenance release process to be determined) to get change into this release - 3.0.1 tag is already created - due to be discussed at PTL meeting 11 Mar 2019 

----------

Dmaap 1066 follow up to see if related - replica DB failed to come up (Closed)

Dmaap 1076

03/15: An overall ticket - to bundle all changes in together - Mandar to source ticket (power point in relation to release?)

03:18: updates from Mandar:
PPT: (TSC policy for maintenance releases)
https://jira.onap.org/secure/attachment/13434/Prosal%20for%20a%20TSC%20Policy%20on%20ONAP%20Release%20Maintenance-pa5.pptx

Tickets:

All changes related to this cert expired issue should be committed before 3/25. Casablanca 3.0.2 will be released on 3/25.

03/22: awaiting for merge to solution

03/29: Bulk PM use case verified on Casablanca wind river environment

04/01: Release new image with cert of 12 month duration - working on this

04/03: Waiting on OOM review to sign off on this

04/05: Code merged in OOM.

02/13: Dynamic handling of feed creation/subscribing to feed

BC API has a 2 step reg processes

query on feed name

determine feed ID

add subscriber to feed ID

an enhancement: support optional feed name in the addition to subscription API

02/15: Dom to create Jira Ticket for enhancement

02/20: this ticket, no impact on DR

Ability to run containers as non-root user

No more running as root on containers

Proposed Updates to Release Templates (Dublin) - Security Questions

Is it required for M2 ?

02/11 Doesn't need to be implemented by M2, nice to have for M4. (Try to plan for M4).

Here's the link to the Jira where applications are being asked to update their impact and concerns regarding this item.

02/13: Dublin timeframe - Not mandatory, if possible complete, if not have a backlog item for El Alto

Mandar to update ticket

02/15: ticket updated to work on this in El Alto - Mandar to ask reporter how do we interpret this ticket?

  created ticket in backlog

02/27: Is this related to Alpine solution - not clear

New requirement : Wanted position to be part of Dublin

03/04: Action to start investigating - how will this be verified - Mandar

03/08: krzysztof to write script to enable verifcation -mandar to chase up

"Actually the script can be merged into oneliner:

kubectl --namespace=onap get pods | tail -n +2 | awk '{print $1}' |\ xargs -d \\n -I % kubectl --namespace=onap exec '%' -- /bin/sh -c \ 'ps aux | sed "s/^/%\t/"' | tee ps-all.txt

What it does it just exec into every pod and list processes that runs as a root. In general, if you run container using docker you can use user namespaces and match a root user inside the container to any uid on the host that you want but according kubernetes doc user namespace are not supported which effectively means that if sth runs as a root inside the container it is also a root on a host (just certain capabilities may be

dropped)

Best regards,

--

Krzysztof Opasiak"

03/13: Mail forward onto team

03/20: what is required, when is it achievable

03/29: Required by M4 (think its delayed to 11/4?)

04/01: work in progress for DR

Code is Merged for DR - https://gerrit.onap.org/r/#/c/83879/

MR - code is ready (images need to be released)

04/12 - Dom working on BC to have this functionality

04/17: Done for BC. Need to check if there are implications in oom deployment

04/24: This has been released

02/04: Discussion on Edge deployment commitments.

Centralized ONAP Deployment (DMAAP + DCAE)

DCAE k8s deployment - Epic

DFC -

PM Mapper -

DFC + Mapper instantiated at edge, central or both - dependency on DR and MR

Service provider provision topic

Work on it and review progress at M3

02/06: is there a central registry - AAI (Vijay in communication with them)

How do you retrieve edge locations? a label/name is what we need (need to be consistent with DCAE)

Secondary consideration : how do we deploy component in edge and are aware of these - managing HELM charts?

2/8: Dom started to capture approach based on ongoing meetings with OOM: DMaaP Edge Deployment

02/13: Edge Deployment group asked for page to be reviewed with their team - DOM to forward out meeting call to all

02/15: OOM meeting 13 Feb 2019 - Experiment with edge, DMaaP have voluntereed to assist here 

lets wait for answer to whether or not to split out components might be solution going forward

02/20: Mike to discuss further - OOM

02/22: Mike still investigating this topic further

Fiachra to look into why DR is structured the way it is? -see 02/27

OOM losing personnel - CNI in particular to get HA K8s up and running - keep an eye on

02/27: might have a solution for this part - mariadb sub requirement under dr-prov, issue in relation to helm with regard this

OOM meeting - 3-4 to discuss further

03/01: Mike to attend on Monday

03/04: Mike is holidays week 11-12, need to ensure Conor commit gets merged

Dom & Sunil have changes to follow suit after Conor's merge

03/06: James Mckinder (OOM team) having issues - potentially - not cleaned correctly - to be discussed at OOM meeting

03/08: Code has been merged to facilitate component deployment in diff sites

release name has release and component name in them - this is causing an issue for configMap at DMAAP level 

helm install vs helm deploy work differently - potential a bug in helm deploy

• can disable components to only deploy individual components (aids testing greatly)
• component changes: helm variables expand are getting long, e.g. filebeat configmap for datarouter at dmapp resource level - component expands out to include component name - potentially move filebeat down into relevant level (keep all dmaap components independent - )

03/15: Dom has done some great work on post install script - plan is to get the components and discuss plan going forward

03/20: Patch set 4, awaiting a MERGE once merged Sunil has follow up changes - mirror maker

03/22: Need to confirm if BC is working after merge (aaf permissions?)

03/29: Commit up for review to resolve 03/22 issue

Dom trying to release 1.0.5 -

04/01 - Image released and updated -https://gerrit.onap.org/r/#/c/83671/ should resolve OOM issue (Code Merged)

04/08: ITEM CLOSED

Should we create a ticket - for BC to sync with MR/DR (an enhancement)

02/05: Jira created

Swagger implementation?

01/28: Swagger is a requirement for Dublin Release (S3P requirements) ?

What are the expectations here? Whom can we discuss with? - Sofia Wallin,

01/30: Rich Bennett Mail on this - still cryptic - is it Mandatory - requirement in Platform Maturity

02/06: Dom sent mail looking for clarification - Erik has yet to respond?

02/20: Still no word

02/22: Erik to document what is required

02/27: Still nothing from Erik documented

03/01: Tom to chase Erik up

03/08: Erik to documented beginning of next (hopefully)

03/13: Attend documentation this week

03/15: read the docs vs swagger files and what are they exactly looking for

03/29: Erik has code, just needs to merge (tom to chase up)

04/12: https://logs.onap.org/production/vex-yul-ecomp-jenkins-1/doc-master-verify-rtd/4424/html/guides/onap-developer/how-to-use-docs/api-swagger-guide.html

need to create annotations for API's

04/15: JSON for DR is complete, need to figure out where it goes, potentially just a link to download

04/17:

Milestone 1: 17/jan

Jira Epics and user stories need to be created by this date

01/07: Bhanu to chat with PTLs to see about platform maturity

01/16: M1 Postponed by a week I think (see https://lists.onap.org/g/onap-release/topic/29128816 for further details) - new date 24 Jan 2019 

To be reviewed and updated by Monday

Authentication -Tomasz to create new Jira and to cater for basic auth

AAF team to dynamically generate certificates - generic issues Tomasz to cross check with AAF team

Issue in onap-discuss with Casablanca - DR not instantiating correctly.

Workaround posted into onap-discuss.

Investigating cause of issue

01/07: SO having similar issues with MariaDB

01/16 : in contact with OOM team in relation to this

01/21:

01/28: Integration team going to create a new ticket? - 

https://jira.onap.org/browse/DMAAP-1010 ??

02/06: Hoping OOM can assist - issue seems to happen on an AWS platform

Integration team tested against WindRiver

Sunil has seen in his local environment - He will share the link

Going forward we are going to try the timeout solution.

02/11 common mariadb-galera chart now used in oom, testing that.

02/13: Passing in OOM environment last couple of days

Will continue to monitor it to see if it can be reproduced.

02/15: Tom to mail Mike in relation to https://jenkins.onap.org/view/External%20Labs/job/lab-windriver-oom-daily/ failures

02/20: Licence Expired - Johnaton to supply new set- 1048

12/10: Helm Charts OOM location folder?

Central charts  - OOM to decide upon?

Mark Scott to investigate if answer can be given by OOM

01/07: one set of charts with variable(what is this variable) for central vs edge

01/09: What OOM Jira ticket is related to this - post clear comment OOM

01/14: While the timing of 

I think 

DOM created sub-task  

12/10: Marcin to update Node Servlet to improve logging

12/17: Marcin sent on mail and is to create Jira ticket to track changes - 

12/5: Fixing code smells in advance of critical and blocking bugs - can we merge the code?

use caution!!!!! time consuming and not very productive

Bhanu to raise at PTL call to add meaningful code only

Anyone wishing to contribute must be willing to participate in call

Bhanu unavailable after 18th -

01/14: Sonar warning fixes -discussion on ONAP discuss - to be eradicated (need to be part of use case to be involved)

01/16:

DR TLS

10/20: Fiachra initiated an email thread about how Datarouter components might scale, which interfaces require TLS, and how certificates need to be managed.

10/22: Dom suggested we include kubernetes design considerations, and try to identify specific use cases to drive requirements on AAF (assumed CA capability)

10/22: Dom to document AT&T ECOMP conventions for certificates

ECOMP conventions:(assumes VM per component) for SSL certificates:

• • CN = hostname.deploymentDomain
  • SAN = serviceName.deploymentDomain (used for MR and DR Prov servers - clients reference the serviceName, which needs to be provisioned in DNS)
    ....doesn't really help us think through the problem for DR Node.

11/21: TLS solution between DR components and any clients of the API (DCAE Data File Collector, DCAE PM Mapper).

Jonathan Gathman mentions AAF Certificate Manager which is available for auto config in Casablanca. Several strategies available:- Common SAN in x509s, NFS cert store, etc.

Ramprasad Koya / Bhanu Ramesh Recommendation is to involve PTLs in Dublin planning for component security requirements.

12/10: SecCom discussing this

01/09: Tony Hansen - AAF supplier of SSL certs & AAF & Istio how the relationship between them - wait till M2

01/21: greater uncertainty in relation to istio in Dublin release? (Mike Elliot)

01/23: DCAE : At F2F in Paris - istio implemented as a sidecar - OOM don't know how to deploy using k8s

Status quo impact on DMAAP -  if istio is implemented aaf will need to alter

Mandar can u check with TSC committee? Is ISTIO a proposal or mandatory for Dublin? and if so is there any expected impacts

02/01: Not mandatory - POC in progress Istio

02/06: No Dublin Impact

11/21: How to handle registration / provisioning of a new dr-node instance to an existing and/or "edge" DR deployment.

Dominic Lunanuova recommended to use bus-controller api. Existing script in OOM BusController deploy.

Is there a k8s way to do this "registration / unregistration"?

12/5: Deploy DR node, thinking on how you register to the API of BusController -

12/12: Similar issues across ONAP teams - continue to discuss with OOM team

DMAAP-534 

01/28: Mail thread in relation to this - should DR move to CADI to incorporate AAF roles (good idea!)

Sunil might run through on Fri

02/01: CADI - Steps sent onto Mariusz

Role based access will remain under AAF

02/04: DR looking into integrating AAF for provisioning via BC

02/06: Fiachra in discussions with Sunil to resolve and need to contact AAF but we already have certificates

02/11: Dom to ask Roman to attend to explain ECOMP strategy for Publisher api access. These changes could be ported back to DR code base.

02/13: Internally Romans team have done it for both prov and publish API - Roman willing to forward on the solution it back to ONAP

DR Team will implement it

02/15: Prashant - need to structure -walkthrough from very high level

02/20: Dom to track down files as per requested

02/22: All files sent - team to continue implementation

Dom to see if he can get Prashant to have chat with Fiachra

02/27: Meetings proceeded - keeping open until implementation complete

03/01: Work ongoing -

03/08: testing on windriver environment - take a call offline

03/13: Mail chain - Still blocked (Sunil to assist after call)

03/15: Issue resolved, so progress being made - mail chain

03/15 Fiachra to request port for dr-node

and Sunil to request AAF team to take snapshot of permissions/roles

03/20: blocked - cannot add prov id to dr-admin role- similar issues to meeting call last week (identity itself is missing)

(use aaf deployed within helm charts - )

What do we need to do now??? Sunil to send mail need AAF team to reset up test environment

03/22: Plan is to have this disabled by default

Can be enabled for integration testing - then whenever we satisfied we can set it to enabled by default

windriver image to be taken next week sometime!!

Impact on BC to be conveyed early next week

How to document this  - DR API and equivalent in BC API

03/25: Cadi enabled for MR by default, use same variable name across all DMaaP.

03/29: Code in for review - Plan is to have it disabled by default for this release (Ability to turn on if desired -aafID passed will dictate its on) - Impacts on Bus controller

Danger: Provisoner tries to use AAF, DR off, BC on - ?

Risk: AAF environment req'd (More stable now with ability to deploy aaf locally)

04/01: Code review in process and docs in progress ( Bc updates also reqd)

04/03: Code merged/doc up for review/OOM code not merged

4/5: Remaining work items in Jira:

04/08 ITEM CLOSED.

Scaled & Distributed Datarouter Solution.

10/24: Fiachra sent additional questions to OOM team on auto-scaling solution. also, OOM-8 tracking auto-scaling.

11/7: Additional considerations: spool directory persistence, stateful sets, and SSL certificates.

11/26: OOM JIRA for Geo-Rep

11/26: Dom to coordinate with DCAE on Edge deployment

11/28: DCAE intend to deploy at edge for Dublin but still unclear as to how.

12/03: OOM call - follow up call with Mike Elliot next Wed, Single K8 that expands multiple sites

If not deployed in edge will it be in central solely

Requirements per components (DCAE included) - need to compile list

01/07: Single k8s solve multi site - geo-diversity k8s deployment

diff solutions for prov and node

Similar problem for MR

OOM Geo-Red Active-Active via Affinity/AntiAffinity to be reviewed

01/28: Sunil to run through Demo - Friday Slot