LOG Meeting Minutes 2019-03-05
- Michael O'Brien
Use just for today - real meeting busted - zoom.us/j/7939937123
Meeting every 2 weeks at 1100 EST Tue - https://zoom.us/j/519971638
https://lists.onap.org/g/onap-discuss/topics
http://onap-integration.eastus.cloudapp.azure.com:3000/group/onap-integration
https://jira.onap.org/secure/RapidBoard.jspa?rapidView=143&view=planning.nodetail&epics=visible
Attendees
Luke Parker , Prudence Au , Michael O'Brien , Sanjay, Lorraine A. Welch
Agenda
- Meetings every 2 weeks now - next meeting the 19th
- Need to accomidate Luke Parker - who is up at 3 am right now in Australia - will be 1 in the morning until April - then 2am
- ideas: Better to be up at 1-3 than 8am - keep at 11 for now
- For NA - stays at 11 - but I will verify if we are on UTC
- DST is coming this weekend - we should agree on the meeting time - Luke does not go DST for 3 more weeks
- Michael: status no progress on logging related jiras involving dublin scope (code/tech) last week
- Passed M2
- Discussed in TSC and PTL meets - issue related to manifest vs k8s values.yaml overrides for casablanca and CMR releases -
- TSC-86 - Getting issue details... STATUS
- TSC 2019-02-28
- OOM-1560 - Getting issue details... STATUS
- In architecture review for M3 - (Addressed comments) Logging Dublin M3 Architecture Review
- Working more for OOM lately - Created RKE script to replace rancher - pending work is to test HA - https://gerrit.onap.org/r/#/c/79067/ for OOM-1670 - Getting issue details... STATUS
- Assisting with ONS April demo prep - 2 or more ONAP installations working together - one as an edge cloud
- Focused on onap devops - primarily for 2 customers deploying ONAP
- 5G training next week
- April 3-5 ONS work
- k8s infrastructure and
- Logging will be a part of the demo - we will show the kibana logs during a vFW operation in an i-frame for the master and edge ONAP deploys
- PTL position is open - let me know if anyone would like to take it.
- Also quickly discussed idea of rolling LOG into OOM - should revisit
- Pending committer for pomba - Committer Request for Trevor Tait and Pierre Rioux - Committer Request for [Logging-Analytics] - will send out reviews once stats are completed - TSC review will be needed
- TODO I need to check MSB issues on onap-discuss -
- Spondon - import the dashboard - fix the one on the wiki - Lorraine A. Welch export to json first ONAP Culprit Locator
- Lorraine A. Welch question about changing the index for the db (Grafana for example) - @Timestamp timestamp for example - TODO - merge the 2 indexes -
- Q) Sanjay: (opentracking related - picking up errors on stdout (err, out)
- how:
- (also review from 2 weeks ago - Lorraine A. Welch System.out standardout - review in terms of not using a hardcoded file appender - in terms of syslogs
- see https://12factor.net/logs)
- log retention on pod failure - those not mapped to PVs - including the current EmptyDir link between the sidecar - etherial PV - logs only end up in the elk stack
- Look into PV/PVC structure for log retention instead of the default emptydir
- https://git.onap.org/logging-analytics/tree/reference/logging-kubernetes/logdemonode/charts/logdemonode/templates/deployment.yaml#n114
- see PV/PVC example .
logs can go to the /dockerdatanfs) - https://git.onap.org/oom/tree/kubernetes/portal/charts/portal-mariadb/templates/deployment.yaml#n84
- Topic: File retention
- Q: rotation schedule, size of retention - what role does the PV on the ELK stack play compared to the individual source pod PVs
- @Luke's example - discuss with Mike Elliott - review a pending upstream contribution
- https://git.onap.org/logging-analytics/tree/reference/provider/helm/logback/chart/resources/logback.xml#n10
- variables set in
- https://git.onap.org/logging-analytics/tree/reference/provider/helm/logback/chart/values.yaml#n14'
- Review any issues with multiple providers in SDNC and APPC
- Lorraine A. Welch check https://www.loomsystems.com/blog/single-post/2017/01/30/a-comparison-of-fluentd-vs-logstash-log-collector
Attendees
Items
| PRI | Title | Responsible | Status OPEN DONE | In Dublin | Last Worked on | Start | Notes |
|---|---|---|---|---|---|---|---|
| Security Vulnerability template | Ongoing | IN | 20190122 | ||||
| M1 template | DONE | IN | 20190124 | 20190122 | |||
| ONS NA 2019 April Talk proposal | DONE | IN | 20190122 | pending 20190208 decision LOG-947 - Getting issue details... STATUS | |||
| Use manifest generation over raw oom values.yaml docker image tag names | DONE | IN | 20190124 | 20190117 | pending documentation in RTD Team, In the TSC it was decided to treat the diff between oom and the manifest by always running the manifest generated yaml in your deployments – you will not need to do this for master work – just for Casablanca and RC0-2 work
Working out the details in https://jira.onap.org/browse/LOG-929 /michael | ||
| S3P Logging compliance TSC/PTL | IN PROGRESS | IN | 20190115 | 20190114 | |||
El-Alto 1.4 logging spec change - plan only todo merge with Dave's below | IN PROGRESS | IN | 20190122 | ||||
Dublin Scope Planning | DONE | IN | 20190124 | LOG-707 - Getting issue details... STATUS | |||
| RTD documentation | OPEN | IN | 20180129 | Attending Thu 1130-1230 meets | |||
restart log4j format and files example | IN PROGRESS | IN | 201901115 | 20190108 | https://gerrit.onap.org/r/#/c/62405/ for LOG-630 - Getting issue details... STATUS and LOG-178 - Getting issue details... STATUS | ||
| Work with portal/sdk library | Michael O'Brien | IN PROGRESS | IN | 20190129 | 20190115 | Update: 20190129 - Existing eclipse environ for the RI being retrofitted At the pom stage bringing in the jar via portal/sdk in use by aai, dmaap, sdk, vid (vid link into so maybe?) <groupId>org.onap.portal.sdk</groupId> epic LOG-600 - Getting issue details... STATUS Jira PORTAL-348 - Getting issue details... STATUS review investigation in Log Streaming Compliance and API#ExistingLibraryResearch\ Luke Parker discussion need to use the portal library in an initiating project for tx processing working likely with the SO team - via the work we are doing for them in https://gerrit.onap.org/r/#/c/69947/ (check the original spec - ODL specific - check appc/sdnc use of ccsdk) | |
| New Committers | OPEN | 20190115 | We have room for 2-5 committers and will be reviewing the list Logging Enhancements Project Proposal#KeyProjectFacts add your details to Logging Committer Promotion Requests 20190129 status - waiting on contributor documentation from each contributor | ||||
| OPNFV/ONAP Paris | DONE | 20190108 | https://ddfplugfest19.sched.com/ Tue-Thu Clover Gambia on prior https://zoom.us/j/115579117 - 7 hours ahead https://ddfplugfest19.sched.com/event/K1Gy/opnfv-clover-utilizing-cloud-native-technologies-for-nfv | ||||
| Security badging | IN PROGRESS | IN | 20190129 | Need to restart this | |||
| Security Vulnerabilities | IN PROGRESS | IN | 20190129 | lower - but for M4 | |||
s3p Secure https endpoints LOG + POMBA | for djhunt | OPEN | IN | Discussion on whether we need to lock down the nodeport exposed ports Can key off POMBA work already done todo: get s3p page | |||
Format compliance - working with AAI team + perf | IN PROGRESS | IN | 20190115 | 20181101 |
https://gerrit.onap.org/r/#/c/75702/ (+) 20190109 #6 on 2018-12-20 AAI Developers Meeting around LOG-376 - Getting issue details... STATUS Discussion with @Sanjay Agraharam and [~pau2882] on checking how cassandra is running on the vm and if debug levels are on should be verified use labelling to split aai-cs and ls - no DaemonSet Michael O'Brien to reduce core count for ls to 1 from 3 LOG-915 - Getting issue details... STATUS edited 2019-01-10 AAI Developers Meeting for the 10th | ||
| AAI team - 2 types of logging AOP/non-AOP | OPEN | IN | 20181101 | #22 on 2018-12-20 AAI Developers Meeting | |||
| Logging requests from Vendors | OPEN | LOG-877 - Getting issue details... STATUS LOG-876 - Getting issue details... STATUS #15,19 and 37 on SP priorities for Dublin | |||||
| LOG Streaming compliance | IN PROGRESS | IN | Log Streaming Compliance and API LOG-487 - Getting issue details... STATUS LOG-487 - Getting issue details... STATUS LOG-852 - Getting issue details... STATUS and | ||||
opentracing via | IN (planning/POC for sure) | 20190123 | @Sanjay discuss integration - out of band processing - LOG-104 - Getting issue details... STATUS see zipkin arch https://zipkin.io/pages/architecture.html possibly tie both as a client of es ? Tie in to ONS NA 2019 April demo booth for LF | ||||
| discussion - remove | 20190108 | discuss tick/tock logging spec behaviour - cassablanca implemented in dublin, dublin implemented in elalto
| |||||
| Log Checker | OUT | 20190109 | MIke to review with Horace | ||||
| Search Guard | OPEN | Maybe | 20180109 | LOG-494 - Getting issue details... STATUS | |||
| spec changes for Dublin | IN (planning) | 20190109 | 20190109 | Dublin spec changes for Elalto environment name release name check mail for reply Michael O'Brien Prudence Au proposal of renaming the log file name itself for the release ie: 3.0.0-ONAP - will discuss later for next week | |||
Cluster logging behaviour S3P | IN | server name in clustered environments - I will add the details and the Jira right after this meet | |||||
LOG ELK stack indexing/dashboards with Prometheus below | OPEN | IN | 20190123 | ||||
Casablanca 3.0.1 work until 10th Jan Including POMBA | DONE | 20190122 | 20190113 | LOG-913 - Getting issue details... STATUS revert Jira for data-router off TSC-92 - pending merge of https://gerrit.onap.org/r/#/c/75999/ | |||
LOG openlab tenant devops cluster creation/testing | Done pending vFW | IN | We have 2 clusters a 1+4 and 1+13 used for testing deployments and running the vFW | ||||
| Wiki edits, RTD review | OPEN | IN | Requiring Updates, Merges or Marked Deprecated | ||||
| Metric Streaming and Prometheus | IN PROGRESS | IN | 20181207 | LOG-911 - Getting issue details... STATUS - experimental chart on http://secure.solar:30000/graph LOG-773 - Getting issue details... STATUS LOG-861 - Getting issue details... STATUS work with Vaibhav Chopra OOM-1504 - Getting issue details... STATUS @Sanjay - note the prom chart assumes a k8s environment - what about bare metal | |||
Finish SO filebeat additions | IN PROGRESS | IN | 20181207 | SO-1110 - Getting issue details... STATUS | |||
| Finish LOG common charts | OPEN | OUT to El-alto | 20190123 | 20181207 | James MacNider - bring in Prianka's common eLK charts and use them in Clamp, LOG, SDC, POMBA https://gerrit.onap.org/r/#/c/64767/ OOM-1276 - Getting issue details... STATUS rever to El-alto under LOG-936 - Getting issue details... STATUS | ||
| Team Members Thank you and review | OPEN | IN |
| ||||
| del | Review last 4 weeks since | IN PROGRESS | |||||
| TSC/PTL meet actions | IN PROGRESS | ||||||
| OOM transfer chart ownership to teams LOG is part of poc | IN PROGRESS | IN | 20190107 | Starting - will have a training session - will send out any meetings to onap-discuss We may have the same symlink repo folder like we do for doc Last discussed TSC 20180109 | |||
| OOM Deployment priority base platform includes LOG | IN PROGRESS | OUT todo review with Mike Elliott | 20190123 | 20181207 | Q2) priority of system level containers like the ELK stack - OOM has a common services JIRA - DMaaP, AAF - TODO get JIRA - make sure log is in this! There is a cd.sh retrofit that sequences the pods in order for deployment stability - this will be phased out when tiered deployment comes in LOG-898 - Getting issue details... STATUS LOG-326 - Getting issue details... STATUS https://gerrit.onap.org/r/#/c/75422/ via ONAP Development#WorkingwithJSONPath DCAEGEN2-1067 - Getting issue details... STATUS | ||
| k8s manifest or oom values.yaml for docker tags - truth | IN PROGRESS | 20190123 | TSC-86 - Getting issue details... STATUS TODO: paste TSC review - manifest is truth RTD doc link to run the script to get the yaml override | ||||
| Nexus3 routing slowdown | DONE | 20181222 | |||||
| LOG compliance diagram/exercise | @Sanjay Agraharam | IN PROGRESS | 20181205 | Log Streaming Compliance and API part of prometheus work now Sanjay - diagram FB must be split between using AOP and AOP+spec compliant - only this one should be green | |||
| Ongoing | |||||||
| CI/CD pipeline TSC poc | IN PROGRESS | 20180107 | TSC-25 - Getting issue details... STATUS LOG is part of this poc |
20190115 - casablanca cherry pick in queue logstack 5 to 3 and 1