Use just for today - real meeting busted - zoom.us/j/7939937123
Meeting every 2 weeks at 1100 EST Tue - https://zoom.us/j/519971638
...
https://jira.onap.org/secure/RapidBoard.jspa?rapidView=143&view=planning.nodetail&epics=visible
Agenda
- Michael: status no progress on logging related jiras involving dublin scope (code/tech) last week
- Passed M2
- Discussed in TSC and PTL meets - issue related to manifest vs k8s values.yaml overrides for casablanca and CMR releases
- In architecture review for M3 - (Addressed comments) Logging Dublin M3 Architecture Review
- Working more for OOM lately - Created RKE script to replace rancher - pending work is to test HA - https://gerrit.onap.org/r/#/c/79067/ for
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key OOM-1670 - Focused on onap devops
- 5G training next week
- PTL position is open - let me know if anyone would like to take it.
- Pending committer for pomba - Committer Request for Trevor Tait and Committer Request for Pierre Rioux - will send out reviews once stats are completed - TSC review will be needed
Attendees
Items
...
Status
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
...
...
...
Status | ||||
---|---|---|---|---|
|
...
...
Status | ||||
---|---|---|---|---|
|
...
pending 20190208 decision
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
...
...
Status | ||||
---|---|---|---|---|
|
...
pending documentation in RTD
Team,
In the TSC it was decided to treat the diff between oom and the manifest by always running the manifest generated yaml in your deployments – you will not need to do this for master work – just for Casablanca and RC0-2 work
Working out the details in
https://jira.onap.org/browse/LOG-929
/michael
...
...
Status | ||||
---|---|---|---|---|
|
...
Log Streaming Compliance and API
discussed at last PTL meet with djhunt
El-Alto 1.4 logging spec change - plan only
todo merge with Dave's below
...
...
Status | ||||
---|---|---|---|---|
|
...
Dublin Scope Planning
...
...
Status colour Blue title Done
...
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key LOG-707
...
Attendees
Luke Parker , Prudence Au , Michael O'Brien , Sanjay, Lorraine A. Welch
Agenda
- Meetings every 2 weeks now - next meeting the 19th
- Need to accomidate Luke Parker - who is up at 3 am right now in Australia - will be 1 in the morning until April - then 2am
- ideas: Better to be up at 1-3 than 8am - keep at 11 for now
- For NA - stays at 11 - but I will verify if we are on UTC
- DST is coming this weekend - we should agree on the meeting time - Luke does not go DST for 3 more weeks
- Michael: status no progress on logging related jiras involving dublin scope (code/tech) last week
- Passed M2
- Discussed in TSC and PTL meets - issue related to manifest vs k8s values.yaml overrides for casablanca and CMR releases -
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key TSC-86 - TSC 2019-02-28
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key OOM-1560
- In architecture review for M3 - (Addressed comments) Logging Dublin M3 Architecture Review
- Working more for OOM lately - Created RKE script to replace rancher - pending work is to test HA - https://gerrit.onap.org/r/#/c/79067/ for
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key OOM-1670 - Assisting with ONS April demo prep - 2 or more ONAP installations working together - one as an edge cloud
- Focused on onap devops - primarily for 2 customers deploying ONAP
- 5G training next week
- April 3-5 ONS work
- k8s infrastructure and
- Logging will be a part of the demo - we will show the kibana logs during a vFW operation in an i-frame for the master and edge ONAP deploys
- PTL position is open - let me know if anyone would like to take it.
- Also quickly discussed idea of rolling LOG into OOM - should revisit
- Pending committer for pomba - Committer Request for Trevor Tait and Pierre Rioux - Committer Request for [Logging-Analytics] - will send out reviews once stats are completed - TSC review will be needed
- TODO I need to check MSB issues on onap-discuss -
- Spondon - import the dashboard - fix the one on the wiki - Lorraine A. Welch export to json first ONAP Culprit Locator
- Lorraine A. Welch question about changing the index for the db (Grafana for example) - @Timestamp timestamp for example - TODO - merge the 2 indexes -
- Q) Sanjay: (opentracking related - picking up errors on stdout (err, out)
- how:
- (also review from 2 weeks ago - Lorraine A. Welch System.out standardout - review in terms of not using a hardcoded file appender - in terms of syslogs
- see https://12factor.net/logs)
- log retention on pod failure - those not mapped to PVs - including the current EmptyDir link between the sidecar - etherial PV - logs only end up in the elk stack
- Look into PV/PVC structure for log retention instead of the default emptydir
- https://git.onap.org/logging-analytics/tree/reference/logging-kubernetes/logdemonode/charts/logdemonode/templates/deployment.yaml#n114
- see PV/PVC example .
logs can go to the /dockerdatanfs) - https://git.onap.org/oom/tree/kubernetes/portal/charts/portal-mariadb/templates/deployment.yaml#n84
- Topic: File retention
- Q: rotation schedule, size of retention - what role does the PV on the ELK stack play compared to the individual source pod PVs
- @Luke's example - discuss with Mike Elliott - review a pending upstream contribution
- https://git.onap.org/logging-analytics/tree/reference/provider/helm/logback/chart/resources/logback.xml#n10
- variables set in
- https://git.onap.org/logging-analytics/tree/reference/provider/helm/logback/chart/values.yaml#n14'
- Review any issues with multiple providers in SDNC and APPC
- Lorraine A. Welch check https://www.loomsystems.com/blog/single-post/2017/01/30/a-comparison-of-fluentd-vs-logstash-log-collector
Attendees
Items
...
PRI | Title | Responsible | Status
| IN | 20180129 | Attending Thu 1130-1230 meetsrestart log4j format and files example |
| Yellow
| In | ProgressDublin | INLast Worked on | 201901115Start | 20190108 |
---|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
pending 20190208 decision
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Log Streaming Compliance and API#DeploymentProfiles
Work with portal/sdk library
|
Status | |
---|---|
|
|
|
Update: 20190129 - Existing eclipse environ for the RI being retrofitted
At the pom stage bringing in the jar via
portal/sdk in use by aai, dmaap, sdk, vid (vid link into so maybe?)
Code Block | ||
---|---|---|
| ||
<groupId>org.onap.portal.sdk</groupId> |
epic
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
review investigation in
Log Streaming Compliance and API#ExistingLibraryResearch\
Luke Parker discussion
need to use the portal library in an initiating project for tx processing
working likely with the SO team - via the work we are doing for them in https://gerrit.onap.org/r/#/c/69947/
(check the original spec - ODL specific - check appc/sdnc use of ccsdk)
Status | ||||
---|---|---|---|---|
|
We have room for 2-5 committers and will be reviewing the list
Logging Enhancements Project Proposal#KeyProjectFacts
add your details to
Logging Committer Promotion Requests
20190129 status - waiting on contributor documentation from each contributor
pending documentation in RTD
Team,
In the TSC it was decided to treat the diff between oom and the manifest by always running the manifest generated yaml in your deployments – you will not need to do this for master work – just for Casablanca and RC0-2 work
Working out the details in
https://jira.onap.org/browse/LOG-929
/michael
Status | ||||
---|---|---|---|---|
|
El-Alto 1.4 logging spec change - plan only
todo merge with Dave's below
Status | ||||
---|---|---|---|---|
|
Dublin Scope Planning
Status colour Blue title Done
https://ddfplugfest19.sched.com/ Tue-Thu
Clover Gambia on prior https://zoom.us/j/115579117 - 7 hours ahead
https://ddfplugfest19.sched.com/event/K1Gy/opnfv-clover-utilizing-cloud-native-technologies-for-nfv
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key LOG-707
Status | ||||
---|---|---|---|---|
|
restart log4j format and files
example
Status | ||||
---|---|---|---|---|
|
s3p Secure https endpoints
LOG + POMBA
for djhunt
Status | ||||
---|---|---|---|---|
|
Discussion on whether we need to lock down the nodeport exposed ports
Can key off POMBA work already done
todo: get s3p page
Format compliance - working with AAI team
+ perf
Status | ||||
---|---|---|---|---|
|
20190115 - casablanca cherry pick in queue logstack 5 to 3 and 1
https://gerrit.onap.org/r/#/c/75702/
(+) 20190109
from aai team
https://wiki.onap.org/display/DW/2019-01-17+AAI+Developers+Meeting+Open+Agenda
"hector has discovered that the stress test jar (liveness probe?) in aai-cassandra is hammering the cpu/ram/hd on the vm that aai is on - this breaks the etcd cluster (not the latency/network issues we suspected that may cause pod rescheduling) "
https://gerrit.onap.org/r/#/c/62405/
for
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
and
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Update: 20190129 - Existing eclipse environ for the RI being retrofitted
At the pom stage bringing in the jar via
portal/sdk in use by aai, dmaap, sdk, vid (vid link into so maybe?)
Code Block | ||
---|---|---|
| ||
<groupId>org.onap.portal.sdk</groupId> |
epic
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Discussion with @Sanjay Agraharam and [~pau2882] on checking how cassandra is running on the vm and if debug levels are on should be verified
use labelling to split aai-cs and ls - no DaemonSet
Michael O'Brien to reduce core count for ls to 1 from 3
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
edited 2019-01-10 AAI Developers Meeting
for the 10th
Status | |
---|---|
|
review investigation in
Log Streaming Compliance and API#ExistingLibraryResearch\
Luke Parker discussion
need to use the portal library in an initiating project for tx processing
working likely with the SO team - via the work we are doing for them in https://gerrit.onap.org/r/#/c/69947/
(check the original spec - ODL specific - check appc/sdnc use of ccsdk)
Status | ||||
---|---|---|---|---|
|
We have room for 2-5 committers and will be reviewing the list
Logging Enhancements Project Proposal#KeyProjectFacts
add your details to
Logging Committer Promotion Requests
20190129 status - waiting on contributor documentation from each contributor
Status colour
Blue title
Done
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
#15,19 and 37 on SP priorities for Dublin
https://ddfplugfest19.sched.com/ Tue-Thu
Clover Gambia on prior https://zoom.us/j/115579117 - 7 hours ahead
https://ddfplugfest19.sched.com/event/K1Gy/opnfv-clover-utilizing-cloud-native-technologies-for-nfv
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Log Streaming Compliance and API
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
and
opentracing via
@Sanjay
discuss integration - out of band processing -
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
see zipkin arch https://zipkin.io/pages/architecture.html
possibly tie both as a client of es ?
Tie in to ONS NA 2019 April demo booth for LF
discuss tick/tock logging spec behaviour - cassablanca implemented in dublin, dublin implemented in elalto
- Capture any changes to the Casablanca spec to be implemented in Dublin Active Logging Specifications - in the Dublin spec to be implemented in E* ONAP Application Logging Specification - Post Dublin
- verified but take care to keep the baseline spec - keep separation for any changes for elalto
Status | ||||
---|---|---|---|---|
|
s3p Secure https endpoints
LOG + POMBA
for djhunt
Status | ||||
---|---|---|---|---|
|
Discussion on whether we need to lock down the nodeport exposed ports
Can key off POMBA work already done
todo: get s3p page
Format compliance - working with AAI team
+ perf
Status | ||||
---|---|---|---|---|
|
20190115 - casablanca cherry pick in queue logstack 5 to 3 and 1
https://gerrit.onap.org/r/#/c/75702/
(+) 20190109
from aai team
https://lf-onap.atlassian.net/wiki/display/DW/2019-01-17+AAI+Developers+Meeting+Open+Agenda
"hector has discovered that the stress test jar (liveness probe?) in aai-cassandra is hammering the cpu/ram/hd on the vm that aai is on - this breaks the etcd cluster (not the latency/network issues we suspected that may cause pod rescheduling) "
#6 on 2018-12-20 AAI Developers Meeting around
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Discussion with @Sanjay Agraharam and [~pau2882] on checking how cassandra is running on the vm and if debug levels are on should be verified
use labelling to split aai-cs and ls - no DaemonSet
Michael O'Brien to reduce core count for ls to 1 from 3
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Dublin spec changes for Elalto
environment name
release name
check mail for reply Michael O'Brien
Prudence Au proposal of renaming the log file name itself for the release ie: 3.0.0-ONAP - will discuss later for next week
Cluster logging behaviour
S3P
server name in clustered environments - I will add the details and the Jira right after this meet
LOG ELK stack indexing/dashboards
with Prometheus below
Status | ||||
---|---|---|---|---|
|
Casablanca 3.0.1 work until 10th Jan
Including POMBA
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
#15,19 and 37 on SP priorities for Dublin
Status | ||||
---|---|---|---|---|
|
Log Streaming Compliance and API
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
revert Jira for data-router off TSC-92 -
pending merge of
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
and
opentracing via
LOG openlab tenant devops
cluster creation/testing
We have 2 clusters a 1+4 and 1+13 used for testing deployments and running the vFW
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
@Sanjay
discuss integration - out of band processing -
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
see zipkin arch https://zipkin.io/pages/architecture.html
possibly tie both as a client of es ?
Tie in to ONS NA 2019 April demo booth for LF
discuss tick/tock logging spec behaviour - cassablanca implemented in dublin, dublin implemented in elalto
- Capture any changes to the Casablanca spec to be implemented in Dublin Active Logging Specifications - in the Dublin spec to be implemented in E* ONAP Application Logging Specification - Post Dublin
- verified but take care to keep the baseline spec - keep separation for any changes for elalto
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
work with Vaibhav Chopra
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
@Sanjay - note the prom chart assumes a k8s environment - what about bare metal
|
Dublin spec changes for Elalto
environment name
release name
check mail for reply Michael O'Brien
Prudence Au proposal of renaming the log file name itself for the release ie: 3.0.0-ONAP - will discuss later for next week
Cluster logging behaviour
S3P
server name in clustered environments - I will add the details and the Jira right after this meet
LOG ELK stack indexing/dashboards
with Prometheus below
Status | ||||
---|---|---|---|---|
|
Casablanca 3.0.1 work until 10th Jan
Including POMBA
Jira Legacy | |||||||
---|---|---|---|---|---|---|---|
|
revert Jira for data-router off TSC-92 -
pending merge of https://gerrit.onap.org/r/#/c/6994775999/
LOG openlab tenant devops
cluster creation/testing
Michael O'Brienstatus
colour | Green |
---|---|
title | Open |
James MacNider - bring in Prianka's common eLK charts and use them in Clamp, LOG, SDC, POMBA
https://gerrit.onap.org/r/#/c/64767/INWe have 2 clusters a 1+4 and 1+13 used for testing deployments and running the vFW
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
- Shane Daniel to be removed - moved on to AI work - thanks for everything done for the logging and oom projects
- Lee Breslau
- Luke Parker
- Michael O'Brien
- Dave Williamson
- @Sanjay Agraharam
- Lorraine A. Welch
- James MacNider
- Prudence Au
- Anthony Manuel
- Cheuk Yiu Horace Ip
Review last 4 weeks since
Status | ||||
---|---|---|---|---|
|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
work with Vaibhav Chopra
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
@Sanjay - note the prom chart assumes a k8s environment - what about bare metal
Finish SO filebeat additions
Status | ||||
---|---|---|---|---|
|
Jira Legacy | |||||
---|---|---|---|---|---|
|
|
|
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy
Status | ||||
---|---|---|---|---|
|
James MacNider - bring in Prianka's common eLK charts and use them in Clamp, LOG, SDC, POMBA
https://gerrit.onap.org/r/#/c/64767/
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
rever to El-alto under
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
- Shane Daniel to be removed - moved on to AI work - thanks for everything done for the logging and oom projects
- Lee Breslau
- Luke Parker
- Michael O'Brien
- Dave Williamson
- @Sanjay Agraharam
- Lorraine A. Welch
- James MacNider
- Prudence Au
- Anthony Manuel
- Cheuk Yiu Horace Ip
Review last 4 weeks since
Status | ||||
---|---|---|---|---|
|
LOG is part of poc
Status | ||||
---|---|---|---|---|
|
Starting - will have a training session - will send out any meetings to onap-discuss
We may have the same symlink repo folder like we do for doc
Last discussed TSC 20180109
base platform includes LOG
Status | ||||
---|---|---|---|---|
|
OUT
todo review with Mike Elliott
Q2) priority of system level containers like the ELK stack - OOM has a common services JIRA - DMaaP, AAF - TODO get JIRA - make sure log is in this!
There is a cd.sh retrofit that sequences the pods in order for deployment stability - this will be phased out when tiered deployment comes in
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
https://gerrit.onap.org/r/#/c/75422/ via ONAP Development#WorkingwithJSONPath
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
for docker tags - truth
Status | ||||
---|---|---|---|---|
|
LOG is part of poc
Status | ||||
---|---|---|---|---|
|
Starting - will have a training session - will send out any meetings to onap-discuss
We may have the same symlink repo folder like we do for doc
Last discussed TSC 20180109
base platform includes LOG
Status | ||||
---|---|---|---|---|
|
OUT
todo review with Mike Elliott
Q2) priority of system level containers like the ELK stack - OOM has a common services JIRA - DMaaP, AAF - TODO get JIRA - make sure log is in this!
There is a cd.sh retrofit that sequences the pods in order for deployment stability - this will be phased out when tiered deployment comes in
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
TODO: paste TSC review - manifest is truth
RTD doc link to run the script to get the yaml override
Status | ||||
---|---|---|---|---|
|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
https://gerrit.onap.org/r/#/c/75422/ via ONAP Development#WorkingwithJSONPath
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
|
for docker tags - truth
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
|
TODO: paste TSC review - manifest is truth
RTD doc link to run the script to get the yaml override
Michael O'Brien@Sanjay Agraharam
Status | ||||
---|---|---|---|---|
|
Log Streaming Compliance and API
part of prometheus work now
Sanjay - diagram FB must be split between using AOP and AOP+spec compliant - only this one should be green
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
20181217-22
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
LOG is part of this poc
...
| |||||||||||||||||||||
LOG compliance diagram/exercise | @Sanjay Agraharam |
| 20181205 | Log Streaming Compliance and API part of prometheus work now Sanjay - diagram FB must be split between using AOP and AOP+spec compliant - only this one should be green | |||||||||||||||||
Ongoing | |||||||||||||||||||||
CI/CD pipeline TSC poc |
| 20180107 |
|
...
...
...