Use just for today - real meeting busted - zoom.us/j/7939937123
Meeting every 2 weeks at 1100 EST Tue - https://zoom.us/j/519971638
...
https://jira.onap.org/secure/RapidBoard.jspa?rapidView=143&view=planning.nodetail&epics=visible
Agenda
- Michael: status no progress on logging related jiras involving dublin scope (code/tech) last week
- Passed M2
- Discussed in TSC and PTL meets - issue related to manifest vs k8s values.yaml overrides for casablanca and CMR releases
- In architecture review for M3 - (Addressed comments) Logging Dublin M3 Architecture Review
- Working more for OOM lately - Created RKE script to replace rancher - pending work is to test HA - https://gerrit.onap.org/r/#/c/79067/ for
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key OOM-1670 - Assisting with ONS April demo prep - 2 or more ONAP installations working together - one as an edge cloud
- Focused on onap devops - primarily for 2 customers deploying ONAP
- 5G training next week
- April 3-5 ONS
- PTL position is open - let me know if anyone would like to take it.
- Also quickly discussed idea of rolling LOG into OOM - should revisit
- Pending committer for pomba - Committer Request for Trevor Tait and Committer Request for Pierre Rioux - will send out reviews once stats are completed - TSC review will be needed
Attendees
Items
...
Status
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
...
...
...
Status | ||||
---|---|---|---|---|
|
...
...
Status | ||||
---|---|---|---|---|
|
...
pending 20190208 decision
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
...
...
Status | ||||
---|---|---|---|---|
|
...
pending documentation in RTD
Team,
In the TSC it was decided to treat the diff between oom and the manifest by always running the manifest generated yaml in your deployments – you will not need to do this for master work – just for Casablanca and RC0-2 work
Working out the details in
https://jira.onap.org/browse/LOG-929
/michael
...
...
Status | ||||
---|---|---|---|---|
|
...
Log Streaming Compliance and API
discussed at last PTL meet with djhunt
El-Alto 1.4 logging spec change - plan only
todo merge with Dave's below
...
...
Status | ||||
---|---|---|---|---|
|
...
Dublin Scope Planning
...
...
Status colour Blue title Done
...
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key LOG-707
...
...
Status | ||||
---|---|---|---|---|
|
...
restart log4j format and files
...
Attendees
Luke Parker , Prudence Au , Michael O'Brien , Sanjay, Lorraine A. Welch
Agenda
- Meetings every 2 weeks now - next meeting the 19th
- Need to accomidate Luke Parker - who is up at 3 am right now in Australia - will be 1 in the morning until April - then 2am
- ideas: Better to be up at 1-3 than 8am - keep at 11 for now
- For NA - stays at 11 - but I will verify if we are on UTC
- DST is coming this weekend - we should agree on the meeting time - Luke does not go DST for 3 more weeks
- Michael: status no progress on logging related jiras involving dublin scope (code/tech) last week
- Passed M2
- Discussed in TSC and PTL meets - issue related to manifest vs k8s values.yaml overrides for casablanca and CMR releases -
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key TSC-86 - TSC 2019-02-28
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key OOM-1560
- In architecture review for M3 - (Addressed comments) Logging Dublin M3 Architecture Review
- Working more for OOM lately - Created RKE script to replace rancher - pending work is to test HA - https://gerrit.onap.org/r/#/c/79067/ for
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key OOM-1670 - Assisting with ONS April demo prep - 2 or more ONAP installations working together - one as an edge cloud
- Focused on onap devops - primarily for 2 customers deploying ONAP
- 5G training next week
- April 3-5 ONS work
- k8s infrastructure and
- Logging will be a part of the demo - we will show the kibana logs during a vFW operation in an i-frame for the master and edge ONAP deploys
- PTL position is open - let me know if anyone would like to take it.
- Also quickly discussed idea of rolling LOG into OOM - should revisit
- Pending committer for pomba - Committer Request for Trevor Tait and Pierre Rioux - Committer Request for [Logging-Analytics] - will send out reviews once stats are completed - TSC review will be needed
- TODO I need to check MSB issues on onap-discuss -
- Spondon - import the dashboard - fix the one on the wiki - Lorraine A. Welch export to json first ONAP Culprit Locator
- Lorraine A. Welch question about changing the index for the db (Grafana for example) - @Timestamp timestamp for example - TODO - merge the 2 indexes -
- Q) Sanjay: (opentracking related - picking up errors on stdout (err, out)
- how:
- (also review from 2 weeks ago - Lorraine A. Welch System.out standardout - review in terms of not using a hardcoded file appender - in terms of syslogs
- see https://12factor.net/logs)
- log retention on pod failure - those not mapped to PVs - including the current EmptyDir link between the sidecar - etherial PV - logs only end up in the elk stack
- Look into PV/PVC structure for log retention instead of the default emptydir
- https://git.onap.org/logging-analytics/tree/reference/logging-kubernetes/logdemonode/charts/logdemonode/templates/deployment.yaml#n114
- see PV/PVC example .
logs can go to the /dockerdatanfs) - https://git.onap.org/oom/tree/kubernetes/portal/charts/portal-mariadb/templates/deployment.yaml#n84
- Topic: File retention
- Q: rotation schedule, size of retention - what role does the PV on the ELK stack play compared to the individual source pod PVs
- @Luke's example - discuss with Mike Elliott - review a pending upstream contribution
- https://git.onap.org/logging-analytics/tree/reference/provider/helm/logback/chart/resources/logback.xml#n10
- variables set in
- https://git.onap.org/logging-analytics/tree/reference/provider/helm/logback/chart/values.yaml#n14'
- Review any issues with multiple providers in SDNC and APPC
- Lorraine A. Welch check https://www.loomsystems.com/blog/single-post/2017/01/30/a-comparison-of-fluentd-vs-logstash-log-collector
Attendees
Items
...
PRI | Title | Responsible | Status
| In Dublin | Last Worked on | Start | Notes | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Security Vulnerability template | Ongoing | IN | 20190122 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
M1 template |
| IN | 20190124 | 20190122 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ONS NA 2019 April Talk proposal |
| Yellow
| In Progress
| IN | 201901115 | 20190108 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Jira Legacy | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
server | System Jira | 20190122 | pending 20190208 decision
| 630
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Log Streaming Compliance and API#DeploymentProfiles
Status | ||||
---|---|---|---|---|
|
Update: 20190129 - Existing eclipse environ for the RI being retrofitted
At the pom stage bringing in the jar via
portal/sdk in use by aai, dmaap, sdk, vid (vid link into so maybe?)
Code Block | ||
---|---|---|
| ||
<groupId>org.onap.portal.sdk</groupId> |
epic
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
review investigation in
Log Streaming Compliance and API#ExistingLibraryResearch\
Luke Parker discussion
need to use the portal library in an initiating project for tx processing
working likely with the SO team - via the work we are doing for them in https://gerrit.onap.org/r/#/c/69947/
(check the original spec - ODL specific - check appc/sdnc use of ccsdk)
Status | ||||
---|---|---|---|---|
|
We have room for 2-5 committers and will be reviewing the list
Logging Enhancements Project Proposal#KeyProjectFacts
add your details to
Logging Committer Promotion Requests
20190129 status - waiting on contributor documentation from each contributor
|
Status | ||||
---|---|---|---|---|
|
pending documentation in RTD
Team,
In the TSC it was decided to treat the diff between oom and the manifest by always running the manifest generated yaml in your deployments – you will not need to do this for master work – just for Casablanca and RC0-2 work
Working out the details in
https://jira.onap.org/browse/LOG-929
/michael
Status | ||||
---|---|---|---|---|
|
El-Alto 1.4 logging spec change - plan only
todo merge with Dave's below
Status | ||||
---|---|---|---|---|
|
Dublin Scope Planning
Status colour Blue title Done
https://ddfplugfest19.sched.com/ Tue-Thu
Clover Gambia on prior https://zoom.us/j/115579117 - 7 hours ahead
https://ddfplugfest19.sched.com/event/K1Gy/opnfv-clover-utilizing-cloud-native-technologies-for-nfv
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key LOG-707
Status | ||||
---|---|---|---|---|
|
restart log4j format and files
example
Status | ||||
---|---|---|---|---|
|
s3p Secure https endpoints
LOG + POMBA
for djhunt
Status | ||||
---|---|---|---|---|
|
Discussion on whether we need to lock down the nodeport exposed ports
Can key off POMBA work already done
todo: get s3p page
Format compliance - working with AAI team
+ perf
Status | ||||
---|---|---|---|---|
|
20190115 - casablanca cherry pick in queue logstack 5 to 3 and 1
https://gerrit.onap.org/r/#/c/75702/
(+) 20190109
from aai team
https://wiki.onap.org/display/DW/2019-01-17+AAI+Developers+Meeting+Open+Agenda
"hector has discovered that the stress test jar (liveness probe?) in aai-cassandra is hammering the cpu/ram/hd on the vm that aai is on - this breaks the etcd cluster (not the latency/network issues we suspected that may cause pod rescheduling) "
https://gerrit.onap.org/r/#/c/62405/
for
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
and
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Update: 20190129 - Existing eclipse environ for the RI being retrofitted
At the pom stage bringing in the jar via
portal/sdk in use by aai, dmaap, sdk, vid (vid link into so maybe?)
Code Block | ||
---|---|---|
| ||
<groupId>org.onap.portal.sdk</groupId> |
epic
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Discussion with @Sanjay Agraharam and [~pau2882] on checking how cassandra is running on the vm and if debug levels are on should be verified
use labelling to split aai-cs and ls - no DaemonSet
Michael O'Brien to reduce core count for ls to 1 from 3
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
edited 2019-01-10 AAI Developers Meeting
for the 10th
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
#15,19 and 37 on SP priorities for Dublin
|
review investigation in
Log Streaming Compliance and API#ExistingLibraryResearch\
Luke Parker discussion
need to use the portal library in an initiating project for tx processing
working likely with the SO team - via the work we are doing for them in https://gerrit.onap.org/r/#/c/69947/
(check the original spec - ODL specific - check appc/sdnc use of ccsdk)
Status | ||||
---|---|---|---|---|
|
We have room for 2-5 committers and will be reviewing the list
Logging Enhancements Project Proposal#KeyProjectFacts
add your details to
Logging Committer Promotion Requests
20190129 status - waiting on contributor documentation from each contributor
Status colour Blue title Done
https://ddfplugfest19.sched.com/ Tue-Thu
Clover Gambia on prior https://zoom.us/j/115579117 - 7 hours ahead
https://ddfplugfest19.sched.com/event/K1Gy/opnfv-clover-utilizing-cloud-native-technologies-for-nfv
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Log Streaming Compliance and API
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
and
opentracing via
@Sanjay
discuss integration - out of band processing -
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
see zipkin arch https://zipkin.io/pages/architecture.html
possibly tie both as a client of es ?
Tie in to ONS NA 2019 April demo booth for LF
discuss tick/tock logging spec behaviour - cassablanca implemented in dublin, dublin implemented in elalto
- Capture any changes to the Casablanca spec to be implemented in Dublin Active Logging Specifications - in the Dublin spec to be implemented in E* ONAP Application Logging Specification - Post Dublin
- verified but take care to keep the baseline spec - keep separation for any changes for elalto
Status | ||||
---|---|---|---|---|
|
s3p Secure https endpoints
LOG + POMBA
for djhunt
Status | ||||
---|---|---|---|---|
|
Discussion on whether we need to lock down the nodeport exposed ports
Can key off POMBA work already done
todo: get s3p page
Format compliance - working with AAI team
+ perf
Status | ||||
---|---|---|---|---|
|
20190115 - casablanca cherry pick in queue logstack 5 to 3 and 1
https://gerrit.onap.org/r/#/c/75702/
(+) 20190109
from aai team
https://lf-onap.atlassian.net/wiki/display/DW/2019-01-17+AAI+Developers+Meeting+Open+Agenda
"hector has discovered that the stress test jar (liveness probe?) in aai-cassandra is hammering the cpu/ram/hd on the vm that aai is on - this breaks the etcd cluster (not the latency/network issues we suspected that may cause pod rescheduling) "
#6 on 2018-12-20 AAI Developers Meeting around
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Discussion with @Sanjay Agraharam and [~pau2882] on checking how cassandra is running on the vm and if debug levels are on should be verified
use labelling to split aai-cs and ls - no DaemonSet
Michael O'Brien to reduce core count for ls to 1 from 3
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Dublin spec changes for Elalto
environment name
release name
check mail for reply Michael O'Brien
Prudence Au proposal of renaming the log file name itself for the release ie: 3.0.0-ONAP - will discuss later for next week
Cluster logging behaviour
S3P
server name in clustered environments - I will add the details and the Jira right after this meet
LOG ELK stack indexing/dashboards
with Prometheus below
Status | ||||
---|---|---|---|---|
|
Casablanca 3.0.1 work until 10th Jan
Including POMBA
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
#15,19 and 37 on SP priorities for Dublin
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Log Streaming Compliance and API
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
revert Jira for data-router off TSC-92 -
pending merge of https://gerrit.onap.org/r/#/c/75999/
LOG openlab tenant devops
cluster creation/testing
We have 2 clusters a 1+4 and 1+13 used for testing deployments and running the vFW
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
and
opentracing via
@Sanjay
discuss integration - out of band processing -
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
|
see zipkin arch https://
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
work with Vaibhav Chopra
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
@Sanjay - note the prom chart assumes a k8s environment - what about bare metal
Finish SO filebeat additionszipkin.io/pages/architecture.html
possibly tie both as a client of es ?
Tie in to ONS NA 2019 April demo booth for LF
discuss tick/tock logging spec behaviour - cassablanca implemented in dublin, dublin implemented in elalto
- Capture any changes to the Casablanca spec to be implemented in Dublin Active Logging Specifications - in the Dublin spec to be implemented in E* ONAP Application Logging Specification - Post Dublin
- verified but take care to keep the baseline spec - keep separation for any changes for elalto
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
James MacNider - bring in Prianka's common eLK charts and use them in Clamp, LOG, SDC, POMBA
https://gerrit.onap.org/r/#/c/64767/
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
rever to El-alto under
|
Dublin spec changes for Elalto
environment name
release name
check mail for reply Michael O'Brien
Prudence Au proposal of renaming the log file name itself for the release ie: 3.0.0-ONAP - will discuss later for next week
Cluster logging behaviour
S3P
server name in clustered environments - I will add the details and the Jira right after this meet
LOG ELK stack indexing/dashboards
with Prometheus below
Status | ||||
---|---|---|---|---|
|
Casablanca 3.0.1 work until 10th Jan
Including POMBA
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
|
revert Jira for data-router off TSC-92 -
pending merge of https://gerrit.onap.org/r/#/c/75999/
LOG openlab tenant devops
cluster creation/testing
We have 2 clusters a 1+4 and 1+13 used for testing deployments and running the vFW
Review last 4 weeks since
LOG Meeting Minutes 2018-12-05Status | ||||
---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
Jira Legacy | |||||
---|---|---|---|---|---|
|
|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
work with Vaibhav Chopra
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
@Sanjay - note the prom chart assumes a k8s environment - what about bare metal
Finish SO filebeat additions
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
James MacNider - bring in Prianka's common eLK charts and use them in Clamp, LOG, SDC, POMBA
https://gerrit.onap.org/r/#/c/64767/
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
rever to El-alto under
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Status | ||||
---|---|---|---|---|
|
- Shane Daniel to be removed - moved on to AI work - thanks for everything done for the logging and oom projects
- Lee Breslau
- Luke Parker
- Michael O'Brien
- Dave Williamson
- @Sanjay Agraharam
- Lorraine A. Welch
- James MacNider
- Prudence Au
- Anthony Manuel
- Cheuk Yiu Horace Ip
Review last 4 weeks since
Status | ||||
---|---|---|---|---|
|
Jira Legacy server System Jira serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176 key TSC-79 https://wikilf-onap.onapatlassian.orgnet/wiki/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy
Status | ||||
---|---|---|---|---|
|
LOG is part of poc
Status | ||||
---|---|---|---|---|
|
Starting - will have a training session - will send out any meetings to onap-discuss
We may have the same symlink repo folder like we do for doc
Last discussed TSC 20180109
base platform includes LOG
Status | ||||
---|---|---|---|---|
|
OUT
todo review with Mike Elliott
Q2) priority of system level containers like the ELK stack - OOM has a common services JIRA - DMaaP, AAF - TODO get JIRA - make sure log is in this!
There is a cd.sh retrofit that sequences the pods in order for deployment stability - this will be phased out when tiered deployment comes in
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
https://gerrit.onap.org/r/#/c/75422/ via ONAP Development#WorkingwithJSONPath
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
for docker tags - truth
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
TODO: paste TSC review - manifest is truth
RTD doc link to run the script to get the yaml override
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
20181217-22
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
@Sanjay Agraharam
Status | ||||
---|---|---|---|---|
|
Log Streaming Compliance and API
part of prometheus work now
Sanjay - diagram FB must be split between using AOP and AOP+spec compliant - only this one should be green
|
@Sanjay Agraharam
Status | ||||
---|---|---|---|---|
|
Jira Legacy | ||||||
---|---|---|---|---|---|---|
|
LOG is part of this poc
...
| 20181205 | Log Streaming Compliance and API part of prometheus work now Sanjay - diagram FB must be split between using AOP and AOP+spec compliant - only this one should be green | |||||||||||||||||||
Ongoing | |||||||||||||||||||||
CI/CD pipeline TSC poc |
| 20180107 |
|
...
...
...