/
Log Streaming Compliance and API
Log Streaming Compliance and API
- Michael O'Brien
Owned by Michael O'Brien
This page details the current state of log streaming in ONAP and plans to increase compliance towards the goal of providing end to end log tracing (ONAP transactions, onap system, undercloud).
ONAP Log Streaming Deployment View
Analysis
using
http://openlab13.onap.info:30253/app/kibana
Compliance
Streaming
Format
Library
Fields
timestampadd t versionadd t ACTIONadd t API_KEYadd t ARTIFACT_DATAadd t AUTH_STATUSadd t AlertSeverityadd AuditBeginTimestampadd t AuditMessageadd BeginTimestampadd t CONSUMER_IDadd t CURR_ARTIFACT_UUIDadd t CURR_STATEadd t CURR_VERSIONadd t ClassNameadd t CustomField1add t CustomField2add t DESCadd t DNOTIF_TOPICadd t DSTATUS_TOPICadd t D_ENVadd t ECOMP_USERadd t ElapsedTimeadd EndTimestampadd t ErrorCategoryadd t ErrorCodeadd t INVARIANT_UUIDadd t InstanceUUIDadd t MODIFIERadd t PREV_ARTIFACT_UUIDadd t PREV_STATEadd t PREV_VERSIONadd t PartnerNameadd t REALMadd t RESOURCE_NAMEadd t RESOURCE_TYPEadd t RemoteHostadd t RequestIdadd t ResponseCodeadd t ResponseDescriptionadd t SERVICE_INSTANCE_IDadd t STATUSadd t ServerFQDNadd t ServerIPAddressadd t ServiceInstanceIdadd t ServiceNameadd t StatusCodeadd t TargetEntityadd t TargetServiceNameadd t Threadadd t Timeradd Timestampadd t URLadd t USERadd t USER_AFTERadd t USER_BEFOREadd t _idadd t _indexadd # _scoreadd t _typeadd t auditOnadd t beat.hostnameadd t beat.nameadd t beat.versionadd t componentLogFileadd t componentNameadd t hostadd t input_typeadd t logleveladd t messageadd # offsetadd t serviceInstanceIDadd t sourceadd t tagsadd t timeradd t typeadd
Existing ONAP Logging State - Infrastructure and Schema
Deployment Dependency Tree
Note: these dependencies - introduced using the readiness container by Mandeep Khinda are at the lowest deployment level and represent a partial view of the REST/API dependency tree - they do not reflect any compile time or runtime/injection code dependencies (pom.xml)
LOG-924 - Getting issue details... STATUS
tested sdnc via sudo ./cd.sh -b master -e onap -p false -a false -t robot -x 1 -z false -f false -s 600 -c false -d true -w false -r false
where dmaap, consul and sdc are raised first via dev0.yaml and sdnc is in dev1.yaml
Deployment Dependency Tree - Container level
Use for any containers stuck at the 0/1 Init:0/1 stage - these are likely waiting on dependent containers
check the --container-name kv pair in StatefulSet/Deployment.yaml:spec:intiContainers:args
or the corresponding defined chart/container names in values.yaml:config:
106 sets in 87 files
overall order
aaf<-aai
aaf<-oof
music<-oof
dmaap<-aai
dmaap<-pomba
dmaap<-sdnc
consul<-sdnc
sdc<-sdnc
consul<-dcaegen2
msb<-dcaegen2
order
dmaap, aaf, msb, consul, dcaegen2, aai, oof, sdc, sdnc, pomba
valid sets of pods
aaf
aai, dmaap, aaf
dcaegen2, consul, msb
oof, aaf, music
pomba, dmaap
sdnc, consul, sdc
..the rest can be singular
aaf
aaf-cm
aaf-locate
aaf-fs
aaf-locate
aaf-gui
aaf-cm
aaf-hello
aaf-locate
aaf-locate
aaf-service
aaf-oauth
aaf-locate
aaf-service
aaf-cs
aaf-sms
aaf-sms-quorumclient (via job)
aaf-sms-vault
aaf-sms-vault-backend
aai
aai
aai-resources
aai-traversal
aai-graphadmin
aai-champ
aai-cassandra
aai-graphadmin
aai-cassandra
aai-resources
aai-cassandra
aai-sparky-be
aai-elasticsearch
aai-search-data
aai
aai-spike
message-router-kafka
aai-traversal
aai
aai-cassandra
aaf-locate (conditional)
appc
appc
mariadb-galera
appc-ansible-server
appc
clamp
clamp
mariadb
clamp-dash-kibana
clamp-dash-es
clamp-dash-logstash
clamp-dash-es
common
controller-blueprints
mariadb-galera
mongo
*-nfs-provisioner
mysql
*-nfs-provisioner
dgbuilder
network-name-gen
mariadb-galera
dcaegen2
dcae-bootstrap
dcae-cloudify-manager
consul-server
msb-discovery
kube2msb
dep-health-init
hbase
dmaap
dmaap-bus-controller
postgres
dmaap-dr-node
dmaap-dr-prov
dmaap-dr-prov
mariadb
message-router
kafka
zookeeper
message-router-kafka
zookeeper
log
log-kibana
log-elasticsearch
log-logstash
log-elasticsearch
msb
kube2msb
msb-discovery
msb-discovery
msb-consul
msb-eag
msb-discovery
msb-iag
msb-discovery
music
music-cassandra
music-tomcat
zookeeper
oof
oof-has-api
oof-has-controller
aaf-service
oof-has-controller
music-tomcat
aaf-sms
oof-has-data
music-tomcat
oof-has-reservation
music-tomcat
oof-has-service
music-tomcat
policy
policy
mariadb
brmsgw
pap
drools
mariadb
nexus
pdb
pap
pomba
pomba-contextaggregator
message-router
pomba-kibana
pomba-elasticsearch
pomba-data-router
pomba-search-data
pomba-search-data
pomba-elasticsearch
portal
portal-widget
portal-db
portal-sdk
portal-db
sdc
sdc-dcae-be
common.name
sdc-be
sdc-dcae-dt
sdc-dcae-be
sdc-dcae-fe
sdc-dcae-be
sdc-dcae-tosca-lab
sdc-dcae-be
sdc-fe
sdc-kb
sdc-wfd-fe
sdc-wfd-be
sdnc
sdnc
mysql
sdnc-ansible-server
sdnc
dmaap-listener
mysql
sdnc
message-router
sdnc-portal
mysql / sdnc-db
sdnc
sdnc-prom
sdnc
consul
ueb-listener
mysql
sdnc
sdc-be
message-router
so
so
mariadb
so-bpmn-infra
mariadb
so-catalog-db-adapter
mariadb
so-openstack-adapter
mariadb
so-request-db-adapter
mariadb
so-sdc-adapter
mariadb
so-sdc-controller
mariadb
so-vfc-adapter
mariadb
vfc
vfc-catalog
vfc-db
vfc-ems-driver
mariadb // commented
vfc-generic-vnfm-driver
mariadb // commented
vfc-hauwei-vnfm-driver
mariadb // commented
vfc-juju-vnfm-driver
mariadb // commented
vfc-multivim-proxy
mariadb // commented
vfc-nokia-vnfm-driver
mariadb // commented
vfc-nokia-v2vnfm-driver
mariadb // commented
vfc-nslcm
vfc-db
vfc-vnfmgr
vfc-db
vfc-resmgr
mariadb // commented
vfc-workflow
mariadb // commented
vfc-workflow-engine
mariadb // commented
vfc-vnflcm
vfc-db
vfc-vnfres
vfc-db
vfc-zte-sdnc-driver
mariadb // commented
vfc-zte-vnfm-driver
mariadb // commented
vid
vid
mariadb-galera
vnfsdk
postgres
Deployment Profiles
For use case specific deployments
# deployment profiles to cd.sh # each of the 28 components in alpha sequence --set pod.enabled=true|false status aaaccccd delmmnop pprssssu vvvv 11000100 10010101 01111011 0100 - vFW - no monitoring 11000100 10110101 11111011 0100 - vFW - with monitoring 11110111 10110111 11111111 0100 - vFW - cl
| # | Team | container | Required y/n/fw/fwCl | Ram | Cpu | nodeport | Type | logback | Dependencies | |
|---|---|---|---|---|---|---|---|---|---|---|
| aaf | ||||||||||
| aai | y | |||||||||
| appc | fwCL | |||||||||
| clamp | fwCL | |||||||||
| cli | ||||||||||
| consul | ||||||||||
| fwCL | ||||||||||
| dmaap | y | |||||||||
| esr | ||||||||||
| log | n | |||||||||
sniro- emulator | n | |||||||||
| oof | n | |||||||||
| msb | n | |||||||||
| multicloud | n | |||||||||
| nbi | ||||||||||
| policy | y | |||||||||
| pomba | ||||||||||
| portal | y | |||||||||
| robot | y | |||||||||
| sdc | y | |||||||||
| sdnc | y | |||||||||
| so | y | |||||||||
| uui | n | |||||||||
| vfc | n | |||||||||
| vid | y | |||||||||
| vnfsdk | n |
| Pod | Container/service | Tech | URLs | Filebeat Streaming | logback.xml | library | dependencies | Log level adjust | |
|---|---|---|---|---|---|---|---|---|---|
| aaf | aaf-cm aaf-fs aaf-gui aaf-hello aaf-locate aaf-oauth aaf-service aaf-sms-vault-0 | ||||||||
| aai | aai-babel aai-champ aai-data-router aai-gizmo aai-graphadmin aai-modelloader aai-resources aai-search-data aai-sparky-be aai-spike aai-traversal | ||||||||
| appc | appc-0 | ||||||||
| clamp | - | ||||||||
| cli | - | ||||||||
| consul | - | ||||||||
| contrib-netbox | - | ||||||||
| dcaegen2 | - | ||||||||
| dmaap | - | ||||||||
| esr | - | ||||||||
| log | - | ||||||||
| logdemonode | - | ||||||||
| msb | - | ||||||||
| multicloud | - | ||||||||
| nbi | - | ||||||||
| oof | - | ||||||||
| policy | - | ||||||||
| pomba | - | ||||||||
| portal | - | ||||||||
| robot | - | ||||||||
| sdc | - | ||||||||
| sdnc | - | ||||||||
| sniro | - | ||||||||
| so | - | ||||||||
| uui | - | ||||||||
| vfc | - | ||||||||
| vid | - | ||||||||
| vnfsdk |