/
Log Streaming Compliance and API

Log Streaming Compliance and API

Jan 22, 2019

This page details the current state of log streaming in ONAP and plans to increase compliance towards the goal of providing end to end log tracing (ONAP transactions, onap system, undercloud).



ONAP Log Streaming Deployment View



Analysis

using

http://openlab13.onap.info:30253/app/kibana

Compliance

Streaming

https://12factor.net/logs

Format

Library

Fields

timestampadd t versionadd t ACTIONadd t API_KEYadd t ARTIFACT_DATAadd t AUTH_STATUSadd t AlertSeverityadd AuditBeginTimestampadd t AuditMessageadd BeginTimestampadd t CONSUMER_IDadd t CURR_ARTIFACT_UUIDadd t CURR_STATEadd t CURR_VERSIONadd t ClassNameadd t CustomField1add t CustomField2add t DESCadd t DNOTIF_TOPICadd t DSTATUS_TOPICadd t D_ENVadd t ECOMP_USERadd t ElapsedTimeadd EndTimestampadd t ErrorCategoryadd t ErrorCodeadd t INVARIANT_UUIDadd t InstanceUUIDadd t MODIFIERadd t PREV_ARTIFACT_UUIDadd t PREV_STATEadd t PREV_VERSIONadd t PartnerNameadd t REALMadd t RESOURCE_NAMEadd t RESOURCE_TYPEadd t RemoteHostadd t RequestIdadd t ResponseCodeadd t ResponseDescriptionadd t SERVICE_INSTANCE_IDadd t STATUSadd t ServerFQDNadd t ServerIPAddressadd t ServiceInstanceIdadd t ServiceNameadd t StatusCodeadd t TargetEntityadd t TargetServiceNameadd t Threadadd t Timeradd Timestampadd t URLadd t USERadd t USER_AFTERadd t USER_BEFOREadd t _idadd t _indexadd # _scoreadd t _typeadd t auditOnadd t beat.hostnameadd t beat.nameadd t beat.versionadd t componentLogFileadd t componentNameadd t hostadd t input_typeadd t logleveladd t messageadd # offsetadd t serviceInstanceIDadd t sourceadd t tagsadd t timeradd t typeadd





Existing ONAP Logging State - Infrastructure and Schema

Deployment Dependency Tree

Note: these dependencies - introduced using the readiness container by @Mandeep Khinda are at the lowest deployment level and represent a partial view of the REST/API dependency tree - they do not reflect any compile time or runtime/injection code dependencies (pom.xml)

https://lf-onap.atlassian.net/browse/LOG-924

tested sdnc via  sudo ./cd.sh -b master -e onap -p false -a false -t robot -x 1 -z false -f false -s 600 -c false -d true -w false -r false

where dmaap, consul and sdc are raised first via dev0.yaml and sdnc is in dev1.yaml

Deployment Dependency Tree - Container level

Use for any containers stuck at the 0/1 Init:0/1 stage - these are likely waiting on dependent containers check the --container-name kv pair in StatefulSet/Deployment.yaml:spec:intiContainers:args or the corresponding defined chart/container names in values.yaml:config: 106 sets in 87 files overall order aaf<-aai aaf<-oof music<-oof dmaap<-aai dmaap<-pomba dmaap<-sdnc consul<-sdnc sdc<-sdnc consul<-dcaegen2 msb<-dcaegen2 order dmaap, aaf, msb, consul, dcaegen2, aai, oof, sdc, sdnc, pomba valid sets of pods aaf aai, dmaap, aaf dcaegen2, consul, msb oof, aaf, music pomba, dmaap sdnc, consul, sdc ..the rest can be singular aaf aaf-cm aaf-locate aaf-fs aaf-locate aaf-gui aaf-cm aaf-hello aaf-locate aaf-locate aaf-service aaf-oauth aaf-locate aaf-service aaf-cs aaf-sms aaf-sms-quorumclient (via job) aaf-sms-vault aaf-sms-vault-backend aai aai aai-resources aai-traversal aai-graphadmin aai-champ aai-cassandra aai-graphadmin aai-cassandra aai-resources aai-cassandra aai-sparky-be aai-elasticsearch aai-search-data aai aai-spike message-router-kafka aai-traversal aai aai-cassandra aaf-locate (conditional) appc appc mariadb-galera appc-ansible-server appc clamp clamp mariadb clamp-dash-kibana clamp-dash-es clamp-dash-logstash clamp-dash-es common controller-blueprints mariadb-galera mongo *-nfs-provisioner mysql *-nfs-provisioner dgbuilder network-name-gen mariadb-galera dcaegen2 dcae-bootstrap dcae-cloudify-manager consul-server msb-discovery kube2msb dep-health-init hbase dmaap dmaap-bus-controller postgres dmaap-dr-node dmaap-dr-prov dmaap-dr-prov mariadb message-router kafka zookeeper message-router-kafka zookeeper log log-kibana log-elasticsearch log-logstash log-elasticsearch msb kube2msb msb-discovery msb-discovery msb-consul msb-eag msb-discovery msb-iag msb-discovery music music-cassandra music-tomcat zookeeper oof oof-has-api oof-has-controller aaf-service oof-has-controller music-tomcat aaf-sms oof-has-data music-tomcat oof-has-reservation music-tomcat oof-has-service music-tomcat policy policy mariadb brmsgw pap drools mariadb nexus pdb pap pomba pomba-contextaggregator message-router pomba-kibana pomba-elasticsearch pomba-data-router pomba-search-data pomba-search-data pomba-elasticsearch portal portal-widget portal-db portal-sdk portal-db sdc sdc-dcae-be common.name sdc-be sdc-dcae-dt sdc-dcae-be sdc-dcae-fe sdc-dcae-be sdc-dcae-tosca-lab sdc-dcae-be sdc-fe sdc-kb sdc-wfd-fe sdc-wfd-be sdnc sdnc mysql sdnc-ansible-server sdnc dmaap-listener mysql sdnc message-router sdnc-portal mysql / sdnc-db sdnc sdnc-prom sdnc consul ueb-listener mysql sdnc sdc-be message-router so so mariadb so-bpmn-infra mariadb so-catalog-db-adapter mariadb so-openstack-adapter mariadb so-request-db-adapter mariadb so-sdc-adapter mariadb so-sdc-controller mariadb so-vfc-adapter mariadb vfc vfc-catalog vfc-db vfc-ems-driver mariadb // commented vfc-generic-vnfm-driver mariadb // commented vfc-hauwei-vnfm-driver mariadb // commented vfc-juju-vnfm-driver mariadb // commented vfc-multivim-proxy mariadb // commented vfc-nokia-vnfm-driver mariadb // commented vfc-nokia-v2vnfm-driver mariadb // commented vfc-nslcm vfc-db vfc-vnfmgr vfc-db vfc-resmgr mariadb // commented vfc-workflow mariadb // commented vfc-workflow-engine mariadb // commented vfc-vnflcm vfc-db vfc-vnfres vfc-db vfc-zte-sdnc-driver mariadb // commented vfc-zte-vnfm-driver mariadb // commented vid vid mariadb-galera vnfsdk postgres



Deployment Profiles

For use case specific deployments

# deployment profiles to cd.sh # each of the 28 components in alpha sequence --set pod.enabled=true|false status aaaccccd delmmnop pprssssu vvvv 11000100 10010101 01111011 0100 - vFW - no monitoring 11000100 10110101 11111011 0100 - vFW - with monitoring 11110111 10110111 11111111 0100 - vFW - cl





#

Team

pod

container

Required

y/n/fw/fwCl

Ram

Cpu

nodeport

Type

logback

Dependencies

#

Team

pod

container

Required

y/n/fw/fwCl

Ram

Cpu

nodeport

Type

logback

Dependencies



aaf





















aai





y















appc





fwCL















clamp





fwCL















cli





















consul



























fwCL















dmaap





y















esr





















log





n















sniro-

emulator





n















oof





n















msb





n















multicloud





n















nbi





















policy





y















pomba





















portal





y















robot





y















sdc





y















sdnc





y















so





y















uui





n















vfc





n















vid





y















vnfsdk





n













Pod

Container/service

Pod

Container/service