...

This document describes minor ONAP-wide changes to regularize logger providers, logger configuration and logger output. This will simplify deployment orchestration and customization, and improve extensibility by allowing ONAP to support bolt-on analytics pipelines such as Elastic Stack.

Introduction

ONAP consists of many components and containers, and consequently writes to many logfiles. The volume of logger output may be enormous, especially when debugging. Large, disparate logfiles are difficult to monitor and analyze. Tracing requests across many files on many file systems is untenable without tooling. 

Tools such as Elastic Stack address  addresses this problem by consuming logfiles, indexing their contents in Elasticsearch, and making them accessible, queryable and navigable via a sophisticated UI, Kibana Discover. This kind of indexing solves the problem of decentralized logger output, and elevates the importance of machine-readability. Logs Logfiles can remain human-readable, but output can be simplified.

Logger configurations in ONAP are diverse and idiosyncratic. Addressing these issues will prevent costs from being externalized to consumers such as analytics. It also affords the opportunity to remedy any issues with the handling and propagation of transaction identifiers (presently passed as X-ECOMP-RequestID). This propagation is critical to tracing requests as they traverse ONAP and related systems, and in turn to creating visualizations, reports and so on based on indexing logger output. 

...

1. Some will be more contentious than others. We  We can implement suggestions selectively if necessary. 
2. Standardization benefits everyone, including those who want non-standard logger configuration.
3. There is an economy of scale if everything can happen under a single remit.

...

All changes augment ONAP application logging guidelines.pdf on the ONAP wiki. 

...

Standardization

Default

...

All Java-based ONAP components to use Logback as their logging provider. 

• Currently: Logback and Log4J. 
• Affects: 
  • The small number of ONAP components which use Log4j.
  • These include SDNC and APPC. 
• Why: 
  • Reduced complexity.
  • Fewer configuration documents to manage, fewer special cases to maintain.
• Notes:
  • Engaging more than one provider does no actual harm.
  • What ultimately matters is the format and location of logging output, and this doesn't demand that every ONAP component uses the same provider. Consequently if some Log4J providers must remain, then that's OK.
  • This also applies to components which log without EELF. No changes are proposed in this area.
  • Standardizing on a single provider within the core set of ONAP components nevertheless reduces complexity, and seems worthwhile.

Default logfile locations

...

logfile locations

All logfiles to default to beneath /var/log, and beneath /var/log/ONAP in the case of core ONAP components.

All logger provider configuration document locations namespaced by component and (if applicable) subcomponent by default:

/var/log/ONAP/<component>[/<subcomponent>]/*.log

• Currently: varies (though a default is loosely prescribed by ONAP application logging guidelines.pdf).
• Affects: most ONAP components, superficially. 
• Why: 
  • Reduced complexity. 
  • Fewer special cases:
    • Easier configuration.
    • Easier indexing.
    • Easier orchestration.
• Notes:
  • These paths may be on a mounted persistent volume so that:
    • Logs are accessible to shippers such as Filebeat or Fluentd.
    • Logs are persistent, and independent of the lifecycle of individual containers.
  • Any indirection for containers writing to a shared volume can be managed by deployment automation; tenants don't need to know. 

Default configuration locations

All logger provider configuration documents to default to beneath /etc/ONAP. 

All logger provider configuration document locations namespaced by component and (if applicable) subcomponent by default:

/var/logetc/ONAP/<component>[/<subcomponent>]/*<provider>.logxml

• Currently: varies (though a no default is loosely prescribed by ONAP application logging guidelines.pdf).
• Affects: most ONAP components, superficially. 
• Why: 
  • Reduced complexity. 
  • Fewer special cases:
    • Easier configuration.
    • Easier indexing.
    • Easier orchestration.
• Notes:
  • These paths may be on a mounted persistent volume so that:
    • Logs are accessible to shippers such as Filebeat or Fluentd.
    • Logs are persistent, and independent of the lifecycle of individual containers.
  • Any indirection for containers writing to a shared volume can be managed by deployment automation; tenants don't need to know. 

Default configuration locations

All logger provider configuration documents to default to beneath /etc/ONAP. 

All logger provider configuration document locations namespaced by component and (if applicable) subcomponent by default:

/etc/ONAP/<component>[/<subcomponent>]/<provider>.xml

• Currently: varies (no default prescribed by ONAP application logging guidelines.pdf).
• Affects: most ONAP components, superficially. 
• Why:
  • Reduced complexity. 
  • Fewer special cases, simplified orchestration.
  • Certain components are presently not configurable, e.g. Portal, for which even trivial reconfiguration of logging requires reassembly of its WAR.
  • See Support for analytics pipelines. 
• Notes:
  • This simplifies deployment automation, reconfiguration, etc. 
  • These paths will normally be local to a container, NOT on a mounted shared volume.
  • The exact path can be fine-tuned, but:
    • The benefit of an ONAP-wide convention should be obvious.
    • This corresponds to the convention implemented (updated for ONAP) by MSO and others.
    • It should be aligned with conventions for other component- and subcomponent-specific configuration documents. 

Configuration overrides

All configuration locations to be overrideable by a system or file property passed to (or read by) the JVM. (No specific convention suggested, because it'll only divert attention from the importance of having a convention). 

• Currently: varies.
• Affects: most ONAP components, superficially.
• Why: 
  • Reduced complexity. 
  • Allow the management of configuration to be further devolved to deployment automation. 
• Notes: 
  • Doesn't need to affect other component- or provider-specific mechanisms for locating logger configuration. 
  • Doesn't have to be a system property. Could be in a file. Approaches vary widely between ONAP components.
  • All components should nonetheless respond to a standard setting.

Default configuration

With the various simplifications discussed in this document, the logger provider configuration for most components can be radically simplified. 

Proposals include:

• Standardized locations, etc. as discussed above. 
• Standardized line formats, as discussed below.
• Standardized rollover and retention (e.g 30 days, rollover at 50MB and at least daily, etc.)
• Standardized sifting into event files, if required.

A reductive example, omitting some finer details:

...

...

• • , simplified orchestration.
  • Certain components are presently not configurable, e.g. Portal, for which even trivial reconfiguration of logging requires reassembly of its WAR.
  • See Support for analytics pipelines. 
• Notes:
  • This simplifies deployment automation, reconfiguration, etc. 
  • These paths will normally be local to a container, NOT on a mounted shared volume.
  • The exact path can be fine-tuned, but:
    • The benefit of an ONAP-wide convention should be obvious.
    • This corresponds to the convention implemented (updated for ONAP) by MSO and others.
    • It should be aligned with conventions for other component- and subcomponent-specific configuration documents. 

Default configuration

With the various simplifications discussed in this document, the logger provider configuration for most components can be radically simplified. 

Proposals include:

• Standardized locations, etc. as discussed above. 
• Standardized line formats, as discussed below.
• Standardized rollover and retention (e.g 30 days, rollover at 50MB and at least daily, etc.)
• Standardized sifting into event files, if required.

A reductive example, omitting some finer details:

<configuration scan="true" scanPeriod="5 seconds" debug="false">

    <property name="component" value="component1"></property>
    <property name="subcomponent" value="subcomponent1"></property>
    <property name="outputDirectory" value="/var/log/ONAP/${component}/${subcomponent}" />
    <property name="outputFilename" value="all" />

    <property name="defaultPattern" value="%nopexception%logger
    \t%date{yyyy-MM-dd'T'HH:mm:ss.SSSXXX,UTC}
    \t%level
    \t%replace(%replace(%message){'\t','\\\\t'}){'\n','\\\\n'}
    \t%replace(%replace(%mdc){'\t','\\\\t'}){'\n','\\\\n'}
    \t%replace(%replace(%rootException){'\t','\\\\t'}){'\n','\\\\n'}
    \t%replace(%replace(%marker){'\t','\\\\t'}){'\n','\\\\n'}
    \t%thread
    \t%n"/>

    <property name="componentqueueSize" value="component1256"></property>>
    <property name="subcomponentmaxFileSize" value="subcomponent150MB"></property>>
    <property name="outputDirectorymaxHistory" value="/var/log/ONAP/${component}/${subcomponent}" 30"/>
    <property name="outputFilenametotalSizeCap" value="all10GB" />

    <property<appender name="defaultPatternconsole" value="%nopexception%loggerclass="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            \t%date{yyyy-MM-dd'T'HH:mm:ss.SSSXXX,UTC}<charset>UTF-8</charset>
      \t%level     \t%replace(%replace(%message){'\t','\\\\t'}){'\n','\\\\n'}
    \t%replace(%replace(%mdc){'\t','\\\\t'}){'\n','\\\\n'}
    \t%replace(%replace(%rootException){'\t','\\\\t'}){'\n','\\\\n'}
    \t%replace(%replace(%marker){'\t','\\\\t'}){'\n','\\\\n'}
    \t%thread
    \t%n"/> <pattern>${defaultPattern}</pattern>
        </encoder>
    </appender>

    <appender name="consoleAsync" class="ch.qos.logback.classic.AsyncAppender">
        <queueSize>${queueSize}</queueSize>
        <appender-ref ref="console" />
    </appender>

    <property<appender name="queueSizefile" valueclass="256"/ch.qos.logback.core.rolling.RollingFileAppender">
    <property name="maxFileSize" value="50MB"/>    <file>${outputDirectory}/${outputFilename}.log</file>
       <property name="maxHistory" value="30"/><rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
     <property  name="totalSizeCap" value="10GB"/>      <appender name="console" class="ch.qos.logback.core.ConsoleAppender"><fileNamePattern>${outputDirectory}/${outputFilename}.%d{yyyy-MM-dd}.%i.log.zip</fileNamePattern>
        <encoder>    <maxFileSize>${maxFileSize}</maxFileSize>
        <charset>UTF-8</charset>    <maxHistory>${maxHistory}</maxHistory>
            <pattern>$<totalSizeCap>${defaultPatterntotalSizeCap}</pattern>totalSizeCap>
        </encoder>rollingPolicy>
    </appender>    <encoder>
 <appender name="consoleAsync" class="ch.qos.logback.classic.AsyncAppender">         <queueSize>${queueSize}</queueSize><charset>UTF-8</charset>
        <appender-ref ref="console" />   <pattern>${defaultPattern}</pattern>
        </encoder>
    </appender>

    <appender name="filefileAsync" class="ch.qos.logback.coreclassic.rolling.RollingFileAppenderAsyncAppender">
        <file>$<queueSize>${outputDirectoryqueueSize}/${outputFilename}.log</file></queueSize>
        <rollingPolicy<appender-ref classref="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">"file" />
    </appender>

    <root level="info"   <fileNamePattern>${outputDirectory}/${outputFilename}.%d{yyyy-MM-dd}.%i.log.zip</fileNamePattern>additivity="false">
        <appender-ref ref="consoleAsync"/>
    <maxFileSize>${maxFileSize}</maxFileSize>    <!--<appender-ref ref="fileAsync"/>-->
    </root>

  <maxHistory>${maxHistory}</maxHistory>  <logger name="org.onap.example.component1.subcomponent2" level="debug" additivity="false">
       <totalSizeCap>${totalSizeCap}</totalSizeCap>
    <appender-ref ref="fileAsync" />
    </rollingPolicy>
        <encoder>
            <charset>UTF-8</charset>
            <pattern>${defaultPattern}</pattern>
        </encoder>
    </appender>

    <appender name="fileAsync" class="ch.qos.logback.classic.AsyncAppender">
        <queueSize>${queueSize}</queueSize>
        <appender-ref ref="file" />
    </appender>

    <root level="info" additivity="false">
        <appender-ref ref="consoleAsync"/>
        <!--<appender-ref ref="fileAsync"/>-->
    </root>

    <logger name="org.onap.example.component1.subcomponent2" level="debug" additivity="false">
        <appender-ref ref="fileAsync" />
    </logger>

</configuration>

...

• Reduced complexity. Fewer special cases. 
• Standardization allows dramatic simplification of other configuration, such as Elastic Stack shipping and indexing. Non-standard behavior externalizes costs.
• Devolution to deployment automation and other tooling. 

...

logger>

</configuration>

• Currently: varies, but mostly based on EELF logback.xml at https://github.com/att/EELF/blob/master/EELF-Samples/src/main/resources/logback.xml. 
• Affects: all ONAP components (but their configuration documents will be changing anyway).
• Why:
  • Reduced complexity. Fewer special cases. 
  • Standardization allows dramatic simplification of other configuration, such as Elastic Stack shipping and indexing. Non-standard behavior externalizes costs.
  • Devolution to deployment automation and other tooling. 
• Notes:
  • In theory provider configuration could be reduced to a single ONAP-wide document servicing all Java-based components.
  • Some suggestions in this proposal will be rejected by the ONAP community. Configuration can be updated accordingly, but the benefits of standardized configuration remain. 

Default logger provider

All Java-based ONAP components to use Logback as their logging provider. 

• Currently: Logback and Log4J. 
• Affects: 
  • The small number of ONAP components which use Log4j, particularly 1.X.
  • These include SDNC and APPC. 
• Why: 
  • Reduced complexity.
  • Log4j 1.X encoding support is limited, especially with regards to escaping. (Log4j 2.X is fine).
  • Fewer configuration documents to manage, fewer special cases to maintain.
• Notes:
  • Engaging more than one provider does no actual harm.
  • What ultimately matters is the format and location of logging output, and this doesn't demand that every ONAP component uses the same provider. Consequently if some Log4J providers must remain, then that's OK.
  • This also applies to components which log without EELF. No changes are proposed in this area.
  • Standardizing on a single provider within the core set of ONAP components nevertheless reduces complexity, and seems worthwhile.

Configuration overrides

All configuration locations to be overrideable by a system or file property passed to (or read by) the JVM. (No specific convention suggested, because it'll only divert attention from the importance of having a convention). 

• Currently: varies.
• Affects: most ONAP components, superficially.
• Why: 
  • Reduced complexity. 
  • Allow the management of configuration to be further devolved to deployment automation. 
• Notes: 
  • Doesn't need to affect other component- or provider-specific mechanisms for locating logger configuration. 
  • Doesn't have to be a system property. Could be in a file. Approaches vary widely between ONAP components.
  • All components should nonetheless respond to a standard setting.

...

Extensability

MDCs

The set of MDCs in ONAP application logging guidelines.pdf is fine, but there may be many arbitrarily many others, and there's no need for them to be itemized in logger provider configuration. Itemizing them means changing all configuration documents when new MDCs are introduced. 

...

• Currently: patchy. 
• Affects: to be determined, but at least a few components.
• Why: reliable propagation is a prerequisite for analytics.
• Notes:

  • This also requires the generation of transaction IDs in initiating components. 

  • For this to be economical, a remit is sought to make make minor changes across the ONAP codebase. 

...