About This Document

...

Original AT&T ONAP Logging guidelines (pre amsterdam release) - for historical reference only: https://wiki.onap.org/download/attachments/1015849/ONAP%20application%20logging%20guidelines.pdf?api=v2

The Acumos logging specification follows this document at https://wiki.acumos.org/display/OAM/Log+Standards 

Logback reference: Logging Developer Guide#Logback.xml based on https://gerrit.onap.org/r/#/c/62405

Introduction

The purpose of ONAP logging is to capture information needed to operate, troubleshoot and report on the performance of the ONAP platform and its constituent components. Log records may be viewed and consumed directly by users and systems, indexed and loaded into a datastore, and used to compute metrics and generate reports. 

...

The library and aop wrapper are written in Java and will work for Clojure (thanks to a discussion with Shwetank that reminded my me of languages that compile to bytecode) and Scala as well. see: https://git.onap.org/logging-analytics/tree/reference

...

For the casablanca release there the logging specification has been updated and finalized as of June 2018.  Implementation of this specification is required but the method of implementation is optional based on each team's level of possible engagement.

...

Use a spring based Aspect library that emits Markers automatically around function calls and retrofit your code to log via Luke's SLF4J for internal log messages.

LoggingWithAOP16339021

Logging Library Location and Use

...

1. By selection of a logging provider such as Logback or Log4j, typically via the classpath. 
2. By way of a provider configuration document, typically logback.xml or log4j.xml. See Providers16339021.

SLF4J

SLF4J is a logging facade, and a humble masterpiece. It combines what's common to all major, modern Java logging providers into a single interface. This decouples the caller from the provider, and encourages the use of what's universal, familiar and proven. 

...

Logback is the most commonly used provider. It is generally configured by an XML document named logback.xml. See Configuration 16339021.

See HELM template https://git.onap.org/logging-analytics/tree/reference/provider/helm/logback

...

Log4j 2.X is somewhat less common than Logback, but equivalent. It is generally configured by an XML document named log4j.xml. See Configuration 16339021.

Log4j 1.X

Strongly discouraged from Beijing onwards, since 1.X is EOL, and since it does not support escaping, so its output may not be machine-readable. See https://logging.apache.org/log4j/1.2/.

...

It isn't the aim of this document to reiterate Best Practices, so advice here is general: 

• Use a logger logging facade such as SLF4J or EELF. 
• Write log messages in English.
• Write meaningful messages. Consider what will be useful to consumers of logger output. 
• Log at the appropriate level. Be aware of the volume of logs that will be produced.
• Safeguard the information in exceptions, and ensure it is never lost.
• Use errorcodes error codes to characterise characterize exceptions. 
• Log in a machine-readable format. See Conventions.
• Log for analytics as well as troubleshooting.

...

See https://www.slf4j.org/api/org/slf4j/Logger.html and https://logback.qos.ch/manual/layouts.html#ClassicPatternLayout for their origins and use.

Logger Name

This indicates the origin of a log name of the logger that logged the message.

It is confusingly named, and since in Java logging it is normally given as a class or package name, it's more often referred to as the logger "class" or "package"In Java it is convention to name the logger after the class or package using that logger.

• In Java, report the class or package name.
• In Python, the class or source filename.

...

Think carefully about the information you report at each loglevellog level. The default log level is INFO.

Some loggers define non-standard levels, like FINE, FINER, WARNING, SEVERE, FATAL or CRITICAL. Use these judiciously, or avoid them.

Message

The freetext free text payload of a log event. 

This is the most important item of information in most log messages. See General 16339021 guidelines.

Internationalization

...

if (logger.isDebugEnabled()) {
    logger.debug("But this WILL hurt: " + costlyToSerialize);
}

Parameterized logging is preferable.

Context

MDCs

A Mapped Diagnostic Context (MDC) allows an arbitrary string-valued attribute to be attached to a Java thread via a ThreadLocal variable. The MDC's value is then emitted with each message logged by that thread. The set of MDCs associated with a log message is serialized as unordered name-value pairs (see Text Output 16339021).

A good discussion of MDCs can be found at https://logback.qos.ch/manual/mdc.html. 

...

...

• It's only a few calls. 
• It can be largely abstracted in the case of EELF logging.
  
  

MDC - InstanceID

(formerly InstanceUUID)

If known, this field contains a universally unique identifier used to differentiate between multiple instances of the same (named) log writing service/application. Its value is set at instance creation time (and read by it, e.g., at start/initialization time from the environment). This value should be picked up by the component instance from its configuration file and subsequently used to enable differentiation of log records created by multiple, locally load balanced ONAP component or subcomponent instances that are otherwise identically configured.

...

Used as valuable URI - to annnote invoke marker

Review in terms of Marker 16339021 - INVOKE - possiblly add INVOKE-return - to filter reporting

...

• There is considerable duplication:
  • BeginTimestamp, EndTimestamp, ElapsedTime. These are all captured elsewhere (and ElapsedTime is even redundant within that triplet).
  • Server, ServerIPAddress, ServerFQDN, VirtualServiceName. Overkill. Should be one, plus optionally ClientIPAddress (or some variant thereof).
  • TargetEntity, TargetServiceName, not obviously different to similar attributes.
• There is junk:
  • Severity? Nagios codes?
  • ProcessKey?
  • All the stuff that's already grayed out in the table above.
  • People may defend these individually, maybe vigorously, but they're domain-specific:
    • That absolutely doesn't mean they can't be used.
    • Beats configuration allows ad hoc contexts to be indexed.
    • But perhaps they don't belong in this kind of spec.
• Redundant attributes *do* matter, because:
  • Populating and propagating everything prescribed by the guide approaches being prohibitive. People won't do it, and people *don't* do it. 
  • If something might be in one of several attributes then that's worse than it being in just one.
• That means:
• We're left with only two MANDATORY attributes, necessary to build invocations graphs:
  • RequestID - top-level transactions. 
  • InvocationID - inter-component invocations.
• And a minimal number of OPTIONAL descriptive attributes: ServiceInstanceID, InstanceUUIDInstanceID, Server, StatusCode, ResponseCode, ResponseDescription.
• Those are the ones we need to document clearly, support in APIs, etc.
• That's <=10, a manageable number. 
• And again, that matters because if the number isn't manageable, people won't (and don't) comply. 

...

Note there are 3 tabs (see p_mak in logback.xml) delimiting the MARKERS (ENTRY and EXIT) at the end of each line

<property name="p_mak" value="%replace(%replace(%marker){'\t', '\\\\t'}){'\n','\\\\n'}"/>

2018-07-05T20:21:34.794Z	http-nio-8080-exec-2	INFO	org.onap.demo.logging.ApplicationService	InstanceUUIDInstanceID=ede7dd52-91e8-45ce-9406-fbafd17a7d4c, RequestID=f9d8bb0f-4b4b-4700-9853-d3b79d861c5b, ServiceName=/logging-demo/rest/health/health, InvocationID=8f4c1f1d-5b32-4981-b658-e5992f28e6c8, InvokeTimestamp=2018-07-05T20:21:26.617Z, PartnerName=, ClientIPAddress=0:0:0:0:0:0:0:1, ServerFQDN=localhost			ENTRY	
2018-07-05T20:22:09.268Z	http-nio-8080-exec-2	INFO	org.onap.demo.logging.ApplicationService	ResponseCode=, InstanceUUIDInstanceID=ede7dd52-91e8-45ce-9406-fbafd17a7d4c, RequestID=f9d8bb0f-4b4b-4700-9853-d3b79d861c5b, ServiceName=/logging-demo/rest/health/health, ResponseDescription=, InvocationID=8f4c1f1d-5b32-4981-b658-e5992f28e6c8, Severity=, InvokeTimestamp=2018-07-05T20:21:26.617Z, PartnerName=, ClientIPAddress=0:0:0:0:0:0:0:1, ServerFQDN=localhost, StatusCode=			EXIT	

...

This should accompany INVOKE when the invocation is synchronous.

...

SLF4J:

public static final Marker INVOKE_SYNCHRONOUS;
static {
    INVOKE_SYNCHRONOUS = MarkerFactory.getMarker("INVOKE");
    INVOKE_SYNCHRONOUS.add(MarkerFactory.getMarker("SYNCHRONOUS"));
}
// ...

// Generate and report invocation ID. 

final String invocationID = UUID.randomUUID().toString();
MDC.put(MDC_INVOCATION_ID, invocationID);
try {
    logger.debug(INVOKE_SYNCHRONOUS, "Invoking synchronously ... ");
}
finally {
    MDC.remove(MDC_INVOCATION_ID);
}

// Pass invocationID as HTTP X-InvocationID header.

callDownstreamSystem(invocationID, ... );

EELF example of SYNCHRONOUS reporting, without changing published APIs. 

...

Error Codes

Errorcodes Error codes are reported as MDCs. 

TODO: add to table

Exceptions should be accompanied by an errrorcodeerror code. Typically this is achieved by incorporating errorcodes error codes into your exception hierarchy and error handling. ONAP components generally do not share this kind of code, though EELF defines a marker interface (meaning it has no methods) EELFResolvableErrorEnum.

A common convention is for errorcodes error codes to have two components:

1. A prefix, which identifies the origin of the error. 
2. A suffix, which identifies the kind of error.

...

1. Logs should be human-readable (within reason). 
2. Shipper and indexing performance and durability depends on logs that can be parsed quickly and reliably.
3. Consistency means fewer shipping and indexing rules are required.

Text Output

TODO: 20190115 - do not take the example in this section until I reverify it in terms of the reworked spec example in 

https://gerrit.onap.org/r/#/c/62405/20/reference/logging-kubernetes/logdemonode/charts/logdemonode/resources/config/logback.xml

ONAP needs to strike a balance between human-readable and machine-readable logs. This means:

...

Logs on each filebeat docker container sidecar - /var/log/onap

...

...

...

...

What's New

(Including what WILL be new in v1.2  / R2). 

...