Versions Compared

Old Version 9

changes.mady.by.user Michael O'Brien

Saved on

compared with

New Version Current

changes.mady.by.user Michael O'Brien

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Going with the casablanca spec for Dublin - for the library to implement this spec - retrofit portal/sdk (or SO's library) - need to have an AOP enabled library like our reference - 3rd choice is to AOP enable AAI's

Active Logging Specifications

deprecated - 20181025: update: originally we decided to continue the casablanca version v111 of ONAP Application Logging Specification v1.2 (Casablanca) in Dublin but after constructive feedback/pushback from one small part of the AT&T based Acumos community and feedback/pushback from part of the AT&T based ECOMP community based on initial pulls into ECOMP and some feedback from 2 high level projects in ONAP - I recommend refocusing on only one requirement - a common ONAP first spec.  Creating a 3-way spec that satisfies Acumos, ECOMP and ONAP can no longer be our focus due to the limited time and very limited resources of the logging spec/ELK deployment work a couple of us need to do.  I have received tremendous support and spec/JIRA  work from Lorraine A. Welch, Spondon Dey, Shishir Thakore and Luke Parker (all the marker/mdc work and slf4j library - and originating the project spec/elk stack).  Creating a combined specification that meets the needs of private AT&T ECOMP that consumes Open Source ONAP, public Open Source Acumos that consumes parts of ONAP and even ONAP itself with it's several Logging implementations across Portal, SO, AAI is not really working or will take too long to accept a new library that everyone can accept.  Since I am the current PTL, I have decided that an "ONAP first" direction is required.  Any implementation/spec changes will need to be prototyped in public first and accepted directly into affected projects as we change the spec - not as an exercise after we have finalized the spec.   I recommend that the way forward as suggested by the core of my coworkers in Acumos is to take an existing library and slowly iterate that library to match an "ONAP" specific spec that includes markers and MDC's and only if possible keep the older formats by the ONAP consuming projects.   I will start from the bottom up for Dublin - using the portal/sdk library and adjust it in sync with any spec changes in parallel (no spec changes without first prototyping them in existing use cases by users of the portal/sdk library).  For the E release we can increase adoption to the rest of the components not currently using portal/sdk.  What is critical this time around is working with the team(s) affected by any adjustment to their library first.  Secondary consideration will be given to non-opensource consumers of this specification inside corporations that privately wrap ONAP.  This will mean becoming part of the public portal and policy weekly meetings as a de-facto sub-project. The first order of work will be infrastructure (shipping logs to ELK) and then an alignment to the existing portal/sdk logback.xml spec - before we start changing things. so we can start mining the ELK logs before we adjust them. 


Table of Contents

Statement of Work

    Detail any changes to the existing casablanca spec (implementing in Dublin) - Active Logging Specifications - for the upcoming E release.

    Each specification is created in the release prior to where it is implemented 

    Work to align the logs, ship them, consolidate, provide for analysis, backup and restore them will be done from a developer perspective for the Dublin release.

Requirements

R1: Provide for consistent logs across ONAP that allow for cross pod indexing

R2: Validate/Implement log shipping to the existing ELK stack

Analysis

Existing ONAP Logging State - Infrastructure and Schema

ref

Code Block
themeMidnight
20181031
ubuntu@a-osn-cd:~/oom/kubernetes$ kubectl get pods --all-namespaces | grep /2
onap          onap-aaf-aaf-cm-7f7494fffd-fmwxp                              0/1       Init:0/2            0          7m
onap          onap-aaf-aaf-fs-7c7b4966d7-h5p9m                              0/1       Init:0/2            0          7m
onap          onap-aaf-aaf-gui-5b9cf459bf-9hlx2                             0/1       Init:0/2            0          7m
onap          onap-aaf-aaf-hello-d5c4f8dd-zbfms                             0/1       Init:0/2            0          7m
onap          onap-aaf-aaf-locate-58bdf576f-jlnsc                           0/1       Init:0/2            0          7m
onap          onap-aaf-aaf-oauth-86c8d8687f-lfntl                           0/1       Init:0/2            0          7m
onap          onap-aaf-aaf-service-66689b78f5-gj7tb                         0/1       Init:1/2            0          7m
onap          onap-aaf-aaf-sms-vault-0                                      0/2       ContainerCreating   0          7m
onap          onap-aai-aai-babel-5b44f78ff4-8mdpw                           0/2       ContainerCreating   0          6m
onap          onap-aai-aai-champ-64bf5d7c59-f9kj9                           0/2       Init:0/1            0          6m
onap          onap-aai-aai-data-router-b74db89d8-76s8v                      0/2       Init:0/1            0          6m
onap          onap-aai-aai-gizmo-9fc98788d-wjkf5                            0/2       ContainerCreating   0          6m
onap          onap-aai-aai-graphadmin-7c4c8f66bf-f6f9w                      0/2       Init:0/1            0          6m
onap          onap-aai-aai-modelloader-7f56f4b6cf-lsclh                     0/2       ContainerCreating   0          6m
onap          onap-aai-aai-resources-64c8b8c4c4-vmvl9                       0/2       Init:0/1            0          6m
onap          onap-aai-aai-search-data-648975d74-5g8ks                      0/2       ContainerCreating   0          6m
onap          onap-aai-aai-sparky-be-c8bfdcbcb-tcm9m                        0/2       Init:0/1            0          6m
onap          onap-aai-aai-spike-77f64cfc5c-wms6r                           0/2       Init:0/1            0          6m
onap          onap-aai-aai-traversal-64fff6d57f-hgwgd                       0/2       Init:0/1            0          6m
onap          onap-appc-appc-0                                              0/2       Init:0/1            0          6m
onap          onap-clamp-clamp-9dfd65f8f-rhdlc                              0/2       Init:0/1            0          6m
onap          onap-dmaap-message-router-kafka-74f9bff668-h6flz              0/1       Init:0/2            0          6m
onap          onap-esr-esr-server-6d7dd55794-rp9zz                          0/2       Pending             0          6m
onap          onap-msb-msb-discovery-59d48c65b-k68cj                        0/2       Init:0/1            0          6m
onap          onap-msb-msb-eag-76857bcc54-8zbpp                             0/2       Pending             0          6m
onap          onap-msb-msb-iag-59697db845-mlpz8                             0/2       Pending             0          6m
onap          onap-multicloud-multicloud-5765cd745-bb57j                    0/2       Pending             0          6m
onap          onap-multicloud-multicloud-azure-6c99454479-4457s             0/2       Pending             0          6m
onap          onap-multicloud-multicloud-ocata-79cc9dfb76-w6bx4             0/2       Pending             0          6m
onap          onap-multicloud-multicloud-pike-547d878bb5-d9cjj              0/2       Pending             0          6m
onap          onap-multicloud-multicloud-vio-67979f99bc-gvzxz               0/2       Pending             0          6m
onap          onap-multicloud-multicloud-windriver-748868744d-69sjr         0/2       Pending             0          6m
onap          onap-policy-pap-5b95866fff-6wxln                              0/2       Pending             0          5m
onap          onap-policy-pdp-0                                             0/2       Init:0/1            0          5m
onap          onap-pomba-pomba-aaictxbuilder-5d45d589-nnxck                 0/2       Pending             0          5m
onap          onap-pomba-pomba-networkdiscovery-8487cc6667-2j6wq            0/2       Pending             0          5m
onap          onap-pomba-pomba-networkdiscoveryctxbuilder-c9c5c8f98-2svzw   0/2       Pending             0          5m
onap          onap-pomba-pomba-search-data-7548469497-9pjq4                 0/2       Pending             0          5m
onap          onap-pomba-pomba-servicedecomposition-8985f9849-2cdl7         0/2       Pending             0          5m
onap          onap-portal-portal-app-5cc7b45bd9-45c6c                       0/2       Init:0/1            0          5m
onap          onap-portal-portal-db-config-dwrrt                            0/2       Pending             0          5m
onap          onap-portal-portal-sdk-5bb84db56c-x4zcj                       0/2       Init:0/1            0          5m
onap          onap-sdc-sdc-be-68d86d55cf-pxf66                              0/2       Pending             0          5m
onap          onap-sdc-sdc-fe-67ccb88464-9qbwj                              0/2       Pending             0          5m
onap          onap-sdc-sdc-onboarding-be-67b4bc94b-2gscq                    0/2       Pending             0          5m
onap          onap-sdc-sdc-wfd-fe-87d9cd457-26crq                           0/2       Pending             0          5m
onap          onap-sdnc-sdnc-0                                              0/2       Pending             0          5m
onap          onap-sdnc-sdnc-db-0                                           0/2       Pending             0          5m
onap          onap-vfc-vfc-catalog-85bd56b986-c6zld                         0/2       Pending             0          4m
onap          onap-vfc-vfc-generic-vnfm-driver-dc4864b96-c9b6h              0/2       Pending             0          4m
onap          onap-vfc-vfc-huawei-vnfm-driver-589c6c8467-swdqt              0/2       Pending             0          4m
onap          onap-vfc-vfc-juju-vnfm-driver-7cb5dc655d-mkb7l                0/2       Pending             0          4m
onap          onap-vfc-vfc-nokia-vnfm-driver-57c4f5d9cd-jh9mb               0/2       ContainerCreating   0          4m
onap          onap-vfc-vfc-nslcm-675c748bb7-t6rfz                           0/2       Init:0/1            0          4m
onap          onap-vfc-vfc-resmgr-599fb9c6f7-64djz                          0/2       Pending             0          4m
onap          onap-vfc-vfc-vnflcm-75cc76d968-9dblx                          0/2       Pending             0          4m
onap          onap-vfc-vfc-vnfmgr-5f866c5458-4h56p                          0/2       Init:0/1            0          4m
onap          onap-vfc-vfc-vnfres-579459469f-f2kns                          0/2       Pending             0          4m
onap          onap-vfc-vfc-zte-vnfm-driver-5dd4b87cd7-ppj42                 0/2       Pending             0          4m
onap          onap-vid-vid-599d9c54f9-79vfl                                 0/2       Pending             0          4m

ONAP Module Dependencies - Compile and Runtime

Existing Library Research

Library Selected for Reference Implementation

Java

Python

Specification

Use Cases

Design Issues

Developer Guide

Testing Guide

...

    see Logging Dublin Scope

Features

Key words are: out of the box, reuse components, iterative changes - not major spec changes.

There is one primary use case and one enabler use case

UC1: Provide for transaction tracing via elasticsearch or dashboard for distributed transactions across ONAP components - enabled by the logs shipped to the ELK stack - in real time.
(the ELK stack is there - we just need to adjust log content - amount and format)

UC2: Provide for UC1 tracing via standardized logs - ideally via marker/mdc (label/key:value pair) markup


Logging Alignment before enhancement

Plan is to take the portal/sdk library - align the rest of onap to this library and then add marker/mdc (labels and key/value pair) support as required in phase 2

Work will include 1) verify pipeline 2) verify logback format

Open Standards

Look at a standard that will allow us to use standard parsing libraries all the way to eventual AI/ML by adopting a standard like open tracing.

Infrastructure Improvements

including filebeat sidecar anti-pattern/replacement, elk stack upgrade, deployment template, library language scope (python, swift...)

Monitoring

Investigate out of the box prometheus monitoring

S3P

backup/restore, security, log format monitoring, elk dashboards

3rd party consumption alignment

Lower in priority will be to align onap with requirements of Acumos and ECOMP for example - the focus will be on alignment of ONAP components only - to the primary goal of providing a transaction tracing system via the ELK stack.