This is a working document.
The below matrix is a representation of the log management categories (lifecycle) in relation to the two categories of run-time logs (logs of ONAP events, logs of events from services orchestrated by ONAP).
Team Members
- Amy Zwarico
- Robert Heinemann
- Muddasar Ahmed
- rouzaut
- Byung-Woo Jun
- Brian Smith (Unlicensed)
- s.silvius
- PUT YOUR NAME HERE
Actions
30 Jul 2021
- Amy: List of proposed events that should be collected from ONAP and Metadata
- Muddasar: Determine if there is a standard terminology regarding logging architecture terms. Eg., Are the categories in the above table industry accepted?
**There probably a body of work we can reference that spells this out. ACTION: Literature review for that: No standard terms, but some popular standard formats like BSD, Syslog (IETF), Common Event Format (CEF), by Arcsight. OWASP, NIST and Major Cloud Vendors have guidance in user docs or SDK regarding logs and formats. NIST SP 800-92 can be found here https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-92.pdf
Application logs some time are split into Application Access and Application Operations. Other major Category in older literature is focusing on Operating System, in Containerized deployments this can be Docker and host OS, Node logs. We should consider listing in best practice some of these categories that do not fall within Application Container.
Do we need to specify format type? WebAPIs, Datanbases and applications way have slightly different format requirements.
- Fabian: Initial investigation of ONAP responding to security events.
13 Aug 2021
- Review Requirements list Amy put together
- Muddasar to provide links to NIST security logging standards:
https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-92.pdf
- Fabian: Initial investigation of ONAP responding to security events.
- Bob to provide Orchestration logging events
- Log Template as suggested by Chakir on Tuesday call ( Apache 2 log template as an example. Can we review work from Logging enhancement project?
Key
X: Indicates what is in-scope for ONAP
BP: Indicates a best practice
P: Partially in-scope (group consensus is mixed).
Definitions:
Application: This refers to runtime containerized application
Container: This refers to the container platform and orchestration software that ONAP interfaces with. For example, docker and K8S.
Infrastructure: This refers to any physical, virtualization, element managers, and/or operating system components.
From a 2017 AT&T Doc on ONAP Logging
"Application logging” refers to logs written by ONAP component “applications”.
"System/infrastructure logging” refers to the separate/related set of logs produced by software components not developed for ONAP (e.g. DBMS, application container, web servers, ‘middle boxes’, JVM, OS, hypervisor, etc.) that are used in the implementation of these components." (See reference #4).
Scope
| Phase | 1 (ONAP Based Events) | 2 (events from services orchestrated by ONAP) | |||||
|---|---|---|---|---|---|---|---|
| ONAP Components (e.g., DCAE, SDC, etc.) | Services (xNF, xApps) | ||||||
| Lifecycle | Application | Container (k8s and Docker) | Infrastructure | Application | Container | Infrastructure | |
| How they are generated | Generation | X | X | ||||
| How they are made available | Collection | X | X | ||||
| Monitoring | |||||||
| Alerting | |||||||
| Response | P | P | X | X | |||
Phase 1 will focus on logs of ONAP events.
Phase 2 will focus on logs of events from services orchestrated by ONAP
Notes
At a high level there are 5 broad categories in regards to Security Event Management (Or is this a Security Event Lifecycle?)
Generation
- Within ONAP both containers and infrastructure generate raw data that have security concerns.
- Containers (xNFs)
- Infrastructure (Docker and K8S)
- There are a set of logs that both Docker and K8S generate that relate to security monitoring.
- That is documented here: https://wiki.onap.org/download/attachments/103419713/Logging%20-%20ATTACK%20to%20SECCOM_v3.pptx?version=1&modificationDate=1622560207000&api=v2
These below refer to the ONAP (Application and Infrastructure Columns)
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.
Further refinement for this document only the keywords REQUIRED, RECOMMENDED and OPTIONAL will be used.
PLEASE CONSIDER THE BELOW THE MOST UP TO DATE LIST. While transferring data here from various spreadsheets and PPTs there were several errors corrected (duplicates, wrong ID number, wrong VNF REQ Numbers).
Logging Practice Requirements (Proposed)
| ID | Type | Description | Reference |
|---|---|---|---|
CON-LOG-REQ-19 | REQUIRED | The container MUST be capable of automatically synchronizing the system clock daily with the Operator’s trusted time source, to assure accurate time reporting in log files. It is recommended that Coordinated Universal Time (UTC) be used where possible to eliminate ambiguity owing to daylight savings time. Sync time source The container MUST be capable of automatically synchronizing the system clock daily with the Operator’s trusted time source, to assure accurate time reporting in log files. It is recommended that Coordinated Universal Time (UTC) be used where possible to eliminate ambiguity owing to daylight savings time. | R-629534 |
| CON-LOG-REQ-20 | REQUIRED | The container and container application MUST use the STDOUT for security logs collection | REQ-374 |
| CON-LOG-REQ-F1 | REQUIRED | Using systems and applications with native logging functionality is essential. This function MUST be taken into account during any design and development process. | |
| CON-LOG-REQ-F6 | REQUIRED | The container application SHALL contextualize events (log enrichment) e.g: Timestamp, IP address having generated the logs, user concerned, functionality concerned, error of application, detail of the error, all access to a resource, success of application, etc… | |
| CON-LOG-REQ-13 | REQUIRED | The container MUST have security logging for the container and container application active from initialization. | R-84160 |
| CON-LOG-REQ-15 | REQUIRED | The container MUST detect when its security audit log storage medium is approaching capacity (configurable) and issue an alarm. | R-63330 |
| CON-LOG-REQ-F9 | REQUIRED | An event log rotation policy MUST be formalized and implemented on all logging system equipment. | |
| CON-LOG-REQ-18 | REQUIRED | The container MUST support the storage of security audit logs for a configurable period of time. | R-54816 |
| CON-LOG-REQ-16 | REQUIRED | The container MUST support the capability of online storage of security audit logs. | R-41252 |
| CON-LOG-REQ-F8 | REQUIRED | A disk partition MUST be dedicated to storing event logs on the equipment that generates them | |
| CON-LOG-REQ-F7 | REQUIRED | Logs MUST be automatically exported to a different physical machine than the one that generated them | |
| CON-LOG-REQ-F2 | REQUIRED | It is recommended that no processing MUST be performed on the logs before they are transferred. (no classification, it is not the behavior of an application to define the categories of an event) Note: this needs to be converted into a requirement | |
| CON-LOG-REQ-F5 | RECOMMENDED | It is recommended the container application SHOULD adopt a tree structure for the storage of event logs. | |
| CON-LOG-REQ-14 | REQUIRED | The container MUST protect all security audit logs by standard operating system access control mechanisms, by sending to a remote system, or by encryption. | R-56920 |
CON-LOG-REQ-F4 CON-LOG-REQ-F10 | REQUIRED | Access to logs MUST be write restricted to a limited number of accounts with a need to know | |
| CON-LOG-REQ-21 | RECOMMENDED | The container SHOULD provide the capability of maintaining the integrity of its static files using a cryptographic method. (Fabian) Propose to remove because this is a hardening requirement, not a logging requirement (Bob) Instead of removing this is now in the Best Practices category and we can make it a recommendation. | R-465236 |
CON-LOG-REQ-12 CON-LOG-REQ-XX | REQUIRED | The container and container application MUST NOT include an authentication credential, e.g., password, in the security audit logs, even if encrypted. The container and container application MUST NOT include a sensitive information in the log | R-04982 |
| CON-LOG-REQ-17 | REQUIRED | The container MUST generate security audit logs that can be sent to Security Analytics Tools for analysis. | R-04492 |
Security Event Generation Requirements (Proposed)
REQUIRED, RECOMMENDED and OPTIONAL
| ID | Type | Description | Reference |
|---|---|---|---|
CON-LOG-REQ-1 | REQUIRED | The container and container application MUST log successful and unsuccessful authentication attempts, e.g., authentication associated with a transaction, authentication to create a session, authentication to assume elevated privilege. | R-54520 |
CON-LOG-REQ-2 | The container and container application MUST log logoffs. | R-55478 | |
CON-LOG-REQ-3 | The container and container application MUST log starting and stopping of security logging. | R-13344 | |
CON-LOG-REQ-4 | The container and container application MUST log success and unsuccessful creation, removal, or change to the inherent privilege level of users. | R-07617 | |
CON-LOG-REQ-5 | The container and container application MUST log connections to the network listeners of the container. | R-94525 | |
| CON-LOG-REQ-6 | The container and container application MUST log the addition, deletion or modification of files in the container. | ||
| CON-LOG-REQ-MP05 | The container MUST log lifecycle events | ||
| CON-LOG-REQ-MP07 | Container administration services activities and executed commands MUST be logged. (e.g., Build requests, Runtime commands) (Available in docker Daemon Logs) | T1609, T1612 | |
| CON-LOG-REQ-MP08 | The container MUST log API calls (such as: syscalls, those that deploy containers, Discovery API). (Available in docker daemon log). | T1610, T1204, T1611, T1068, T1552, T1613, T1525 | |
| CON-LOG-REQ-MP09 | The container MUST log creation of scheduled jobs in containers. ( Available at the K8S level) | T1053 | |
| CON-LOG-REQ-MP10 | Image registry events MUST be logged (e.g., additions) | T1204 | |
| CON-LOG-REQ-MP06 | Log anonymous requests | ||
Steps for approval: POC → Best Practice → Global Requirement
Metadata for Security Events (Proposed)
Security Log Structure
Date / Time | Log Level | Transaction ID | Status Code | Severity | Container Data | Protocol | Service / Program Name | Log Message | |||||
Image Tag / Name | Image Digest | ID | Name | Principal ID | Role / Attribute ID | ||||||||
NOTE:
- Grey boxes indicate that a (yet to be determined) container logger function / service will provide.
- White boxes indicate the developer of a container or container application will provide.
Example:
From Fabian:
2021-09-10T14:50:37.929Z|d855a2c6-c58f-4d8d-b199-3382d11504d2|http-nio-8083-exec-5|/manage/health|kube-probe/1.19|||DEBUG|500||Headers : X-Content-Type-Options:nos
Security Log Field Definitions
Type Synonyms:
REQUIRED: SHALL OR MUST
RECOMMENDED: SHOULD
OPTIONAL: MAY
| ID | Type | Field Name | Description | Reference |
|---|---|---|---|---|
CON-SEC-LOG-01
| REQUIRED | Date and Time | The container and container application MUST log the field “date/time” in the security audit logs. The value should be represented in UTC and formatted per ISO 8601, such as “2015-06-03T13:21:58+00:00”. The time should be shown with the maximum resolution available to the logging component (e.g., milliseconds, microseconds) by including the appropriate number of decimal digits. For example, when millisecond precision is available, the date-time value would be presented as, as “2015-06-03T13:21:58.340+00:00”. | |
| CON-LOG-REQ-MP04 | REQUIRED | Log Level | The container and container application MUST use an appropriately configured logging level that can be changed dynamically. The intention of this field is to not cause performance degradation via excessive logging. The value of this field should be on of the following: "FATAL", "ERROR", "WARN", "INFO", "DEBUG", "TRACE" The verbosity of the logging increases from left to right. How do we synchronize these levels across projects and what the logging API they are using? | (4) |
| CON-LOG-REQ-MP13 | REQUIRED | Transaction ID | The container and container application MUST log Transaction ID A requestID is a universally unique value that identifies a single transaction request within the ONAP platform. Its value is conformant to RFC4122 UUID. This value is readily and easily obtained in most programming environments. | (4) |
CON-LOG-REQ-10 | REQUIRED | Status Code | The container and container application MUST log a "status code" in the security audit logs. This field indicates the high level status for transactional or sub operational events. It must be one of the following values:
| (4) |
| CON-SEC-LOG-11 | REQUIRED | Severity | The container and container application MUST log the severity level of a processing event. This is to be used for error reporting in internal processing in conjunction with the status code field. The value of this field MUST be on of the following: {"NONE", "MINOR", "MAJOR", "CRITICIAL"} Optional: 0, 1, 2, 3 see Nagios monitoring/alerting for specifics/details. | (4) |
| CON-LOG-REQ-MP03 | Container Image Name / Tag | The container and container application MUST log the Container Image Name/Tag. The image name/tag is as returned by the docker images command. NOTE: Images are not required to have tags | ||
| CON-LOG-REQ-MP11 | Container Image Digest | The container and container application MUST log the container image digest. The digest is a cryptographic digest as returned by the docker images --digests command. | T1036, T1525 | |
| CON-LOG-REQ-MP01 | Container ID | The container and container application MUST log the container ID. The container ID is the same that is returned by the docker ps -q command. NOTE: The container ID is unique for life time of the the container instance. Once the container is killed, this ID goes away. | ||
| CON-LOG-REQ-MP02 | Container Name | The container and container application MUST log the container name. This is the unique name of the image ( webserver, FW, DCAE01). This is returned by the docker ps command. | ||
| CON-LOG-REQ-11 | REQUIRED | Principal ID | The container and container application MUST log the Principal identity of a requestor in the security audit logs. This field should contain the identification name of the client application (user agent, client id, user, user id, login ID, non-person entity (NPE), Token, etc.) of the entity accessing or invoking the service or API (Service / Program Name). This field should contain the identification of the entity (user agent, client id, user, user id, login ID, non-person entity (NPE), Token, etc.) that made the request of the service or API indicated in the Service/Program Name field. For a serving API that is authenticating the request, this should be the authenticated username or equivalent. There are not a concrete set of values for this field. The developer should keep the following set of guidelines when determining what value to use or set for this field.
REF: See PartnerName in v1.3 and (4). | |
| CON-LOG-REQ-MP12 | REQUIRED | Role / Attribute ID | The container and container application MUST log the Role or Attribute ID of the Principal identity of the entity accessing the requested service or API. Note: The group ID is in reference to a Role or Attribute as part of a RBAC or ABAC scheme. | N/A |
CON-LOG-REQ-8 | REQUIRED | Protocol | The container and container application MUST log the field “protocol” in the security audit logs. This refers to the communication mechanism for a request. The value of this field should be representative of the OSI application layer protocol. This is represented as a decimal formatted TCP/IP port number. | |
CON-LOG-REQ-9 | REQUIRED | Service / Program Name | The container and container application MUST log the field “service or program used for access” in the security audit logs. This intention is to capture the service name endpoint or an externally advertised API invoked, e.g., where are you connecting to. This is represented as a URI or URL. | (4) |
| REQUIRED | Log Message | The free text payload of a log event. | (6) |
Best Practices and Risk Analysis for an Operator
<TODO>
Best Practices for operators to collect and correlate logs
<TODO>
Tagging
Muddasar put your thoughts here :Adding metadata or label tags close to log source or by the log source is a good practice. Tags can be added by a local driver for Service and Container ID/name (fqdn) as logs received by logging driver. As ONAP XNF containers will log to stdout/stderr I/O streams, a host or sidecar based collector should be able to add tags for sending source prior to moving the logs to centralized collection location.
As part of log generation other information elements can be added by the application. we should consider what needs to be a requirement: Event_Type (Access, Operation, Error). Logging enahncement project in the past listed format and options, see Logging Enhancements Project Proposal.
Collection
- There currently is a SECCOM proposal that specifies what type of data should be logged where it should be logged to.
- How these logs would be collected and aggregated is specified by the ONAP NextGen Presentation by Byung.
- ONAP Next Generation Security & Logging Architecture#ONAPLogging
- old presentation slide deck (see the above link for the latest on) https://wiki.onap.org/download/attachments/103416997/ONAP-Next-Generation-Security-Logging-2021-5-25-v1.pptx?version=1&modificationDate=1621953519000&api=v2
Proposed Collection of Container Logs
[CON-LOG-REQ-13] The container MUST have security logging for the container and container application active from initialization. [Reference: R-84160]
[CON-LOG-REQ-20] The container and container application MUST use the STDOUT for security logs collection [Reference: REQ-374]
Data Stewardship
What is the data life cycle within ONAP?
What happens to the data as it goes to log stash?
Will it go to AAI?
TODO: Draw out a few scenarios
If there is no consumer it may be written to archive.
Archival data vs live data
Monitoring
- Includes Enrichment, Analysis, and Reporting.
- It is expected that this function out of scope for ONAP. A CSP / MNO will make used of a SIEM. ONAP's role is to provide a means to export security event data. This is where analytics are stored and applied to the data the is ingested from ONAP.
- Presentation by Fabian pertaining to Analysis: ONAP Logs Security Managment1.pptx
Alerting
- Possibly to include mitigation and actions.
- If we expect ONAP to respond to security events in a closed loop manner, then there needs to be a way for events generated by the SIEM to be ingested back into ONAP.
Response
Comments from Chakar, paraphrased, (7/20/2021 SECCOM Meeting)
- We need to disambiguate "Logging" vs "Data Collection".
- Logging from ONAP and Logging from xNF are not the same.
There are two types of responses to consider.
- ONAP responding to a security event in a service.
- ONAP responding to a security event within ONAP. ONAP's ability to respond to itself is only possible in some limited and specific situations. What are these situations?
Terms
This is place where we can standardize our language.
- Security Data: This is raw data that by itself may not be enough to indicate a security event.
- Security Event:
- Analytic
QUESTIONS (Or Advanced Use Cases)
- In terms of security logging, should we handle ONAP components differently than Service Components hosted in ONAP?
Muddasar: Any transections carried out for a service(5G, virtualization and SDN) should generate Application Logs by ONAP Containers. I would think life cycle management of a service (instantiation and changes) may have some information buried in the ONAP logs. Transections events for service design, service deployment within and outside the ONAP components thru ONAP APIs should be part of the ONAP logging. - How do we handle the use case where ONAP is being used to deploy and manage a security infrastructure?
Muddasar: I think it may be similar to above. I think ONAP will not be used to do OAM of security infrastructure, with exception that ONAP may play a role in the instantiation of some of security network elements. Example: A service design may require deployment of FW/IDS/IPS. ONAP transection may be limited to requesting VIMs/EMs to deploy/change network elements and perhaps deploy base configuration. Logs generated by network elements may flow thru a different path(different virtualized enclaves) to a different collector similar to XNFs. - What about security events in regards to the closed loop model? Adversarial AI will be an issue that will need security monitoring in the near future. Does this mean that orchestration / life cycle data from the DCAE needs to ingested by a SIEM?
Muddasar: I believe Data Exposure Service (DES) can provide this facility. I think question here should be that once logs are created, are there any internal ONAP consumers for that information?
References
- https://www.enisa.europa.eu/publications/security-in-5g-specifications
- https://www.enisa.europa.eu/publications/enisa-threat-landscape-report-for-5g-networks
- VNF Requirements List: 9. Requirement List — onap master documentation
- ONAP application1 logging guidelines – Revision 1.0 (4/11/2017
- VNFCloud Readiness Requirements for OpenECOMP
- What to Log - Developer Wiki - Confluence (onap.org)
- Types of EELF Logs - Developer Wiki - Confluence (onap.org)
- Logging Enhancements Project — onap master documentation
Attachments
ONAP Logs Security Management | Logging Source Reference Diagrams |
Proposed Container Logging Requirements | Container Logging Requirements GAP Analysis against ATT&CK |