Control loop flow - current and desired
- Lukasz Grech
- Adam Krysiak
Work in progress
Brief introduction
Definitions and abbreviations
AC = analytic component
CL = control loop
TCA
Holmes
ONAP artifacts descriptions
Describes ONAP artifacts used in various scenarios in this document.
DCAE blueprint
- used in SDC to describe AC
- TOSCA-based (YAML)
- artifact name needs to end with event_proc.bp
- example name: dcae_blueprint.event_proc_bp.yaml
- example name: dcae_blueprint.event_proc_bp.yaml
# # ============LICENSE_START==================================================== # ============================================================================= # Copyright (c) 2018 AT&T Intellectual Property. All rights reserved. # ============================================================================= # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============LICENSE_END====================================================== tosca_definitions_version: cloudify_dsl_1_3 description: > This blueprint deploys/manages the TCA module as a Docker container imports: - http://www.getcloudify.org/spec/cloudify/3.4/types.yaml - https://nexus.onap.org/service/local/repositories/raw/content/org.onap.dcaegen2.platform.plugins/R2/k8splugin/1.1.0/k8splugin_types.yaml - https://nexus.onap.org/service/local/repositories/raw/content/org.onap.dcaegen2.platform.plugins/R2/dcaepolicyplugin/2.3.0/dcaepolicyplugin_types.yaml inputs: aaiEnrichmentHost: type: string default: "aai.onap.svc.cluster.local" aaiEnrichmentPort: type: string default: "8443" enableAAIEnrichment:Micro-service policy type: string default: true dmaap_host: type: string default: message-router dmaap_port: type: string default: "3904" enableRedisCaching: type: string default: false redisHosts: type: string default: dcae-redis:6379 tag_version: type: string default: "nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.tca-cdap-container:1.1.0" consul_host: type: string default: consul-server consul_port: type: stringMicro-service policy default: "8500" cbs_host: type: string default: config-binding-service cbs_port: type: string default: "10000" policy_id: type: string default: "none" external_port: type: string description: Kubernetes node port on which CDAPgui is exposed default: "32011" node_templates: tca_k8s: type: dcae.nodes.ContainerizedServiceComponent relationships: - target: tca_policy type: cloudify.relationships.depends_on properties: service_component_type: dcaegen2-analytics-tca application_config: {} docker_config: {} image: get_input: tag_version log_info: log_directory: "/opt/app/TCAnalytics/logs" application_config: app_config: appDescription: DCAE Analytics Threshold Crossing Alert Application appName: dcae-tca-ak-serv tcaAlertsAbatementTableName: TCAAlertsAbatementTable tcaAlertsAbatementTableTTLSeconds: '1728000' tcaSubscriberOutputStreamName: TCASubscriberOutputStream tcaVESAlertsTableName: TCAVESAlertsTable tcaVESAlertsTableTTLSeconds: '1728000' tcaVESMessageStatusTableName: TCAVESMessageStatusTable tcaVESMessageStatusTableTTLSeconds: '86400' thresholdCalculatorFlowletInstances: '2' app_preferences: aaiEnrichmentHost: get_input: aaiEnrichmentHost aaiEnrichmentIgnoreSSLCertificateErrors: 'true' aaiEnrichmentPortNumber: '8443' aaiEnrichmentProtocol: https aaiEnrichmentUserName: DCAE aaiEnrichmentUserPassword: DCAE aaiVMEnrichmentAPIPath: /aai/v11/search/nodes-query aaiVNFEnrichmentAPIPath: /aai/v11/network/generic-vnfs/generic-vnf enableAAIEnrichment: get_input: enableAAIEnrichment enableRedisCaching: get_input: enableRedisCaching redisHosts: get_input: redisHosts enableAlertCEFFormat: 'false' publisherContentType: application/json publisherHostName: get_input: dmaap_host publisherHostPort: get_input: dmaap_port publisherMaxBatchSize: '1' publisherMaxRecoveryQueueSize: '100000' publisherPollingInterval: '20000' publisherProtocol: http publisherTopicName: unauthenticated.DCAE_CL_OUTPUT subscriberConsumerGroup: clamp subscriberConsumerId: c12 subscriberContentType: application/json subscriberHostName: get_input: dmaap_host subscriberHostPort: get_input: dmaap_port subscriberMessageLimit: '-1' subscriberPollingInterval: '30000' subscriberProtocol: http subscriberTimeoutMS: '-1' subscriberTopicName: unauthenticated.VES_MEASUREMENT_OUTPUT tca_policy: '{ \"domain\": \"measurementsForVfScaling\", \"metricsPerEventName\": [ { \"eventName\": \"vFirewallBroadcastPackets\", \"controlLoopSchemaType\": \"VNF\", \"policyScope\": \"DCAE\", \"policyName\": \"DCAE.Config_tca-hi-lo\", \"policyVersion\": \"v0.0.1\", \"thresholds\": [ { \"closedLoopControlName\": \"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\", \"version\": \"1.0.2\", \"fieldPath\": \"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\", \"thresholdValue\": 300, \"direction\": \"LESS_OR_EQUAL\", \"severity\": \"MAJOR\", \"closedLoopEventStatus\": \"ONSET\" }, { \"closedLoopControlName\": \"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\", \"version\": \"1.0.2\", \"fieldPath\": \"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\", \"thresholdValue\": 700, \"direction\": \"GREATER_OR_EQUAL\", \"severity\": \"CRITICAL\", \"closedLoopEventStatus\": \"ONSET\" } ] }, { \"eventName\": \"Measurement_vGMUX\", \"controlLoopSchemaType\": \"VNF\", \"policyScope\": \"DCAE\", \"policyName\": \"DCAE.Config_tca-hi-lo\", \"policyVersion\": \"v0.0.1\", \"thresholds\": [ { \"closedLoopControlName\": \"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\", \"version\": \"1.0.2\", \"fieldPath\": \"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\", \"thresholdValue\": 0, \"direction\": \"EQUAL\", \"severity\": \"MAJOR\", \"closedLoopEventStatus\": \"ABATED\" }, { \"closedLoopControlName\": \"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\", \"version\": \"1.0.2\", \"fieldPath\": \"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\", \"thresholdValue\": 0, \"direction\": \"GREATER\", \"severity\": \"CRITICAL\", \"closedLoopEventStatus\": \"ONSET\" } ] } ]}' service_component_type: dcaegen2-analytics_tca interfaces: cloudify.interfaces.lifecycle: start: inputs: envs: DMAAPHOST: { get_input: dmaap_host } DMAAPPORT: { get_input: dmaap_port } DMAAPPUBTOPIC: "unauthenticated.DCAE_CL_OUTPUT" DMAAPSUBTOPIC: "unauthenticated.VES_MEASUREMENT_OUTPUT" AAIHOST: { get_input: aaiEnrichmentHost } AAIPORT: { get_input: aaiEnrichmentPort } CONSUL_HOST: { get_input: consul_host } CONSUL_PORT: { get_input: consul_port } CBS_HOST: { get_input: cbs_host } CBS_PORT: { get_input: cbs_port } CONFIG_BINDING_SERVICE: "config_binding_service" ports: - concat: ["11011:", { get_input: external_port }] tca_policy: type: dcae.nodes.policy properties: policy_id: get_input: policy_id
Micro-service schema
Schema that can be used by Tosca Lab (SDC DCAE-D Tosca Lab) to generate Micro-service policy model (more info: MicroServices Onboarding in ONAP)
{ "self": { "name": "cdap.tca.hi.lo.app", "version": "1.0.5", "description": "cdap tca hi lo application", "component_type": "cdap" }, "streams": { "subscribes": [ { "config_key": "tca_handle_in", "format": "VES_specification", "version": "5.28.4", "type": "message router" } ], "publishes": [ { "config_key": "tca_handle_out", "format": "DCAE_CL_Output", "version": "1.0.1", "type": "message router" } ] }, "services": { "calls": [], "provides": [] }, "auxilary": { "streamname": "TCASubscriberOutputStream", "artifact_name": "dcae-analytics-cdap-tca", "artifact_version": "2.0.0", "namespace": "cdap_tca_hi_lo", "programs": [ { "program_type": "flows", "program_id": "TCAVESCollectorFlow" }, { "program_type": "workers", "program_id": "TCADMaaPMRSubscriberWorker" }, { "program_type": "workers", "program_id": "TCADMaaPMRPublisherWorker" } ] }, "artifacts": [ { "uri": "https://nexus.onap.org/service/local/repositories/staging/content/org/onap/dcaegen2/analytics/tca/dcae-analytics-cdap-tca/2.0.0/dcae-analytics-cdap-tca-2.0.0.jar", "type": "jar" } ], "parameters": { "app_config": [ { "name": "appName", "description": "CDAP Application Name for TCA App", "value": "dcae-tca" }, { "name": "appDescription", "description": "CDAP Application Description for TCA App", "value": "DCAE Analytics Threshold Crossing Alert Application" }, { "name": "tcaSubscriberOutputStreamName", "description": "Name of Stream where TCA DMaaP Subscriber pushes incoming messages to TCA App for processing", "value": "TCASubscriberOutputStream" }, { "name": "thresholdCalculatorFlowletInstances", "description": "Number of instances for Threshold Calculator", "value": 2 }, { "name": "tcaVESMessageStatusTableName", "description": "TCA Message Status Table Name where status of each TCA message is stored", "value": "TCAVESMessageStatusTable" }, { "name": "tcaVESMessageStatusTableTTLSeconds", "description": "Time to Live for entries in Message Status Table", "value": 86400 }, { "name": "tcaVESAlertsTableName", "description": "Table where TCA generated alerts are temporarily stored before published to DMaaP MR topic", "value": "TCAVESAlertsTable" }, { "name": "tcaVESAlertsTableTTLSeconds", "description": "Time to Live for entries in Alerts Table", "value": 1728000 }, { "name" : "tcaAlertsAbatementTableName", "description" : "Table where information to determine sending of Abatement Alerts is stored", "value" : "TCAAlertsAbatementTable" }, { "name" : "tcaAlertsAbatementTableTTLSeconds", "description": "Time to Live for entries in Alerts Abatement Table", "value" : 1728000 } ], "app_preferences": [ { "name" : "subscriberHostName", "description" : "", "value" : "HOSTNAME" }, { "name" : "subscriberHostPort", "description" : "", "value" : 1234 }, { "name" : "subscriberTopicName", "description" : "", "value" : "com.att.dcae.dmaap.mtnje2.DcaeTestVESSub" }, { "name" : "subscriberProtocol", "description" : "", "value" : "https" }, { "name" : "subscriberUserName", "description" : "", "value" : "USERNAME" }, { "name" : "subscriberUserPassword", "description" : "", "value" : "PASSWORD" }, { "name": "subscriberContentType", "description": "Subscriber Content Type", "value": "application/json" }, { "name": "subscriberConsumerId", "description": "Subscriber Consumer Id", "value": "c12" }, { "name": "subscriberConsumerGroup", "description": "Subscriber Group Id", "value": "OpenDCAE-c12" }, { "name": "subscriberTimeoutMS", "description": "Subscriber timeout in milliseconds", "value": -1 }, { "name": "subscriberMessageLimit", "description": "Subscriber max message fetch limit", "value": -1 }, { "name": "subscriberPollingInterval", "description": "Subscriber Polling Interval in milliseconds", "value": 30000 }, { "name" : "publisherHostName", "description" : "", "value" : "HOSTNAME" }, { "name" : "publisherHostPort", "description" : "", "value" : 1234 }, { "name" : "publisherTopicName", "description" : "", "value" : "com.att.dcae.dmaap.mtnje2.DcaeTestVESPub" }, { "name" : "publisherProtocol", "description" : "", "value" : "https" }, { "name" : "publisherUserName", "description" : "", "value" : "USERNAME" }, { "name" : "publisherUserPassword", "description" : "", "value" : "PASSWORD" }, { "name": "publisherContentType", "description": "Publisher Content Type", "value": "application/json" }, { "name": "publisherMaxBatchSize", "description": "Publisher max batch size", "value": 1 }, { "name": "publisherMaxRecoveryQueueSize", "description": "Publisher Max Recovery Queue Size", "value": 100000 }, { "name": "publisherPollingInterval", "description": "Publisher Polling Interval in milliseconds", "value": 20000 }, { "name": "enableAlertCEFFormat", "description": "Enable output alert to be in CEF Format", "value": "false" }, { "name": "enableAAIEnrichment", "description": "Enable A&AI Enrichment of generated TCA Alerts", "value": "true" }, { "name": "aaiEnrichmentHost", "description": "A&AI Enrichment API Host name", "value": "123.123.123.123" }, { "name": "aaiEnrichmentPortNumber", "description": "A&AI Enrichment API Host port number", "value": 8443 }, { "name": "aaiEnrichmentProtocol", "description": "A&AI Enrichment API protocol", "value": "https" }, { "name": "aaiEnrichmentUserName", "description": "A&AI Enrichment API username", "value": "USERNAME" }, { "name": "aaiEnrichmentUserPassword", "description": "A&AI Enrichment API password", "value": "PASSWORD" }, { "name": "aaiEnrichmentIgnoreSSLCertificateErrors", "description": "A&AI Ignore SSL Certification Errors", "value": "true" }, { "name": "aaiVNFEnrichmentAPIPath", "description": "A&AI VNF Enrichment API Path", "value": "/aai/v11/network/generic-vnfs/generic-vnf" }, { "name": "aaiVMEnrichmentAPIPath", "description": "A&AI VM Enrichment API Path", "value": "/aai/v11/search/nodes-query" }, { "name": "aaiEnrichmentProxyURL", "description": "A&AI Enrichment API Proxy URL", "value": "URL" }, { "name": "tca_policy", "description": "TCA Policy JSON", "value":"{\"domain\":\"measurementsForVfScaling\",\"metricsPerEventName\":[{\"eventName\":\"vFirewallBroadcastPackets\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":300,\"direction\":\"LESS_OR_EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ONSET\"},{\"closedLoopControlName\":\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":700,\"direction\":\"GREATER_OR_EQUAL\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]},{\"eventName\":\"vLoadBalancer\",\"controlLoopSchemaType\":\"VM\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.vNicUsageArray[*].receivedTotalPacketsDelta\",\"thresholdValue\":300,\"direction\":\"GREATER_OR_EQUAL\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]},{\"eventName\":\"Measurement_vGMUX\",\"controlLoopSchemaType\":\"VNF\",\"policyScope\":\"DCAE\",\"policyName\":\"DCAE.Config_tca-hi-lo\",\"policyVersion\":\"v0.0.1\",\"thresholds\":[{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"EQUAL\",\"severity\":\"MAJOR\",\"closedLoopEventStatus\":\"ABATED\"},{\"closedLoopControlName\":\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\",\"version\":\"1.0.2\",\"fieldPath\":\"$.event.measurementsForVfScalingFields.additionalMeasurements[*].arrayOfFields[0].value\",\"thresholdValue\":0,\"direction\":\"GREATER\",\"severity\":\"CRITICAL\",\"closedLoopEventStatus\":\"ONSET\"}]}]}", "policy_editable": true, "policy_schema": [{ "name": "domain", "description": "Domain name to which TCA needs to be applied", "type": "string", "value": "measurementsForVfScaling", "constraints": [{ "equal": "measurementsForVfScaling" }] }, { "name": "metricsPerEventName", "description": "Contains eventName and threshold details that need to be applied to given eventName", "type": "list", "entry_schema": [{ "name": "eventName", "description": "Event name to which thresholds need to be applied", "type": "string", "value": "" }, { "name": "controlLoopSchemaType", "description": "Specifies Control Loop Schema Type for the event Name e.g. VNF, VM", "type": "string", "value": "", "constraints": [{ "valid_values": ["VM", "VNF"] }] }, { "name": "policyScope", "description": "TCA Policy Scope", "type": "string", "value": "" }, { "name": "policyName", "description": "TCA Policy Scope Name", "type": "string", "value": "" }, { "name": "policyVersion", "description": "TCA Policy Scope Version", "type": "string", "value": "" }, { "name": "thresholds", "description": "Thresholds associated with eventName", "type": "list", "entry_schema": [{ "name": "closedLoopControlName", "description": "Closed Loop Control Name associated with the threshold", "type": "string", "value": "" }, { "name": "version", "description": "Version number associated with the threshold", "type": "string", "value": "" }, { "name": "fieldPath", "description": "Json field Path as per CEF message which needs to be analyzed for TCA", "type": "string", "value": "" }, { "name": "thresholdValue", "description": "Threshold value for the field Path inside CEF message", "type": "number", "value": "" }, { "name": "direction", "description": "Direction of the threshold", "type": "string", "value": "", "constraints": [{ "valid_values": [ "LESS", "LESS_OR_EQUAL", "GREATER", "GREATER_OR_EQUAL", "EQUAL" ] }] }, { "name": "closedLoopEventStatus", "description": "Closed Loop Event Status of the threshold", "type": "string", "value": "", "constraints": [{ "valid_values": ["ONSET", "ABATED"] }] }, { "name": "severity", "description": "Threshold Event Severity", "value": "", "type": "string", "constraints": [{ "valid_values": [ "CRITICAL", "MAJOR", "MINOR", "WARNING", "NORMAL" ] }] }] }] }] } ], "program_preferences": [] } }
Micro-service policy model
Model generated by tosca lab. It's currently not used. It could be translated into Micro-service policy schema.
tosca_definitions_version: tosca_simple_yaml_1_0_0 data_types: policy.data.metricsPerEventName: properties: controlLoopSchemaType: type: string description: Specifies Control Loop Schema Type for the event Name e.g. VNF, VM constraints: - valid_values: - VM - VNF eventName: type: string description: Event name to which thresholds need to be applied policyName: type: string description: TCA Policy Scope Name policyScope: type: string description: TCA Policy Scope policyVersion: type: string description: TCA Policy Scope Version thresholds: type: list description: Thresholds associated with eventName entry_schema: type: policy.data.thresholds policy.data.tca_policy: properties: domain: type: string description: Domain name to which TCA needs to be applied constraints: - equal: measurementsForVfScaling metricsPerEventName: type: list description: Contains eventName and threshold details that need to be applied to given eventName entry_schema: type: policy.data.metricsPerEventName policy.data.thresholds: properties: closedLoopControlName: type: string description: Closed Loop Control Name associated with the threshold closedLoopEventStatus: type: string description: Closed Loop Event Status of the threshold constraints: - valid_values: - ONSET - ABATED direction: type: string description: Direction of the threshold constraints: - valid_values: - LESS - LESS_OR_EQUAL - GREATER - GREATER_OR_EQUAL - EQUAL fieldPath: type: string description: Json field Path as per CEF message which needs to be analyzed for TCA severity: type: string description: Threshold Event Severity constraints: - valid_values: - CRITICAL - MAJOR - MINOR - WARNING - NORMAL thresholdValue: type: integer description: Threshold value for the field Path inside CEF message version: type: string description: Version number associated with the threshold node_types: policy.nodes.Root: derived_from: tosca.nodes.Root properties: policyDescription: required: false type: string policyName: required: true type: string policyScope: required: true type: string policyVersion: required: true type: string policy.nodes.cdap.tca.hi.lo.app: derived_from: policy.nodes.Root properties: tca_policy: type: map description: TCA Policy JSON entry_schema: type: policy.data.tca_policy
Micro-service policy schema
Schema that currently can by used by Policy to generate form to configure Micro-service policy
{
"MyTestModelNameConfiguration": {
"param1": "java.lang.String:defaultValue-NA:required-true:MANY-false",
"param2": "java.lang.String:defaultValue-NA:required-false:MANY-true",
"param3": "java.lang.String:defaultValue-NA:required-false:MANY-false",
"param4": "java.lang.String:defaultValue-NA:required-false:MANY-true",
"param5": "java.lang.String:defaultValue-NA:required-false:MANY-true",
"param6": "java.lang.String:defaultValue-NA:required-false:MANY-false"
}
}
Micro-service policy (instance)
Configured micro-service policy. In this case Clamp is creating it using Policy API.
e.g. in Beijing TCA configuration Micro-service policy contains information about metrics and tresholds
Operational policy (instance)
Policy with configured Drools Rule. In this case Clamp is creating it using Policy API.
e.g. in Beijing this policy decided whether to start APPC LCM or not.
Current control loop design and execution
Stages required to run control loop:
AC blueprint design (SDC)
AC blueprint distribution (SDC)
CL configuration (CLAMP)
CL submit (CLAMP)
CL deploy (CLAMP)
Desired control loop design and execution
Stages required to run control loop:
Deployment Artifact design using SDC DCAE-D
In this step user should design and add to service 2 deployment artifacts:
AC blueprint (DCAE_BLUEPRINT)
Policy model (example: policy.yaml)
Distribution of CSAR containing Policy model and AC blueprint (SDC)
- Both Policy and Clamp should receive Artifact distribution notification.
- Both Policy and Clamp should receive Artifact distribution notification.
AC configuration (CLAMP)
- Policy should extract logic of creating forms for configuring Policy using micro-service policy schema (in this confluence it's called Policy SDK)
- Clamp should use Policy SDK to create generic form out of saved micro-service policy schema.
Action Policy configuration (CLAMP)
CL submit (CLAMP)
CL deploy (CLAMP)