Backup *** Edge Scoping MVP for Casablanca - ONAP Enhancements *** Backup

Summary: Edge Architecture & Work Items#ONAPEdgeMVP 

Distributed Edge Cloud Infrastructure Object Hierarchy (Stretch Goal)

Value:

• Fine grained resource management & analytics for Distributed Edge Clouds 

References: 

• Infrastructure Modelling: ONAP R3+ Cloud Infrastructure Modeling; Cloud Infrastructure Aggregate Representation Classes

https://lf-onap.atlassian.net/browse/MULTICLOUD-153

Assumption for Policy, SO, OOF:

• This uses the current Generic VNF workflow in SO

Note 1: 

• Configured Capacity and Utilized (or Currently Used) Capacity are managed by the specific cloud.

Note 2:

• Cloud SW Capability example 

  • Cloud region "x" with SR-IOV, GPU, Min-guarantee support

  • Cloud region "y" with SR-IOV support

• Cloud HW Capability example 

  • Resource cluster "xa" in Cloud region "x" with SR-IOV and GPU support 

  • Resource cluster "xb" in Cloud region "x" with GPU support

  • Resource cluster "ya" in Cloud region "y" with SR-IOV support

Note 3:

• 5G Service/VNF placement example

  • Constraints used by Optimization Framework (OOF)

    • 5G UPF VNF location to be fixed to a specific physical DC based on 5G DU, bounded by a max distance from 5G DU

  • Optimization Policy used by OOF

    • Choose optimized cloud region (or instance) for the placement of 5G UPF for subscriber group based on the above constraints

Note 4:

• For the 5G Service/VNF placement example in Note 3

  • 5G CU-UP VNF preferably maps to a specific Cloud region & Physical DC End Point 

Note 5:

• For the 5G Service/VNF placement example in Note 3

  • OOF will pass the Physical DC End Point to SO as a opaque data

Note 6:

• For the 5G Service/VNF placement example in Note 3

  • SO passes the Physical DC End Point to Multi-Cloud as a opaque data, besides the Cloud Region

Cloud-agnostic Placement/Networking & Homing Policies (Phase 1 - Casablanca MVP, Phase 2 - Stretch Goal)

End-to-end use case Applicability:

• All (especially the data plane VNFs with fine-grained VNF placement and high performance networking requirements)

Value:

• Improve "workload deployability" by avoiding exposure of "cloud specific" capabilities to several ONAP components and addressing "separation of concerns" 

• Applicable to all workloads - VM-based or Container-based

https://lf-onap.atlassian.net/browse/MULTICLOUD-272

Phase 1 (Casablanca MVP) Summary:

• Multi-Cloud Policy Framework

  • Assist OOF in target cloud region selection for VNF placement (aka homing) through intent

    • Cloud Agnostic Intent (Policy) Execution Workflow  - Steps 1- 6

  • Dynamically modify the cloud specific VNF deployment template based on cloud-specific realization of the specified intent (e.g. Infra HA for VMs within a VNF could have different realizations across different clouds)

    • Cloud Agnostic Intent (Policy) Execution Workflow  - Step 7

• Intent Support

  • Single realization option per Cloud Region for the specified Intent

• Impact Projects:

  • Multi-Cloud (Highest), OOF, SO (Minimal)

• End-to-end use case demonstration:

  • vCPE (higher priority – no additional implementation dependency), vDNS

• Types of intent supported (through OOF Policy)

  • "Infrastructure High Availability for VNF" 

  • "Infrastructure Resource Isolation for VNF": "Burstable QoS" 

  • "Infrastructure Resource Isolation for VNF": "Guaranteed QoS"

Phase 2 (Casablanca Stretch Goal) Summary (Build on Phase 1 Work):

• Multi-Cloud Policy Framework

  • Dynamically modify the cloud specific VNF deployment template based on cloud-specific realization of the specified intent – Impact to VNF configuration 

    • E.g. High performance Intra-DC data plane networking with several realization choices

• Intent Support

  • Multiple realization options per Cloud Region for the specified Intent

• Major Impact Projects:

  • Multi-Cloud

• Minor Impact Projects:

  • SO, OOF, GNF Controller

• Wiki Link:

  • Edge Scoping - Casablanca Stretch Goal/Beyond Casablanca

References: 

 The sequence diagram below expands "Multi-Cloud/VNFM Deploy Apps" in Edge Architecture & Work Items Sequence Diagram

Cloud Agnostic Intent (Policy) Workflow Summary (Phase 1 - Casablanca MVP):

Cloud Agnostic Intent (Policy) Workflow Details (Phase 1 - Casablanca MVP):

Private Cloud Setup - OpenStack-based

• Pre-defined (including custom) flavors map to Instance types in Public Clouds

  • Pre-defined flavors are created by the Cloud Admin before the Cloud is used by ONAP for workload deployment

• VMware VIO Configuration for Min Guarantee feature

  • Ref: https://docs.vmware.com/en/VMware-Integrated-OpenStack/5.0/com.vmware.openstack.admin.doc/GUID-D2C17D33-EC6C-4B8C-A9A3-25417917461F.html

  • Ref: https://docs.openstack.org/horizon/latest/admin/manage-flavors.html

• Create necessary tenants per <cloud owner, cloud region>

  • Mapping of VNFC, VNF (VF module), Service (e.g. vCPE) to the corresponding tenant happens in the respective Multi-Cloud plugin.

VNFC to Instance Type Mapping

• One or more VNFCs (e.g. vCPE VGW) could map to an Instance Type

  • Use Case: Residential Broadband vCPE (Approved)

    • Ref: https://github.com/onap/demo/blob/a19bd423c1e4d2f5552bab090ba2650c38fa7192/heat/vCPE/vgw/base_vcpe_vgw.yaml

    • Ref: https://github.com/onap/demo/blob/ab98f6a0db3df8f5232395d899e96462dbc11159/heat/vCPE/vgw/base_vcpe_vgw.env

  • OpenStack-based Clouds

    • Instance type maps to pre-defined Flavors

  • Microsoft Azure

    • Pre-defined Instance Types

      • Ref: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/sizes?toc=%2fazure%2fvirtual-machines%2flinux%2ftoc.json

Step 1. SO → OOF - Get Target <Cloud Owner, Cloud Region> for the Service Instances

Step 2. OOF → Policy - Fetch Cloud Selection Policy for Homing 

2a) OOF Processing - the fetched Policy (example below) is stored in a local data structure and is available for further use (need OOF code changes).

OOF Homing Enhanced Cloud Selection Policy -- JSON Schema with Use Case Examples as runnable python code:

Step 3. OOF → A&AI - Fetch Cloud-Agnostic (Standardized) Capabilities for the Service Instance

3a) OOF Processing - Perform Cloud Agnostic Capability check for each <cloud owner, cloud region>. OOF will prune any <cloud owner, cloud region> which is not satisfying the standardized capabilities.

Step 4, OOF --> MC evaluate cost of deployment (aligned to the SO/MC API defined by SO Casablanca HPA Design to minimize the terminology set)

*** Assume: VNFC is equalivent to VM, VNF is equivalent of heat stack.

Step 4. OOF → MC - Push Cloud Agnostic Policy for the Service Instance 

4a) OOF Processing

The OOF ↔ MC cloud selection API, described below, is filled based on the Cloud Selection Policy for Homing retrieved in step 2) – need OOF code changes. 

OOF <-> MC Cloud Selection API -- JSON Schema with Use Case Examples as runnable python code:

MC Workload Deployment Cost Policy -- JSON Schema with Use Case Examples as runnable python code:

5a) MC Processing (need MC code changes)

For each cloud owner

• Parse OOF → MC Policy (Intent) API 

• If a Cloud owner does not support a specific "deployment-intent"

  • Drop all the cloud regions for the cloud owner from the candidate list

• For each cloud region // Public cloud could have different costs in different geographic locations

  • Compute net_value based on cost

    • net_value = net_value + workload_deployment_cost 

      • If Plugin of cloud owner supports cost based on "dollarCostEvaluationVM-Type" and/or "dollarCostEvaluationVM-FeatureGroup"

        • The workload deployment cost is computed per <instance type, cloud region> based on workload deployment cost policy described in Step 5b).

          • Instance Type is derived from <Service, VNFC, cloud owner>

          • More details are in 5b)

        • Implementation Notes:

          • It is not mandatory for all plugins to implement this feature since the OOF → MC API has the flexibility of turning on this feature per <cloud owner, cloud region>

      • Else

        • The workload deployment cost is computed as a fixed cost per plugin

5b) Workload Deployment Cost Policy - Configured by the Operator

The operator/service provider who uses ONAP will choose which VIMs to use and include the appropriate MC plugins in his ONAP deployment. For example, let’s assume they pick private Openstack, private VMWare, and public Azure as the platform to run their services on.

For R3, Workload Deployment Cost Policy can be stored in the form of configuration file(s) in the OOM K8S Persistent Volumes visible to the relevant MC plugin to simplify implementation.  Beyond R3, this could be moved to the Policy DB. The details of the configuration are described below.

• By default, each plugin supports a fixed cost for all workloads

  • Optionally, plugin of cloud owner can support cost based on "dollarCostEvaluationVM-Type" and/or "dollarCostEvaluationVM-FeatureGroup"

    • Where workload deployment cost includes dollar cost of VM Instance Type (based on <Service, VNFC, cloud owner>) and dollar cost (or discount) of other cloud-specific feature groups corresponding to the intent expressed under the deployment-intent keyword in the OOF → MC API

      • As an example, with respect to the deployment-intent, "Infrastructure Resource Isolation for VNF" with "Burstable QoS" can yield potential cost savings as compared to "Guaranteed QoS" by allowing smart over-subscription while still guaranteeing isolation

• Note that the operator is free to choose the method of calculating the cost which includes initial cost, support cost & operational cost. 

• Note that the operator is free to choose what time duration the cost metric is specified for each of the MultiVIM plugins (e.g., cost per hour, cost per month) since they will do it consistently for each of the VIMs. 

"Workload Deployment Cost Policy Example" depicted above has an exemplary description of this.

Step 5. MC →  OOF – Return a net value for each <cloud owner, cloud region> 

6a) OOF Processing - cloud_net_value input in Multi-objective Optimization (need OOF code changes)

Casablanca Goal for implementation simplification

Stretch Goal for Casablanca

Step 6. OOF → SO - Return the target <cloud owner, cloud region> for the Service Instance + deployment-intent per vnfc

OOF ↔ SO API extension (VNFC deployment-intent) -- identical in content to SO <-> MC Policy API

Step 7. SO → MC - Deploy VNF template in the target <cloud owner, cloud region> for the Service Instance

7) MC Processing (need MC code changes)

SO ↔ MC API extension - aligned to the SO/MC API defined by SO Casablanca HPA Design to minimize the terminology set

(This data is sent from OOF to SO. SO transparently echoes this data to MC)

Follow ups:

• Use Cases for Integration testing

  • vCPE

    • In the current state, this use case cannot support the intent "Infra HA for VMs in a VNF"

    • This use case has been tested in R2 with OOF↔MC capacity check API

  • vDNS 

    • Can support intent "Infra HA for VMs in a VNF" and "Infrastructure Resource Isolation for VNF"

    • Nothing additional needed in OOF or MC

    • Changes needed in SO to call OOF API

      • Marcus from Intel is driving this

• Policy DB – is there any restriction on the type of json objects that can be stored?

  • Matti to follow up with Ankit

Implementation trade offs for Casablanca (R3) and potential Dublin (R4) plan:

• Deployment-Intent 

  • 1. "Infrastructure Resource Isolation for VNF" – { "qosProperty": { {"Burstable QoS": "TRUE", "Burstable QoS Oversubscription Percentage": "25"} } }

    • Casablanca Plan

      • Only certain pre-defined over-subscription values are allowed to reflect practical deployment and simplify implementation 

    • Dublin & Beyond Potential Plan

      • Creating instance types on demand for private clouds - to study

  • 2. Cloud-agnostic Workload Deployment Policy (Intent) 

    • Casablanca Plan

      • Cloud-Agnostic Workload Deployment Policy (Intent) can be directly mapped to specific realization (e.g. OpenStack Flavor, Azure Instance Type) to simplify implementation.  

    • Dublin & Beyond Potential Plan

      • VIM Capability Discovery to populate Intent in A&AI aligning taking into account Cloud selection policy based on cost specific to Intent (leverage similarities to HPA label discovery supported since R2)

        • VIM selection – Intent to be populated in A&AI for capability matching 

        • VIM Deployment realization - Intent(s) to specific realization mapping (e.g. OpenStack Flavor, Azure Instance Type) to be populated in A&AI 

• Policy-based & cloud-selection

  • 3. Tenant Information is not passed in the OOF → MC API

    • Casablanca Plan

      • The tenant information is derived from a simple mapping function per <cloud owner, cloud region>

        • A simple mapping would be a tenant per <cloud owner, cloud region> as part of Multi-VIM plugin configuration.

        • Need to make sure that this scheme is synchronous with the SO → MC API path

    • Dublin & Beyond Potential Plan

      • Pass Tenant Information per <cloud owner, cloud region> in the OOF → MC API

  • 4. VM Instance type/VM Feature Group dollar-cost-based cloud selection 

    • Casablanca Plan

      • By default, the workload deployment cost is computed as a fixed cost per plugin

        • VM Instance type/VM Feature Group dollar-cost-based cloud selection is optional for all Multi-Cloud Plugins

    • Dublin & Beyond Potential Plan

      • Deep dive further on dollar-cost-based cloud selection models/implementation for public/private clouds 
        
        

Cloud Resource Partitioning for Differentiated QoS (Combined with Previous)

Value:

• Applicable to all use cases

• Casablanca Targets:

  • vCPE (Enable Tiered service offering); 5G Network Slicing (Stretch Goal) 

References:

• https://lf-onap.atlassian.net/wiki/download/attachments/16277535/ONAP-mc-of-resource-reservation.pptx?version=1&modificationDate=1526771712000&api=v2

Edge Automation Requirement:

Support three types of slices in the Cloud Infrastructure (Definition Reference: https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/)

• Guaranteed Resource Slice (hard isolation) for various infra Resources (CPU/Memory/Network)

  • Max (limit), Min (request) are the same; resource guarantee is "Max"

  • Maps to 5G Applications such as Connected Car which fall in the category of ultra-reliable machine-type communications (ref. 1)

• Burstable Resource Slice (soft isolation) for various infra Resources

  • Min (request) <= Max (limit); resource guarantee is "Min"

  • Maps to Burstable Network Slice such > 1Gbps broadband which fall in the category of extreme mobile broadband (ref. 1)

• Best Effort Resource Slice (no isolation) for various infra Resources

  • No Min (request) ; resource guarantee is "None"

  • Maps to 5G Applications such as IoT which fall in the category of massive machine-type communications (ref. 1)

Implementation:

• Leverage current HPA framework with appropriate extensions

References:

• https://metis-ii.5g-ppp.eu/wp-content/uploads/white_papers/5G-RAN-Architecture-and-Functional-Design.pdf

• Driving Superior Isolation for Tiered Services using Resource Reservation -- Optimization Policies for Residential vCPE
  -https://jira.onap.org/browse/OPTFRA-240

Note:

• Any VMs/Containers which are part of a resource slice will adhere to the specs of the resource slice

Aggregated Infrastructure Telemetry Streams (Aligns with HPA requirements, Combining efforts with HPA)

Value

• Edge Infrastructure Analytics complementing VNF Analytics

• Increase the accuracy of placement decisions

• Addresses gap in cloud provider solution – e.g. open source OpenStack does not have a comprehensive telemetry solution

Casablanca MVP

• HPA metrics visualization

• End-to-end use cases: vCPE, vDNS

Casablanca Stretch Goal

• OOF to use aggregated telemetry information for fine-grained optimization

https://lf-onap.atlassian.net/browse/MULTICLOUD-254

ONAP, as in R2, collects the statistics/alarms/events from workloads (VMs) and take any close loop control actions such as Heal a process, scale-out, restart etc.. In R3 and beyond, infrastructure related statistics/alarms/events will be collected, generate actionable insights and take life cycle actions on the workloads.  Infrastructure statistics normally include performance counters, NIC counters, IPMI information on per physical server node basis.  To reduce the load on the ONAP, it is necessary that aggregated (summarized) information is sent to the ONAP from edge-clouds. 

As part of this activity, intention is to create aggregation micro-service that collects the data from physical nodes (over collected and other mechanisms), aggregate the information (time based aggregation, threshold based aggregation, silencing etc.,..) based on the configurable rules and export the aggregate data to DCAE.  This micro service can be instantiated by ONAP itself - one or more instances for edge-clouds at the ONAP-central itself using OOM, it could be instantiated at the edge-cloud using their own deployment tools or it could be deployed edge service providers at the regional site level.  

In R3, functionality is limited to HPA features and visualization.  R3 stretch goal: It collects information from each compute node for all HPA features and keeps track of health and resource information. It would use this information in placement decisions by OOF for accurate results.

Even though the aggregation service is being developed in Multi-Cloud project, it is expected that this can be deployed at various places. The decision to deploy at various levels can be due to performance and regulatory reasons.  Following deployments are envisaged at this time:

• At the edge site level.

• At the regional site level (on behalf of set of edge sites).

• At the ONAP level (on behalf of set of edge sites)

Impacted projects (development activities)