Overview

MEF LSO defines a set of specifications and reference points aimed at providing end to end service orchestration across multiple network domains using standardized APIs. One of the reference points in this set is interlude which focuses on providing control related management interactions between service provider and partner (link). The other inter-provider reference point in LSO is SONATA which mainly focuses on the OSS/BSS level business interactions. As ONAP is not specifically on the SONATA level of interactions, the rest of the document focuses mainly on the interlude reference point in MEF and similar specifications in other standard organizations. While interlude is one of the reference point and specification which addresses inter-provider interaction, it is worthwhile to look at the broader scope considering typical operational, business use cases and aspects impacting such interactions in ONAP External API specifically and ONAP in general. MEF Interlude scope is covered in detail in a separate section.

5GExchange (link) as part of 5GPPP is one of the relevant project worth referring which focuses on "cross-domain orchestration of services over multiple administrations or over multi-domain single administrations" (link). 5GEx project defines a multi-domain logical interworking architecture which covers multi-operator interaction and multi-domain interaction within the same operator. As part of the 5GEx project a detailed study is being conducted around inter-domain and inter-provider interactions and results are published here. The 5GEx proposed system consists of multi-domain orchestrator (MdO), domain orchestrators and interactions between MdO (marked as #2 in the diagram below), the interaction between MdO and Domain orchestrators (marked as #3 in the diagram), the interaction between customer and MdO (marked as #1), interaction between Domain Orchestrators and controllers (marked as #5) and interaction between domain orchestrators (marked as #4). Each of these interactions is identified by different types of interfaces. There is also a classification based on business level interactions, management/orchestration level interaction, control level interactions, and data level interaction. Out of these #2 is the one which close matches the MEF interlude reference point, but the scope is slightly different because in 5GEx project business, management/orchestration related interactions are expected to be handled by the same interface (i.e #2). While #3 (between MdO and DO) is quite relevant in the case of External API, this may be an item for future study as domain orchestration concept is currently under discussion in ONAP Tiger team as of September 2018. For the sake of this document, MdO is functionally mapped to the External API component in ONAP as it is providing an end to end service management capability.

Following diagram captures the interface mapping to standards as defined by 5GEx in their functional model found here

Looking at the picture above, it can be observed that 5GEx mostly follows the ETSI MANO specific interfaces for interactions across MdOs or between MdO and DO. But the picture also includes some additional scope as listed below

• Topology distribution across domains for exchanging network topology details that may be used by MLPCE.
• Multi-domain path computation engine interaction across domains for exchanging path specific data
• SLA Management interaction across domains for exchanging business agreement.
• Service Catalogue interactions across domains for exchanging service specifications

Summarizing the scope in5GExproject, its key focus is in virtualized infrastructure with ETSI MANO building blocks with additional scope for exchanging the network topology, network path, business agreement and service specification across different domains. For ONAP Ext-API this may not be quiterelevantasitfunctionsatalayerabovetheNFVO. However, if External API scope is expanded to have cross-layer interaction, i.e MdO of one operator domain interacting with DO of another operator domain 5GEx specific interfaces may be relevant. But what can be learned from the 5GEx project is the concept of SLA Management, Catalogue exchange, Security mechanisms and approaches for supporting use cases such as Slicing.

The topic of inter-provider/inter-domain interaction is being discussed in TMF ODA, ETSI ZSM, but these specifications are still in the early stages of development and may not be relevant in the near future of ONAP development.

Another specification worth referring is ETSI IFA 028 v3.1.1 - wherein MANO architectural options to support multiple administrative domains is being discussed. This specification introduces two concepts - MLPOC (Multiple Logical Point of Contact) andSLPOC (Single Logical Point of Contact) with varying degrees of cross-layer interaction and information abstraction across domains. This specification also defines an Or-Or interface across NFVOs in different administrative domains. As described in the case of 5GEx, ETSI Or-Or level interaction may not be of immediate focus for External API project in the context of Interlude. However this may be relevant in case Ext-API is promoted as a sole external facing component from ONAP and interaction between two VFC or SO systems need to be enabled across operators favoring the ETSI SOL0011 interface. This requires further discussion with the architecture team and EUAG.

SDO/OSSP activities around inter-provider APIs are covered in detail in subsequent sections.

Multi-domain Interaction

As defined in 5GEx project multi-domain can be multiple network operators or it can be multiple subdomains within a single operator. The scope of interaction might be slightly different within single operator domains and across multiple operators because the latter will be governed by SLAs with strict policies and predefined trust/contract between the two operators. So security and trust are some of the key criteria for interaction across multiple parties. All interaction should be governed, policy controlled based on the trust agreement. Within the same operator domain, there can be multiple administrative domains which can be governed by SLO/OLOs and trust agreement as in the case of inter-provider interaction. But there can also be a model of distributed deployment which may not fit into the purview of multi-domain interaction. For example, geo-redundancy and HA deployments may not be classified as multi-domain interaction, but governed mostly by policies defined between two software components and interaction over internal APIs. Another consideration for multi-domain is the scope of domain - for example a domain might just focus on legacy network or a domain might focus on providing WAN services or another domain may implement only the network control capability without orchestration functions , or it can be a classification based on infrastructure, platform services. In all these cases the relevance of multi-domain might be applicable but need to be read along with the federation-delegation models and the administrative boundaries that separate through a business agreement. Form Ext API point of view the relevance of interlude level interaction is mostly between two administrative domains those are governed by predefined policies and offering services that can be queried and consumed.

Federation and Delegation

Similar to multi-domain interaction, federation and delegation are two terminologies used for interaction between two logically separated endpoints. While there is no standard terminology defined at the ONAP level, we can assume federation to be the east-west interaction between systems/components at same logical domains- for example between Orchestrators in two administrative domains, or controller in two administrative domains. The delegation terminology can be associated with the interactions between systems/components at different logical domains- for example, an end to end orchestrator interacting with a domain orchestrator. Federation and Delegation can be classified with reference to the diagram above from 5GEx. Interactions marked 2 can be classified in federation and interaction marked 3 and 5 can be classified as delegation. Here the federation is across domains of different operators, whereas delegation is between the same operator domains. Another differentiator is that federation is between logical domains with similar scope whereas delegation is between logical domains with a different scope. The logical separation can be based on the technology abstraction, geographic abstraction or deployment model. One example of the federation model is the interactions in the CCVPN use case an example of the delegation model is interaction possible between the central site and edge site orchestrators in an edge automation use case.

In the ETSIIFA028there are two models of inter-administrative domain interactions -SLPOC (Single Logical Point of Contact) and MLPOC (Multi Logical Point of Contact). In SLPOC there is a single interaction point between two administrative domains whereas in MLPOC there are multiple interaction points between administrative domains. In simple terms, it is possible based on ETSI MLPOC model for NFVO in one administrative domain to interact with VNFM or VIM in another administrative domain over the ETSI interfaces. Since External API functions at a layer above NFVO, the current scope of interaction is limited to the federation model described above -i.einteraction between External API in two operator domains.

The delegation model support in External API requires further discussion based on specific use cases. There is also discussion around Recursive Orchestration, Orchestration Hierarchy and Domain Orchestration. The delegation model can be considered within the scope of External API once some concrete decision is made by the Architecture team. For the current scope, only the federation model is considered.

Business Agreement and Policy

MEF Interlude does not have a specific scope for managing the Business Agreement between SP and Partner, however, the interaction between the parties might be governed and controlled based on the predefined business agreement and associated policy rules, security mechanism. 5GEx document on Business and Economic Layer (link) elaborates this aspect in detail but limits the focus on the SLA between parties. Some interesting aspects to be considered for Interlude are as follows

• Roles of the parties (SP and Partner) which will determine the mode of communication, specific controls required at either end, Policies to be enforced, the direction of communication (Some examples given for the 5G case are - Infrastructure Service Provider, Network Service Provider, Communication Service Provider, Over the top service provider, Exchange point service provider . The roles defined by 5GEx are mostly inspired by those defined by 3GPP 28.801.
• Centralized vs Distributed Interaction: The interaction between parties can be centralized i.e coordinated by an aggregator provider acting as an exchange point between parties or it can be one to one. The aggregator model is better as it will avoid the need for multiple business agreements while ensuring centralized enforcement SLA and policies
• Coordination model: Consists of two phases - publishing phase where information is exchanged between parties on the offered services and service composition phase when the actual customer request for a service is forwarded from one party to other. In the case of Ext-API this will be translated to a query on the other parties Service Catalog and initiation of service Configuration/Control Request
• Agreement Push vs Pull Model: In Pushmodelthe Business Agreement and policies are predetermined/agreed before any interaction between parties over the inter-providerAPIwhereasinpull model business agreement is dynamically decided based on the customer request and required SLA, monitoring levels.

The 5GEx project defines Business Agreement in terms of SLA and the document referred at the beginning of this section also gives a template for defining SLA. For ONAP Ext-API this may not be useful as it currently does not have any referenceable entities for defining policies for interaction between parties, which is quite relevant at the interlude level.

Other relevant SDO References for adapting Business Agreement are as follows

• TMF B2B2X Partnering Step by Step Guide (link): Lifecycle model for B2B partner management, templates for B2B agreement mostly focused on the business layer, but the operating agreement may be relevant for defining the interlude policies
• Recent ONS presentation on BlockChain based inter-operator agreement implemented based on HyperLedger (link)

For External-API project a new set of APIs needs to be defined for the Business Application layer to push the policies for interacting with the partner. In the absence of this API, it may be assumed that Ext-API will consult the Policy Engine in ONAP for determining the control mechanisms that need to be established before interacting with the partner over the inter-provider API.

Cross-layer Interaction

IN MEF LSO, the interlude reference point is between SOF in Service Provider domain and Partner Domain. If this principle is strictly followed the interaction can be confined to External API level or External API+SO combined. i.e SOF functionality can be assumed to be fulfilled using Ext-API and SO as a unified block. However, in practical deployments, there may be scenarios which might require cross-layer interaction, for example, MLPOC proposed in ETSI IFA028 wherein the Orchestration function in one domain interacts with VIM or VNFM in another domain. The multilayer interaction might also be possible in a hybrid orchestration scenario wherein the Virtualization and Non-Virtualized domains might have to interact at different levels - for example, a MEF LSO compatible system needs to interact with a non-MEF LSO compatible system. One more practical case is the domain orchestration scenario wherein different logical domains interact with each other. This is a wider consideration and decision requiring input from architecture subcommittee and EUAG.

Another aspect of inter-operator cross-layer interaction is cross-layer data reconciliation say at the inventory level or at the assurance level assuming the cross-layer interaction is permissible as per the business agreement and requires for efficiency. But this aspect is outside the scope of the interlude and may be a topic for wider discussions across SDOs.

For the scope of External API in the short term it is assumed that the cross-layer interaction is limited to the interaction between two systems in SP and Partner domains at least one of those is Ext-API component and other one is the entity which is logically equivalent in functionality and having permissible scope and APIs as defined in the MEF interlude specification and compatible with the Agreement between the two parties (SP and Partner)

Design Time Impact

In the CCVPN use case the Partner capabilities and management connectivity details are represented using a place-holder resource by name SPPartner. The SPPartner model is given in the link here. The CCVPN use case assumes that Services are designed independently by SP and Partner by respective Design time environment. This means that there will be redundant Service descriptors on the SP and Partner Design time catalog. The Service Model ID in SDC catalog is used to associate a Partner Service ID with a Service Specification in Inventory on either domain. Additionally, the Service Model ID maintained in the SDC catalog by the other party is manually referred for invoking operations - for example as a payload parameter on Service Order Management across SP and Partner. In practical scenarios, such redundancies and manual referencing cannot be avoided but if either domain use ONAP it shall be possible with additional capability on the design-time environment to design services using a single instance (managed by SP) and distribute the service models to both SP and Partner runtime environment.

From External API point of view, since it refers currently to the SDC Catalog for any Catalog management operations, there may be a need to identify the service specifications uniquely between SP and partner and refer to such specification for any interactions like Service Configuration and Control. There is also a need to reconcile the service specification based on incremental changes or versions updated in the design time catalog on either domain. Without such reconciliation and associated validations, the interactions may be inconsistent.

Another important consideration is maintaining the internal view of the service specification and external view. For example the Service Model Identifier maintained within External API can be considered as an external reference for the service specification whereas the Service model id within SDC catalog can be viewed as the internal ONAP identifier for the service model. The mapping between these two may be stored either in SDC or in External API. While referring to service specification over interlude reference point it may be more appropriate to use the identifier used by External API.

Mastership Management

In all practical cases, a typical SP may have to manage a group of Partners and corresponding inter-provider interactions. In such scenarios, it is important to assign the roles of the SP and Partner as to what each party is authorized on the SOF-SOF reference point. It is also possible that a Partner SOF is managed by multiple SPs or the SPs have a peer to peer relationship between them with each playing different roles at different contexts. As the interlude reference point focus on the Service Control and Configuration, it is important to ensure that simultaneous Service modification is avoided. One of the key requirement here would be to assign designated roles (mastership) for SP or Partner as to who is authorized to make Service related changes. Another requirement might be that before initiating any Service control the respective resources/applications are locked in inventory or through access control mechanisms. From Ext-API point of view this leads to a requirement to define the roles and authorizations while creating the Partner registration in inventory. This can be supplied either at the design time (through SDC - for example, while creating the SPPartner resource) or at runtime/design time through Policy association with a Service. Another option would be to leverage the ESR provided capabilities to manage the Partner registration where appropriate restrictions can be incorporated. The particular model of mastership assignment needs to be discussed further and selected during the implementation time.

This section mainly focuses on the identifiers shared between SP and Partner - For e.g.Service Specification being referred by the Service Provider and Partner at the Ext-API level, the mapping between Service specification used in Ext-API and ONAP SDC Catalog Service Model Identifier.

Service Assurance and Closed Loop Control

Currently, ONAP DCAE supports the deployment of a closed-loop application chain based on a preconfigured blueprint. This includes deployment of collectors, mappers and analytics applications. The closed-loop control is mainly driven by the event/data propagation from NF to collectors and further to analytics application. The decision for closed-loop control is preconfigured through operational policies defined in the policy engine. This model works fine in a single operator domain. However, this may not work very well across operator boundaries as thereisstrict access restrictions and security guidelines which will prevent SP to launch control loop chains on Partner domain.

MEF Interlude defines the scope for receiving Service specific event notifications from the partner. In ONAP Ext-API scope there can be two options for supporting this capability

1) Have custom control loop chains on either side of the operator domains to collect the service-specific event notifications and publish on internal message bus (DMaaP) which may be processed by Ext-API

2) Have a REST-based callback mechanism enabled between operator API gateways to support such service notifications - similar to the Hub resources used for Service Order status monitoring

The first option is scalable and probably more suitable in the long run, but this has the limitation that the event notification needs to follow the currently supported VES specification. The second method is more suitable in the short term with the assumption that either Operator systems are capable of generating/handling service level notification.

While Interlude specification does not define the type of Service notification possible the initial view is that it can be used for sharing the service instance status, service performance or service faults. So any resource level performance/faults need to be aggregated before notifying to the other party. In the case of ONAP, if both sides are realized using ONAP, there might be a need to have analytics applications to consolidate resource level performance/faults to service level and send the aggregate data across Interlude reference point.

Management Connectivity to Partner

TBD (MSB extension, REST API Call, ESR, DMaaP)

S3P Requirements

• Security: There are multiple aspects of security to be considered in the context of inter-provider interaction 1) Physical Security: Ensuring the partner and service provider interactions are not tampered through physical access 2) Information/Platform Security: Securing the data at SP and Partner side so that unauthorized and unintended data access can be avoided, Securing the SP and Partner Access credentials, Keys in a secure storage 3) Communication security : Securing the communication channel between SP and Partner 4) Regulatory controls : Lawful intercept support, Inter-provider exchange guidance, Country-specific controls etc.5) Policy-based controls: Security controls driven by business agreement between parties.
  For the Ext-API scope it is assumed that the management/control interaction between SP and Partner domains will be carried out through a secure HTTP based channel having SSL/TLS based encryption with a trusted Certificate issued by a third party CA. Additionally, at the inter-provider interface reference point, it is also important to ensure the interaction is controlled as per the inter-provider business agreement. From Ext-API point of view the expectation is that any interaction between the two endpoints will be governed by the policies defined in Policy Engine in ONAP. So there might be additional APIs or Policy Configurations might be required to govern the interaction of Ext-API. Generally, the security controls vary across operators based on the implemented IT/Network security standards (such as ISO27001, BS7759, ISO/IEC 27002 etc) and also the restrictions/regulatory controls in specific countries. So defining an Ext-API level consideration may be difficult. These aspects need to be driven by overall guidance by the Security Subcommittee and EUAG in ONAP.
• Scalability: Currently there is no specific scalability requirement with respect to inter-provider interactions, except for supporting the scaling of management modules like collectors, Hub resources etc corresponding to the Partner systems. However, this aspect may be revisited in the future based on mandatory S3P level requirements to be achieved for each release.
• Stability: No specific requirements as of now. To be revisited in future.
• Performance: No specific requirements as of now. To be revisited in future.

Service Impact Assessment

As per the MEF 55 interlude scope of Service Configuration and control, there may be an impact on service if it is already in an Active state. There shall be a need to carry out a service impact assessment before initiating the Service Configuration and Control. This is typically done in operational environments by placing a work order and associated workflows will carry out the impact and corrective rerouting and roll back measure before actually initiating the Service Configuration or Control. This is very specific to the use case and it also depends on whether the service configuration and control is service impacting or not. In the current MEF 55 scope, a dedicated change management is not scoped. However, intheONAPExt-API scope this may be one of the prerequisites that need to be initiated before or along with any operation over the inter-provider interface. Generally, in the traditional OSS environments, any request for partner services are initiated through a work order. The need for supporting inter-provider service request through a work order needs to be assessed based on the opinion of EUAG. For Ext-API, it is assumed that the inter-provider API request may be preceded by an optional change management request, or the Service Configuration/Control Request itself may be piggybacked in the change management request.

Standard APIs

MEF Interlude is a reference point which is expected to accommodate many APIs going forward based on the scope defined. Few examples of APIs that may be suitable as per the scope in MEF 55 are as follows

• Service Provider controls aspects of the Service within the Partner domain (on behalf of the Customer) by requesting changes to dynamic parameters as permitted by service policies. - TMF 641 Service Order Management API or TMF 640 Service Configuration and Activation Management API
• Service Provider queries the operational state of the Service - TMF 638 Service Inventory Management API (This may be restricted in some deployment scenarios)
• Service Provider requests change to the administrative state of a service or service component (e.g. Service Interface) - TMF 640 Service Configuration and Activation API
• Service Provider requests update to defaulted service parameters which are allowed to be customized (policy-controlled) - TMF 641 Service Order API or TMF 640 Service Configuration and Activation API
• Service Provider requests the creation of connectivity between two Service Interfaces as permitted by established business arrangement - TMF 641 Service Order API or TMF 640 Service Configuration and Activation API
• Service Provider provider queries the Partner's Service Inventory for services provided by the Partner to the Service Provider. - TMF 638 Service Inventory Management API
• Service Provider receives Service specific event notifications from the Partner - TMF 642 Alarm Management API or TMF 640/641 Service Order API (ServiceOrderChangeNotification)
• Service Provider receives Service specific performance information from the Partner - TMF 628 Performance Management API , TMF 649 Performance Management Threshold API
• Service Provider requests test initiation and receive test results from the Partner. - TMF 653 Service Test Management API
  

Update 11/14: A proposal from Orange available here suggest the possibility of two types of APIs - TMF 641 if the request originated from BSS and TMF640 if it is originated from an end user (provided BSS has authorized to carry out the operation without notifying BSS). Going by this proposal, TMF640 will be more suitable for interlude primarily for the specific items where both TMF640 and TMF641 are listed as options. But the following are the potential issues that need to be considered

• The mechanism of authorization by BSS - in terms of policy or by a design option in SDC
• Where to maintain such authorization data - SDC or Policy or Locally in Ext-API
• The prerequisites before issuing a TMF 640 request - verifying the state of the service, access policies etc
• The scope of authorizing the configuration - i.e how deep, what specific parameters, what conditions.
• Keeping track of the service configuration request
• The mechanism for monitoring such configuration if it is overridden by a power user from the partner side.

List of TMF Open APIs can be found here

TMF Notification Patterns - link

Information/Data Model

There are multiple models found to be relevant for inter-provider API

• MCM aligned E-Line Service Model defined in MEF Interlude Contribution - Access E-Line Service Control Classes - 5th Draft
• Work in progress MEF Services Common Model (link) - Initial Proposal for Work Item
• CFS/RFS being referenced by the TMF 641 (based on SID) (link) - Currently followed by CCVPN use case

The choice of a specific model will depend on the decision of EUAG, TOSCA Task Force in ONAP. From Ext-API point of view it is expected to leverage the CFS/RFS model being referred by the TMF 641 API or the Generic Resource Model used in TMF 655. In future as Interludespecific model in MEF and MEF Services Common Model matures appropriate mapping can be incorporated to accommodate specific service characteristics to the TMF APIs. In MEF LSO there is also NRM model being used for the Presto interface (derived from ONF). The NRM model is assumed to be out of scope for Ext-API unless there is a cross-layer interaction between SOF in SP domain and ICM in Partner domain is required.

There are other aspects of Modelling wherein the Partner Service and Resources need to be represented in an Abstract manner without sharing confidential information. Additionally, there are requirements to represent Partner registration data including connectivity details, Catalog, Inventory access details etc. One more aspect to be considered is the list of Management Services (Management APIs) that are supported by either party and how this information is discovered and maintained in the registration data.

There was related discussion as part of Casablanca CCVPN use case for representation of cross domain logical link ( ONAPARC-177 - Getting issue details... STATUS ) in inventory where in one of the input from Architecture subcommittee was to consider representing partner as a PNF. Additionally there was also a recommendation to consider primary and subordinate relationship between SP and Partner.

Summarizing above there are two types of models to be considered with high level information of potential data to be maintained at run time and design time.

• Design time model :
  • Partner as an abstract resource placeholder for management connectivity details
  • Role of the Partner - Primary or Subordinate (Mastership Relation)
  • Partner activation status
  • Services subscribed
  • Related business agreement/policy
• Run-time Model :
  • Partner abstract resource model with runtime connectivity parameters such as session details
  • Partner provided services as an association between Partner abstract resource and Service IDs
  • Partner consumed services as an association between Partner abstract resource and Service Specifications (SDC Model ID)
  • Partner connectivity state
  • Mastership status
  • Partner Subscriptions (Hub Resources)
  • Partner Service State
    • Partner Service Performance (future)
    • Partner Service Faults (future)
    • Partner Service Health