Background

Design Overview

Link to API

Multi-cluster Orchestrator micro-service components:

Project Module:  Projects allow multi-tenancy in the application from a user perspective.

Composite App Module:  K8splugin artifacts start in the form of a composite application representing an application. A composite application is a collection of Helm Charts. Each Helm chart represents an application inside the composite application. The orchestrator decides the placement of each application and their associated Kubernetes resources after considering inputs from the placement controllers like the HPA controller, action controllers like traffic management controller and other controllers in future.

Profile Module: Once a composite application is uploaded, then some profiles can be created to customize that Helm charts and instantiate it in Kubernetes. 

Deployment Intent Group Module: Intent Group API is used to create Intent Groups that can then will be used at instantiation time. Some intents that can be part of an Intent Group are Generic Placement Intent, HPA Intent, and traffic controller Intent.

Generic Placement Module: Generic Placement Intent API is used to create generic placement intents for applications within a composite application. Intent created using this API will be added to an Intent Group to use during instantiation.

Instantiation Module: Instantiates an application based on the Intent group. During instantiation, placement controllers and traffic controllers will be called to help with placement decisions and creating configurations for applications.

Cluster Registration Module:  Registering a cluster with cluster-specific information to be used by the Resource sync module. 

Day 2 Config Module:  Day 2 Configurations for applications are applied using K8S kinds (typically CRDs) implemented by application-specific operators.   

Resource sync micro-service components:

Resource Sync Module: Responsible for installed Kubernetes resources on the identified clusters

Status Update Module: Maintains the state of all the resources deployed for a composite application and its applications.

Internal Design details

Guidelines that need to keep in mind

• Support for metrics that can be retrieved by Prometheus

• Support for Jaeger distributed tracing by including operating libraries around HTTP calls.

• Support for logging that is understood by fluentd

• Mutual exclusion of database operations (keeping internal modules accessing database records simultaneously and also by replication entities of the scheduler micro-service).

• Resilience - ensure that the information returned by controllers is not lost as the synchronization of resources to remote edge clouds can take hours or even days when the edge is not up and running and possibility of restart of scheduler micro service in the meantime.

• Concurrency - Support multiple operations at a time and even synchronizing resources in various edge clouds in parallel.

• Performance - Avoiding file system operations as much as possible.

Composite Application Structure

Composite application resource represents a collection of applications and each application is represented by helm charts. Each Helm chart is wrapped in tar.gz format.

Instantiation Creation Flow

1. Get the clusters that form the logical cloud to schedule the application. Validate the application, intent group, and generic placement intent.

2. Create etcd Context for the application

3. Process the Generic placement context and populate the etcd context with the edge list.

4. Generate Kubernetes resources from the Helm charts by using Helm templating and from macros. Copy resources in etcd store based on the context for the application.

5. Placement stage: Call all the placement controllers including the HPA  controller in order. Wait for one controller to complete before going to the next controller.

   1. HPA controller: Kubernetes resource manipulation. Add additional information to the deployments. In the case where the generic intent calls for anyof clusters, the HPA controller can make a selection for one or more clusters meeting the HPA requirements for the application.

6. Action Controller Stage:  Call all the action controllers including Traffic Controller in order. Wait for one controller to complete before going to the next controller.

   1. Traffic Controller: Creates more resources for traffic routing and management between clusters.

7. After all the controllers are done processing if there are still multiple clusters that meet the requirements, the orchestrator will pick up the first cluster to schedule the resource.

8. Send notification to the resource sync to schedule the resources on the selected clusters. Once the resources are scheduled to be created the resource scheduler sends back notification to the orchestrator which in turn returns HTTP response as a success for the instantiation API.

9.  Resource Sync will periodically check the status of the resources and update the database with the status of the resource.

Instantiation Deletion Flow

1. Get etcd Context for the application

2. Send notification to the resource sync to delete the resources on the selected clusters. Once the resources are scheduled to be deleted the resource scheduler sends back notification to the orchestrator which in turn returns http response as a success for the instantiation delete API.

3.  Resource Sync will periodically check the status of the resources and update the database with the status of the resource.

Instantiation Update Flow 

This flow is for the case no changes are made to the Helm charts of the application. Changes to the intents are realized here. 

1. Get the clusters that form the logical cloud to schedule the composite application. Validate the application, intent group, and generic placement intent.

2. Create etcd Context for the application

3. Process the Generic placement context and populate the etcd context with the edge list.

4. Generate Kubernetes resources from the Helm charts by using Helm templating and from macros. Copy resources in etcd store based on the context for the application.

5. Placement stage: Call all the placement controllers including the HPA  controller in order. Wait for one controller to complete before going to the next controller.

   1. HPA controller: Kubernetes resource manipulation. Add additional information to the deployments. In the case where the generic intent calls for anyof clusters, the HPA controller can make a selection for one or more clusters meeting the HPA requirements for the application.

6. Action Controller Stage:  Call all the action controllers including Traffic Controller in order. Wait for one controller to complete before going to the next controller.

   1. Traffic Controller: Creates more resources for traffic routing and management between clusters.

7. After all the controllers are done processing if there are still multiple clusters that meet the requirements, the orchestrator will pick up the first cluster to schedule the resource.

8. Diff the original application resources and the edges that they are created on with this application context. Generate a list of resources to be changed.

9. Send notification to the resource sync to create/update/delete the resources on the selected clusters. Once the resources are scheduled to be updated the resource scheduler sends back notification to the orchestrator which in turn returns HTTP response as a success for the instantiation API.

10.  Resource Sync will periodically check the status of the resources and update the database with the status of the resource.

Orchestrator High-Level Design

Infrastructure Library

• MongoDB - Create, Read, Write

• Etcd - Create, Read, Write

Module Access Library

Project API- CreateProject, DeleteProject, GetProject

Composite App API - CreateCompositeApp, GetCompositeApp, DeleteCompositeApp, AddApptoCompositeApp, DeleteAppFromCompositeApp

Composite Profile API - CreateCompositeProfile, GetCompositeProfile, DeleteCompositeProfile, AddProfiletoCompositeProfile, DeleteProfileFromCompositeProfile

Generic Placement API - CreateGenericPlacement, DeleteGenericPlacement, GetGenericPlacement, AddGenericIntent, DeleteGenericIntent, GetIntent

Deployment Intent API - CreateDeploymentIntent, DeleteDeploymentIntent, GetDeploymentIntent, AddIntent, DeleteIntent, 

Status API - GetStatus(deploymet)

Runtime Resource Context Library

CreateContext(project, name) returns contextId

AddApp(contextId, name) return error

AddCluster(contextId, app-name, cluster-name)

AddResource(contextId, app-name, cluster-name, resourceName)

UpdateResourceField(contextId, app-name, cluster-name, resourceName, fieldName, Value)

Test cases

Composite App with EdgeX, Firewall with Intents