Background

L7 Proxy Service Mesh Controller intends to provide connectivity, shape the traffic, apply policies, RBAC and provide

mutual TLS for applications/microservices running across clusters (with service mesh), within the cluster

and with external applications. The functionalities are subjected to the usage of underlying service mesh technology.

Design Overview

Traffic Controller Design Internals

Internal Implementation Details

NOTE - Current implementation will support the ISTIO service mesh technology and SD-WAN load balancer and ExternalDNS as DNS provider. The plugin architecture of the controller makes it extensible to work with any Service mesh technology and any external load balancer as well. It is also designed to configure and communicate with external DNS servers.

JIRA

Component	JIRA Items
REST API Interface	MULTICLOUD-913 - Getting issue details... STATUS
2. Controller Interface, Backend Process	MULTICLOUD-914 - Getting issue details... STATUS
3. Developing backend code with interfaces	MULTICLOUD-915 - Getting issue details... STATUS
4. Loadbalancer configuration (Firewall, IPVS, NAT, and other L3 connectivity)	MULTICLOUD-924 - Getting issue details... STATUS MULTICLOUD-925 - Getting issue details... STATUS MULTICLOUD-926 - Getting issue details... STATUS
5. External DNS Configuration	MULTICLOUD-909 - Getting issue details... STATUS
6. Testing	MULTICLOUD-918 - Getting issue details... STATUS
7.Documentation	MULTICLOUD-923 - Getting issue details... STATUS

Elements of Traffic Controller with ISTIO as the service mesh

Gateways - The inbound/outbound access for the service mesh. It is an envoy service
VirtualServices - To expose the service outside the service mesh
DestinationRule - To apply rules for the traffic flow
AuthorizationPolicy - Authorization for service access
serviceEntry - add an external service into the mesh
Authentication Policy - Authenticate external communication

These are the Kubernetes resources generated per cluster. There will be multiple of these resources depending on the intent

API

RESTful North API (with examples)

Types	Intent APIs	Functionality
intercluster communication	/v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-intent-set/us-to-us-intents/	communication between microservices deployed between two clusters
2. external outbound service communication	/v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-intent-set/inbound-intent/	communication from external service to internal micro service
4. external inbound service communiation	/v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-intent-set/outbound-intent/	communication from internal service to access external service

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-intent-set
POST BODY:
{
	"name": "john",
	"description": "Traffic intent groups"
    "set":[
      {
         "us-to-us":"abc"
      },
      {
         "inbound":"abc"
      },
      {
         "outbound":"abc"
      }
   ]
}

1. Micro-service communication intents (Inter/Intra) - Edit the intent to have inbound services to a target service than the outbound services - check the API level access! - implement for all APIS!

POST

POST

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-intent-set/us-to-us-intents/

POST BODY:
{
	"metadata":{
	"name": "servicehttpbin" //unique name for each intent
    "description": "connectivity intent for micro-service to micro-service communication"
	"userdata1": <>,
	"userdata2": <>
	}

	"spec":{ // update the memory allocation for each field
	"application": "<app1>",
	"servicename": "httpbin01" //actual name of the client service
	"protocol": "HTTP",
	"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
	"mutualTLS": "MUTUAL", // Support 2 modes. SIMPLE, MUTUAL with external client. For inter and intra cluster, mtls is enabled by default
	"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
	"serviceMesh": "istio", // get it from cluster record

	// Traffic configuration
	"loadbalancingType": "ConsistenHash", // "Simple" and "consistentHash" are the two modes
	"loadBalancerMode": "httpCookie" // Modes for consistentHash - "httpHeaderName", "httpCookie", "useSourceIP", "minimumRingSize", Modes for simple - "LEAST_CONN", "ROUND_ROBIN", "RANDOM", "PASSTHROUGH" // choices of the mode must be explicit
	"httpHeader": john-user // Input for the hash when in "consistentHash" LB type and mode as "httpHeader"
	"httpCookie": user // Input for Hash in "ConsistenHash" LB and mode as "httpCookie" . Name of the cookie to maitain stick sessions.
	"maxConnections": 10 //connection pool for tcp and http traffic
	"timeOut" : 5 // in Seconds. Connection timeout for tcp and idleTimeout for http

	// credentials for mTLS
	"Servicecertificate" : {serverCertificate.pem} // Present actual certificate here. Optional, default "", required only if mTLS is set to "MUTUAL"
	"ServicePrivateKey" : {serverPrivateKey.pem} // Present actual private key here. Required only if mTLS is "MUTUAL"
	"caCertificate": {caCertificate.pem} // file should contain the public certificates for all root CAs that is trusted to authenticate your clients // not required for cluster level communication 
	
	}
}

RETURN STATUS: 201
RETURN BODY: 
{ 
  "name": "servicehttpbin"
  "Message": "Inbound service created"
}

GET

GET

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-intent-set/uservice-to-uservice-intent/servicehttpbin

RETURN STATUS: 201
RETURN BODY:
{
	"metadata":{
	"name": "servicehttpbin" //unique name for each intent
    "description": "connectivity intent for micro-service to microservice communication" 
	}

	spec:{	 
	"inboundservicename": "httpbin01" //actual name of the client service 
	"protocol": "HTTP",
	"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
	"mutualTLS": "MUTUAL", // Support 2 modes. SIMPLE, MUTUAL with external client. For inter and intra cluster, mtls is enabled by default
	"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
	"serviceMesh": "istio", // get it from cluster record

	// Traffic configuration
	"loadbalancingType": "ConsistenHash", // "Simple" and "consistentHash" are the two modes
	"loadBalancerMode": "httpCookie" // Modes for consistentHash - "httpHeaderName", "httpCookie", "useSourceIP", "minimumRingSize", Modes for simple - "LEAST_CONN", "ROUND_ROBIN", "RANDOM", "PASSTHROUGH"
	"httpHeader": x-user // Input for the hash when in "consistentHash" LB type and mode as "httpHeader"
	"httpCookie": user // Input for Hash in "ConsistenHash" LB and mode as "httpCookie" . Name of the cookie to maitain stick sessions.
	"maxConnections": 10 //connection pool for tcp and http traffic
	"timeOut" : 5 // in Seconds. Connection timeout for tcp and idleTimeout for http
	
	// credentials for mTLS
	"Servicecertificate" : {serverCertificate.pem} // Present actual certificate here. Optional, default "", required only if mTLS is set to "MUTUAL"
	"ServicePrivateKey" : {serverPrivateKey.pem} // Present actual private key here. Required only if mTLS is "MUTUAL"
	"caCertificate": {caCertificate.pem} // file should contain the public certificates for all root CAs that is trusted to authenticate your clients // not required for cluster level communication
	}
}

DELETE

DELETE

DELETE
URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-group-intent/uservice-to-uservice-intent/servicehttpbin

RETURN STATUS: 204

POST - with the client details

POST

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-group-intent/uservice-to-uservice-intent/clients
POST BODY:

{
		"clientServiceName": "sleep01", // Name of the client service.
		"headless": "true", // default is false. Option "True" will generate the required configs for all the instances of headless service
		"egressgateway": "true" ,  // Optional, default = false, All the outbound traffic from this service will flow through a dedicated egress gateway
}

RETURN STATUS: 201
RETURN BODY:
{ 
  "name": "sleep01"
  "Message": "Client created"
}

GET - The Client resource

GET

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-group-intent/uservice-to-uservice-intent/clients/sleep01
RETURN STATUS: 201
RETURN BODY:

	"clientService": {
		"clientServiceName": "sleep01", // if any then allow all the external applications to connect, check for serviceaccount level access
		"headless": "true", // default is false. Option "True" will generate the required configs for all the instances of headless service
		"egressgateway": "true" ,  // Optional, default = false, All the outbound traffic from this service will flow through a dedicated egress gateway
		
	 }

DELETE

GET

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-group-intent/uservice-to-uservice-intent/clients/sleep01

RETURN STATUS: 204

Security Resource

POST

GET

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-group-intent/uservice-to-uservice-intent/clients/sleep01/security/security-intent

{
	
	"userAccess": [{userName: "Todd", accessList:Action:["/health_check": GET, "/status/: GET, "/upload": POST]}, {userName: "Thor", accessList:["/health_check": GET, "/status/: GET, "/upload": POST]} ] // These are the user in k8s
}


RETURN STATUS: 204

Traffic Resource??

GET

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-group-intent/uservice-to-uservice-intent/clients/sleep01/traffic/traffic-intent

{
	
	
}

RETURN STATUS: 204

The above intent will generate the following configuration provided the service mesh is istio.

Name of the Cluster	Microservices	Istio objects
Cluster01	httpbin01	serviceentry - "sleep01" destinationrules- loadbalancing, mTLS, virtualservice - "externalName" authentication policy - authentication for intracluster services Authentication Policy - External user authentication
2. Cluster02	httpbin02	egressgateway - if "true", create service destinationrules - direct traffic from service to egressgateway headless - create virtualservices per instance of headless service

NOTE - The default authorization policy must have with "deny-all" under spec as we need to disable all the communication between microservices during istio installation

2. External service to access Inbound service - Inbound access

NOTE - These are the services whose nature is not known. These services are assumed to have FQDN as a point of connectivity

POST

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-group-intent/inbound-intent/
POST BODY:
{
	"name": <name> //unique name for each intent
    "description": <description> 
	"inboundservicename": "mysql" //actual name of the client service
	"description": "bookinfo app", 
	"protocol": "HTTP",
	"externalName": "", // Optional, default = "", Not required for Outbound access since the communication will be initialted from inboundservice
	"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
	"mutualTLS": "true", // Setting this to true will create a dedicated egrees gateway for the service "httpbin01" on whichever cluster it is running on
	"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on 
	"serviceMesh": "istio", // get it from cluster record
	"loadbalancing": "true", // optional
}
RETURN STATUS: 201
RETURN BODY: 
{ 
  "Message": "outbound coonectivity intent creation success "
  "description": "Connectivity intent for inbound service to connect to external services"
}

POST - External service to access inbound service

POST

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-group-intent/inbound-intent/<intent-name>/clients
POST BODY:
{
	"name": <name> //unique name for each intent
   	"description": <description>
	"externalServiceName": {cnn.edition.com} // Only the FQDN of the service name is required
	"externalCaCertificate" : {clientCaCert.pem} // Present the actual client certificate
}

RETURN STATUS: 201
RETURN BODY: 
{ 
  "Message": "Success "
  "description": "External service given access to inbound service"
}

Security

GET

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-group-intent/inbound-intent/<intent-name>/clients/client01/security

{
	"name": <name> //unique name for each intent
    "description": <description>
	"externalAuthenticationissuer": "https://accounts.google.com",
    "externalAuthenticationjwksURI" : "https://www.googleapis.com/oauth2/v3/certs",
	"userAccess": [{userName: "Todd", accessList:Action:["/health_check": GET, "/status/: GET, "/upload": POST]}, {userName: "Thor", accessList:["/health_check": GET, "/status/: GET, "/upload": POST]} ] // These are the external users
}


RETURN STATUS: 204

3. Outbound access

POST -

POST

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-group-intent/outbound-intent/<intent-name>/clients/


POST BODY:
{
	"name": "<name>" //unique name for each intent
    "description": <description> 
	"inboundservicename": "httpbin01" //actual name of the client service 
	"protocol": "HTTP",
	"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
	"mutualTLS": "MUTUAL", // Support 2 modes. SIMPLE, MUTUAL with external client. For inter and intra cluster, mtls is enabled by default
	"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
	"serviceMesh": "istio", // get it from cluster record
	// Traffic configuration
	"loadbalancingType": "ConsistenHash", // "Simple" and "consistentHash" are the two modes
	"loadBalancerMode": "httpCookie" // Modes for consistentHash - "httpHeaderName", "httpCookie", "useSourceIP", "minimumRingSize", Modes for simple - "LEAST_CONN", "ROUND_ROBIN", "RANDOM", "PASSTHROUGH"
	"httpHeader": x-user // Input for the hash when in "consistentHash" LB type and mode as "httpHeader"
	"httpCookie": user // Input for Hash in "ConsistenHash" LB and mode as "httpCookie" . Name of the cookie to maitain stick sessions.
	"maxConnections": 10 //connection pool for tcp and http traffic
	"timeOut" : 5 // in Seconds. Connection timeout for tcp and idleTimeout for http
	// credentials for mTLS
	"Servicecertificate" : {serverCertificate.pem} // Present actual certificate here. Optional, default "", required only if mTLS is set to "MUTUAL"
	"ServicePrivateKey" : {serverPrivateKey.pem} // Present actual private key here. Required only if mTLS is "MUTUAL"
	"caCertificate": {caCertificate.pem} // file should contain the public certificates for all root CAs that is trusted to authenticate your clients // not required for cluster level communication 
}

RETURN STATUS: 201
RETURN BODY: 
{ 
  "name": "servicehttpbin"
  "Message": "Inbound service created"
}

POST - Provide access to an external service from inbound service

POST

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/traffic-group-intent/inbound-intent/
POST BODY:
	{
		"externalServiceName": {cnn.edition.com} // Only the FQDN of the service name is required
	}

RETURN STATUS: 201
RETURN BODY: 
{ 
  "Message": "Success "
  "description": "External service given access to inbound service"
}

Keywords	Supported fields	Description
{connectivity-type}	intercluster/intracluster	types in API for {connectivity-type}
{connectivity-sub-type}	intermicroservice/internalapplication/externalmicroservice	sub-types in API for {connectivity-sub-type}
name	name of the microservice/application depending on the context

Development

go API library - https://github.com/gorilla/mux
backend - mongo - https://github.com/onap/multicloud-k8s/tree/master/src/k8splugin/internal/db - Reference
intent to config conversion - use go templates and admiral? https://github.com/istio-ecosystem/admiral
writing the config to etcd - WIP
Unit tests and Integration test - go tests

External DNS - Design and intent API

See here: External DNS provider update design and intent API

External application communication intents

Considering DNS resolution, No DNS resolution (IP addresses), Egress proxies of the Service Mesh, Third-party egress proxy

User facing communication intents

Considering Multiple DNS Servers

Considering multiple user-facing entities

Considering RBAC/ABAC

Internal Design details

Guidelines that need to keep in mind

Support for metrics that can be retrieved by Prometheus
Support for Jaeger distributed tracing by including open tracing libraries around HTTP calls.
Support for logging that is understood by fluentd
Mutual exclusion of database operations (keeping internal modules accessing database records simultaneously and also by replication entities of the scheduler micro-service).
Resilience - ensure that the information returned by controllers is not lost as the synchronization of resources to remote edge clouds can take hours or even days when the edge is not up and running and possibility of restart of scheduler micro service in the meantime.
Concurrency - Support multiple operations at a time and even synchronizing resources in various edge clouds in parallel.
Performance - Avoiding file system operations as much as possible.

Modules (Description, internal structures etc..)

....

Background

Design Overview

Traffic Controller Design Internals

Internal Implementation Details

JIRA

API

RESTful North API (with examples)

1. Micro-service communication intents (Inter/Intra) - Edit the intent to have inbound services to a target service than the outbound services - check the API level access! - implement for all APIS!

Security Resource

Traffic Resource??

The above intent will generate the following configuration provided the service mesh is istio.

NOTE - The default authorization policy must have with "deny-all" under spec as we need to disable all the communication between microservices during istio installation

2. External service to access Inbound service - Inbound access

Security

3. Outbound access

Development

External DNS - Design and intent API

External application communication intents

User facing communication intents

Internal Design details

Guidelines that need to keep in mind

Modules (Description, internal structures etc..)

Sequence flows

Test cases