Background
L7 Proxy Service Mesh Controller intends to provide connectivity, shape the traffic, apply policies, RBAC and provide
mutual TLS for applications/microservices running across clusters (with service mesh), within the cluster
and with external applications. The functionalities are subjected to the usage of underlying service mesh technology.
Design Overview
Traffic Controller Design Internals
NOTE - Current implementation will support the ISTIO service mesh technology and SD-WAN load balancer. The plugin architecture of the controller makes it extensible to work with any Service mesh technology and any external load balancer as well. It is also designed to configure and communicate with external DNS servers.
JIRA
| Component | JIRA Items |
|---|---|
| MULTICLOUD-913 - Getting issue details... STATUS |
| 2. Controller Interface, Backend Process | MULTICLOUD-914 - Getting issue details... STATUS |
| 3. Developing backend code with interfaces | MULTICLOUD-915 - Getting issue details... STATUS |
| 4. Loadbalancer configuration (Firewall, IPVS, NAT, and other L3 connectivity) | MULTICLOUD-924 - Getting issue details... STATUS |
| 5. External DNS Configuration | MULTICLOUD-909 - Getting issue details... STATUS |
| 6. Testing | MULTICLOUD-918 - Getting issue details... STATUS |
| 7.Documentation | MULTICLOUD-923 - Getting issue details... STATUS |
Elements of Traffic Controller with ISTIO as the service mesh
- Gateways - The inbound/outbound access for the service mesh. It is an envoy service
- VirtualServices - To expose the service outside the service mesh
- DestinationRule - To apply rules for the traffic flow
- AuthorizationPolicy - Authorization for service access
- serviceEntry - add an external service into the mesh
- Authentication Policy - Authenticate external communication
These are the Kubernetes resources generated per cluster. There will be multiple of these resources depending on the intent
API
RESTful North API (with examples)
| Types | Intent APIs | Functionality |
|---|---|---|
| /v2/project/{project-name}/rb/{rb-name}/{version}/intent/{intent-name}/connectivity/intercluster/ | communication between microservices deployed between two clusters |
| 2. external outbound service communication | /v2/project/{project-name}/rb/{rb-name}/{version}/intent/{intent-name}/connectivity/external/outbound/ | communication from microservice to external service |
| 3. intracluster communication | /v2/project/{project-name}/rb/{rb-name}/{version}/intent/{intent-name}/connectivity/intracluster/ | communication between microservices in the same cluster |
| 4. external inbound service communiation | /v2/project/{project-name}/rb/{rb-name}/{version}/intent/{intent-name}/connectivity/external/inbound/ | API for external service to access the microservices inside the mesh |
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/
POST BODY:
{
"name": "john",
"description": "Traffic intent groups"
"set":[
{
"interclusterservice":"abc"
},
{
"externaloutboundaccess":"abc"
},
{
"intraclusterservice":"abc"
},
{
"externalinboundaccess":"abc"
},
{
"dnsproviders":"abc"
}
]
}
1. Inter Micro-service communication intents - Edit the intent to have inbound services to a target service than the outbound services - check the API level access! - implement for all APIS!
POST
POST
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/{trafficset-name}/inboundservice
// URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-groups/{trafficgroup-name}/interclusterservice/{intercluster-record //only ms name}/clientservice// list of clients
POST BODY:
{
"name": "servicehttpbin" //unique name for each intent
"description": "connectivity intent for microservice replication across multiple locations and clusters"
"inboundservicename": "httpbin01" //actual name of the client service
"description": "bookinfo app",
"protocol": "HTTP",
"externalName": "", // Optional, default = "", This is the prefix used to expose this service outside the cluster, not mandatory for "intercluster" API, But mandatory foe external inbound access
"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
"mutualTLS": "MUTUAL", // Support 2 modes. SIMPLE, MUTUAL with external client, for inter and intra cluster, mtls is enabled by default
"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
"serviceMesh": "istio", // get it from cluster record
"loadbalancing": "true", // optional
"externalAuthenticationissuer": "",
"externalAuthenticationjwksURI" : "",
"clientcertificateDetails" : [.pem, .key, .pem]
"accessPoints": ["/health", "/status"]
}
RETURN STATUS: 201
RETURN BODY:
{
"name": "servicehttpbin"
"Message": "Inbound service created"
}
GET
GET
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/{trafficset-name}/inboundservice/servicehttpbin
// URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-groups/{trafficgroup-name}/interclusterservice/{intercluster-record //only ms name}/clientservice// list of clients
RETURN STATUS: 201
RETURN BODY:
{
"name": "servicehttpbin" //unique name for each intent
"description": "connectivity intent for microservice replication across multiple locations and clusters"
"inboundservicename": "httpbin01" //actual name of the client service
"description": "bookinfo app",
"protocol": "HTTP",
"externalName": "", // Optional, default = "", This is the prefix used to expose this service outside the cluster, not mandatory for "intercluster" API, But mandatory foe external inbound access
"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
"mutualTLS": "MUTUAL", // Support 2 modes. SIMPLE, MUTUAL with external client, for inter and intra cluster, mtls is enabled by default
"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
"serviceMesh": "istio", // get it from cluster record
"loadbalancing": "true", // optional
"externalAuthenticationissuer": "",
"externalAuthenticationjwksURI" : "",
"clientcertificateDetails" : [.pem, .key, .pem]
"accessPoints": ["/health", "/status"]
}
DELETE
DELETE
DELETE
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/{trafficset-name}/inboundservice/servicehttpbin
RETURN STATUS: 204
POST - with the client details
POST
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/{trafficset-name}/inboundservice/servicehttpbin/clients
POST BODY:
{
"name": "servicehttpbin" //unique name for each intent
"description": "connectivity intent for microservice replication across multiple locations and clusters"
"inboundservicename": "httpbin01" //actual name of the client service
"description": "bookinfo app",
"protocol": "HTTP",
"externalName": "", // Optional, default = "", This is the prefix used to expose this service outside the cluster, not mandatory for "intercluster" API, But mandatory foe external inbound access
"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
"mutualTLS": "MUTUAL", // Support 2 modes. SIMPLE, MUTUAL with external client, for inter and intra cluster, mtls is enabled by default
"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
"serviceMesh": "istio", // get it from cluster record
"loadbalancing": "true", // optional
"externalAuthenticationissuer": "",
"externalAuthenticationjwksURI" : "",
"clientcertificateDetails" : [.pem, .key, .pem]
"accessPoints": ["/health", "/status"]
"clientService": {
"clientServiceName": "sleep01", // if any then allow all the external applications to connect, check for serviceaccount level access
"headless": "true", // default is false. Option "True" will generate the required configs for all the instances of headless service
"egressgateway": "true" , // Optional, default = false, All the outbound traffic from this service will flow through a dedicated egress gateway
}
}
RETURN STATUS: 201
RETURN BODY:
{
"name": "sleep01"
"Message": "Client created"
}
GET
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/{trafficset-name}/inboundservice/servicehttpbin/clients/clients01intent
RETURN STATUS: 201
RETURN BODY:
{
"name": "servicehttpbin" //unique name for each intent
"description": "connectivity intent for microservice replication across multiple locations and clusters"
"inboundservicename": "httpbin01" //actual name of the client service
"description": "bookinfo app",
"protocol": "HTTP",
"externalName": "", // Optional, default = "", This is the prefix used to expose this service outside the cluster, not mandatory for "intercluster" API, But mandatory foe external inbound access
"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
"mutualTLS": "MUTUAL", // Support 2 modes. SIMPLE, MUTUAL with external client, for inter and intra cluster, mtls is enabled by default
"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
"serviceMesh": "istio", // get it from cluster record
"loadbalancing": "true", // optional
"externalAuthenticationissuer": "",
"externalAuthenticationjwksURI" : "",
"clientcertificateDetails" : [.pem, .key, .pem]
"accessPoints": ["/health", "/status"]
"clientService": {
"clientServiceName": "sleep01", // if any then allow all the external applications to connect, check for serviceaccount level access
"headless": "true", // default is false. Option "True" will generate the required configs for all the instances of headless service
"egressgateway": "true" , // Optional, default = false, All the outbound traffic from this service will flow through a dedicated egress gateway
}
}
The above intent will generate the following configuration provided the service mesh is istio.
| Name of the Cluster | Microservices | Istio objects | Description/comments |
|---|---|---|---|
| httpbin01 |
| |
| 2. Cluster02 | httpbin02 |
| |
2. Intent API for Intracluster communication
NOTE - Call this API only if the services are running in the same cluster, The default authorization policy must have with "deny-all" under spec as we need to disable all the communication between microservices during istio installation implement this API
POST
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/{trafficset-name}/intraclusterservice/client01intent
POST BODY:
{
"name": "johndoe" //unique name for each intent
"description": "connectivity intent for microservice replication across multiple locations and clusters"
"inboundservicename": "httpbin01" //actual name of the client service
"description": "bookinfo app",
"protocol": "HTTP",
"externalName": "", // Optional, default = "", Not required for "intraclusterservice"
"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
"mutualTLS": "true", // Setting this to true will create a dedicated egrees gateway for the service "httpbin01" on whichever cluster it is running on
"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
"serviceMesh": "istio", // get it from cluster record
"loadbalancing": "true", // optional
"externalAuthenticationissuer": "https://accounts.google.com",
"externalAuthenticationjwksURI" : "https://www.googleapis.com/oauth2/v3/certs",
"clientcertificateDetails" : [.pem, .key, .pem]
"endpoints": [{"/v2/api/httpin": "kim"}, {"/v2/api/auth": "roger"}] # End points of microservice which is exposed to specific user group
"clientService": {
"clientServiceName": "sleep01", // if any then allow all the external applications to connect, check for serviceaccount level access
"headless": "true", // default is false. Option "True" will generate the required configs for all the instances of headless service
"egressgateway": "true" , // Optional, default = false, All the outbound traffic from this service will flow through a dedicated egress gateway
}
}
RETURN STATUS: 201
RETURN BODY:
{
"Message": "Intercluster Connectivity intent success "
"description": "Connectivity intent for Intra cluster services"
}
3. microservice connectivity to an external service intent API - Outbound access
NOTE - These are the services whose nature is not known. These services are assumed to have FQDN as a point of connectivity
POST
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/{trafficset-name}/outboundservice/client01intent
POST BODY:
{
"name": "johndoe" //unique name for each intent
"description": "connectivity intent for microservice replication across multiple locations and clusters"
"inboundservicename": "httpbin01" //actual name of the client service
"description": "bookinfo app",
"protocol": "HTTP",
"externalName": "", // Optional, default = "", Not required for Outbound access since the communication will be initialted from inboundservice
"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
"mutualTLS": "true", // Setting this to true will create a dedicated egrees gateway for the service "httpbin01" on whichever cluster it is running on
"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
"serviceMesh": "istio", // get it from cluster record
"loadbalancing": "true", // optional
"externalAuthenticationissuer": "https://accounts.google.com",
"externalAuthenticationjwksURI" : "https://www.googleapis.com/oauth2/v3/certs",
"clientcertificateDetails" : [.pem, .key, .pem]
"user": [""] # Not required for Outbound access
"clientService": {
"clientServiceName": {"sleep01.service.com} // Only the FQDN of the service name is required.
}
}
RETURN STATUS: 201
RETURN BODY:
{
"Message": "outbound coonectivity intent creation success "
"description": "Connectivity intent for inbound service to connect to external services"
}
5. API for external services to access a microservice - Inbound access
POST
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/{trafficset-name}/outboundservice/client01intent
POST BODY:
{
"name": "johndoe" //unique name for each intent
"description": "connectivity intent for microservice replication across multiple locations and clusters"
"inboundservicename": "httpbin01" //actual name of the client service
"description": "bookinfo app",
"protocol": "HTTP",
"externalName": "", // Optional, default = "", must be defined for Inbound access
"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
"mutualTLS": "true", // Setting this to true will create a dedicated egrees gateway for the service "httpbin01" on whichever cluster it is running on
"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
"serviceMesh": "istio", // get it from cluster record
"loadbalancing": "true", // optional
"user": [""] # Optional. Restricts the users from accessing these services
"clientService": {
"clientServiceName": {"sleep01.service.com} // Only the FQDN of the service name is required
}
}
RETURN STATUS: 201
RETURN BODY:
{
"Message": "Inbound coonectivity intent creation success"
"description": "Connectivity intent for external services to connect to inboundservice"
}
| Keywords | Supported fields | Description |
|---|---|---|
| {connectivity-type} | intercluster/intracluster | types in API for {connectivity-type} |
| {connectivity-sub-type} | intermicroservice/internalapplication/externalmicroservice | sub-types in API for {connectivity-sub-type} |
| name | name of the microservice/application depending on the context | |
Development
- go API library - https://github.com/gorilla/mux
- backend - mongo - https://github.com/onap/multicloud-k8s/tree/master/src/k8splugin/internal/db - Reference
- intent to config conversion - use go templates and admiral?
- writing the config to etcd - WIP
- Unit tests and Integration test - go tests
External DNS - Design and intent API
See here: External DNS provider update design and intent API
External application communication intents
Considering DNS resolution, No DNS resolution (IP addresses), Egress proxies of the Service Mesh, Third-party egress proxy
User facing communication intents
Considering Multiple DNS Servers
Considering multiple user-facing entities
Considering RBAC/ABAC
Internal Design details
Guidelines that need to kept in mind
- Support for metrics that can be retrieved by Prometheus
- Support for Jaeger distributed tracing by including opentracing libraries around HTTP calls.
- Support for logging that is understood by fluentd
- Mutual exclusion of database operations (keeping internal modules accessing database records simultaneously and also by replication entities of the scheduler micro-service).
- Resilience - ensure that the information returned by controllers is not lost as the synchronization of resources to remote edge clouds can take hours or even days when the edge is not up and running and possibility of restart of scheduler micro service in the meantime.
- Concurrency - Support multiple operations at a time and even synchronizing resources in various edge clouds in parallel.
- Performance - Avoiding file system operations as much as possible.
Modules (Description, internal structures etc..)
....