Scenario 04 - Multiple Stateless (HTTP) application communication with "MUTUAL" TLS On Multiple logical clouds along with external communication

Description: - Connect multiple microservices on multiple logical clouds deployed across multiple clusters

Diagram



NOTE - For this scenario, the default mesh wide policy must be set to "PERMISSIVE" on both the clusters. It will not work if the default Mesh Policy is "STRICT"

Important Info - cert-chain.pem is Envoy’s cert that needs to be presented to the other side. key.pem is Envoy’s private key paired with Envoy’s cert in cert-chain.pem. root-cert.pem is the root cert to verify the peer’s cert. In this example, we only have one Citadel in a cluster, so all Envoys have the same root-cert.pem.

Add Inbound service 01

POST - traffic intent for the inbound service (service hosted behind the cluster)

POST
URL: /v2/projects/{project-name}/composite-apps/blue-app/{version}/traffic-intent-set/inbound-intents/

POST BODY:
{
	"metadata": {
	"name": "<>" // unique name for each intent
    "description": "connectivity intent for inbound communication"
	"userdata1": <>,
	"userdata2": <>
	}

	"spec": { // update the memory allocation for each field as per OpenAPI standards
	"application": "<app1>",
	"servicename": "httpbin" //actual name of the client service - {istioobject - serviceEntry of client's cluster}
	"externalName": "httpbin.k8s.com"
	"protocol": "HTTP",
	"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
	"mutualTLS": "MUTUAL", // default is simple. Option MUTUAL will enforce mtls {istioobject - destinationRule}
	"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
	"serviceMesh": "istio", // get it from cluster record
	"sidecar-proxy": "yes", // The features (mTLS, LB, Circuit breaking) are not available to services without istio-proxy. Only inbound routing is possible.
	// Traffic management fields below are valid only if the sidecar-proxy is set to "yes"
	"traffic-management-info" : {
	// Traffic configuration - Loadbalancing is applicable per service. The traffic to this service is distrbuted amongst the pods under it.
	"loadbalancingType": "ConsistenHash", // "Simple" and "consistentHash" are the two modes - {istioobject - destinationRule}
	"loadBalancerMode": "httpCookie" // Modes for consistentHash - "httpHeaderName", "httpCookie", "useSourceIP", "minimumRingSize", Modes for simple - "LEAST_CONN", "ROUND_ROBIN", 	"RANDOM", "PASSTHROUGH" // choices of the mode must be explicit - {istioobject - destinationRule}
	"httpCookie": "user1" // Name of the cookie to maitain sticky sessions - {istioobject - destinationRule}

	// Circuit Breaking
	"maxConnections": 10 //connection pool for tcp and http traffic - {istioobject - destinationRule}
	"concurrenthttp2Requests": 1000 // concurent http2 requests which can be allowed - {istioobject - destinationRule}
	"httpRequestPerConnection": 100 // number of http requests per connection. Valid only for http traffic - {istioobject - destinationRule}
	"consecutiveErrors": 8 // Default is 5.  Number of consecutive error before the host is removed - {istioobject - destinationRule}
	"baseEjectionTime" : 15 // Default is 5, - {istioobject - destinationRule}
	"intervalSweep": 5m, //time limit before the removed hosts are added back to the load balancing pool. - {istioobject - destinationRule}
	}

	// credentials for mTLS.
	"Servicecertificate" : "" // Present actual certificate here.
	"ServicePrivateKey" : "" // Present actual private key here.
	"caCertificate" : "" // present the trusted certificate to verify the client connection, Required only when mtls mode is MUTUAL	
	}
}

RETURN STATUS: 201
RETURN BODY: 
{ 
  "name": "<name>"
  "Message": "inbound service created"
}

Authorization for Inbound Service 01

POST
URL: /v2/projects/{project-name}/composite-apps/blue-app/{version}/traffic-intent-set/inbound-intents/{serviceName}/authorization-policies

POST BODY:
{
	"metadata": {
	"name": "<>" // unique name for each intent
    "description": "Authorization Policy for inbound services"
	"userdata1": <>,
	"userdata2": <>
	}

	"spec": {
	// Access Control
	"namespaces": [] // Workloads from this namespaces can access the inbound service - {istioobject - authorizationPolicy}
	"serviceAccountAccess" : {[ "cluster.local/ns/<Namespace>/sa/sleep": ["GET": "/status"]} // {istioobject - authorizationPolicy, will be applied for the inbound service}
	
	}
}

RETURN STATUS: 201
RETURN BODY: 
{ 
  "name": "<name>"
  "Message": "Authorizations Policy created"
}

Client 01 

POST - traffic intent to add clients for accessing a specific inbound service - NOTE - Clients will have the mTLS mode same as the inbound service

POST
URL: /v2/projects/{project-name}/composite-apps/brown-app/{version}/traffic-group-intent/outbound-intents/
POST BODY:
{
	"metadata": {
	"name": "<name>" // unique name for each intent
    "description": "connectivity intent for outbound communication"
	"application": "<app1>",
	"userdata1": <>,
	"userdata2": <>
	}

	spec: {
		"ServiceName": "sleep", // Name of the client service
		"type": "istio", // options are istio, k8s and external
		"headless": "false", // default is false. Option "True" will generate the required configs for all the instances of headless service
		"targetServiceName": "httpbin.k8s.com" // FQDN expected since the client belongs to a different composite app
	}
}

RETURN STATUS: 201
RETURN BODY:
{ 
  "name": "<name>"
  "Message": "Client created"
}

Client 02 

POST - traffic intent to add clients for accessing a specific inbound service - NOTE - Clients will have the mTLS mode same as the inbound service

POST
URL: /v2/projects/{project-name}/composite-apps/blue-app/{version}/traffic-group-intent/outbound-intents/
POST BODY:
{
	"metadata": {
	"name": "<name>" // unique name for each intent
    "description": "connectivity intent add client communication"
	"application": "<app1>",
	"userdata1": <>,
	"userdata2": <>
	}

	spec: {
		"clientServiceName": "sleep", // Name of the client service
		"type": "istio", // options are istio, k8s and external
		"headless": "false", // default is false. Option "True" will generate the required configs for all the instances of headless service
		"targetService": "httpbin.k8s.com" // Both client and service belong to the same composite. This notation is still used for consistency
	}
}

RETURN STATUS: 201
RETURN BODY:
{ 
  "name": "<name>"
  "Message": "Client created"
}

Client 03

POST - traffic intent to add clients for accessing a specific inbound service - NOTE - Clients will have the mTLS mode same as the inbound service

POST
URL: /v2/projects/{project-name}/composite-apps/brown-app/{version}/traffic-group-intent/outbound-intents/
POST BODY:
{
	"metadata": {
	"name": "<name>" // unique name for each intent
    "description": "connectivity intent add client communication"
	"application": "<app1>",
	"userdata1": <>,
	"userdata2": <>
	}

	spec: {
		"clientServiceName": "onap.k8s.org", // Name of the client service
		"type": "external", // options are istio, k8s and external
		"headless": "false", // default is false. Option "True" will generate the required configs for all the instances of headless service
		"targetService": "httpbin.k8s.com"
	}
}

RETURN STATUS: 201
RETURN BODY:
{ 
  "name": "<name>"
  "Message": "Client created"
}

Add Inbound service 02

POST

POST
URL: /v2/projects/{project-name}/composite-apps/blue-app/{version}/traffic-intent-set/inbound-intents/

POST BODY:
{
	"metadata": {
	"name": "<httpbin>" // unique name for each intent
    "description": "connectivity intent for stateless micro-service to stateless micro-service communication"
	"userdata1": <>,
	"userdata2": <>
	}

	"spec": { 
	"application": "<app1>",
	"servicename": "productpage" //actual name of the client service
	"externalName": "productpage.k8s.com"
	"protocol": "HTTP",
	"headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service
	"mutualTLS": "MUTUAL", // default is simple. Option MUTUAL will enforce mtls
	"port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on
	"serviceMesh": "istio", // get it from cluster record
	"sidecar-proxy": "yes", // The features (mTLS, LB, Circuit breaking) are no avaialble to services without istio-proxy. Only inbound routing is possible.

	// Traffic management fields below are valid only if the sidecar-proxy is set to "yes"
	traffic-management-info : {
	// Traffic configuration - Loadbalancing is applicable per service. The traffic to this service is distrbuted amongst the pods under it.
	"loadbalancingType": "ConsistenHash", // "Simple" and "consistentHash" are the two modes - {istioobject - destinationRule}
	"loadBalancerMode": "httpCookie" // Modes for consistentHash - "httpHeaderName", "httpCookie", "useSourceIP", "minimumRingSize", Modes for simple - "LEAST_CONN", "ROUND_ROBIN", 	"RANDOM", "PASSTHROUGH" // choices of the mode must be explicit - {istioobject - destinationRule}
	"httpCookie": "user2" // Name of the cookie to maitain sticky sessions - {istioobject - destinationRule}

	// Circuit Breaking
	"maxConnections": 10 //connection pool for tcp and http traffic - {istioobject - destinationRule}
	"concurrenthttp2Requests": 1000 // concurent http2 requests which can be allowed - {istioobject - destinationRule}
	"httpRequestPerConnection": 100 // number of http requests per connection. Valid only for http traffic - {istioobject - destinationRule}
	"consecutiveErrors": 8 // Default is 5.  Number of consecutive error before the host is removed - {istioobject - destinationRule}
	"baseEjectionTime" : 15 // Default is 5, - {istioobject - destinationRule}
	"intervalSweep": 5m, //time limit before the removed hosts are added back to the load balancing pool. - {istioobject - destinationRule}
	
	}

	// credentials for mTLS.
	"Servicecertificate" : "" // Present actual certificate here.
	"ServicePrivateKey" : "" // Present actual private key here.
	"caCertificate": "" // Trusted caCertificates used to verify the client

	// Access Control
	"namespaces": [] // Workloads from this namespaces can access the inbound service
	serviceAccountAccess : {"cluster.local/ns/default/sa/sleep": {"GET": "/static"}} // {istioobject - authorizationPolicy, will be applied for the inbound service}
	
	}
}

RETURN STATUS: 201
RETURN BODY: 
{ 
  "name": "<name>"
  "Message": "inbound service created"
}

Add Authorization Policy to the inbound service

POST
URL: /v2/projects/{project-name}/composite-apps/blue-app/{version}/traffic-intent-set/inbound-intents/{serviceName}/authrization-policies

POST BODY:
{
	"metadata": {
	"name": "<httpbin>" // unique name for each intent
    "description": "Authorization Policy for the client"
	"userdata1": <>,
	"userdata2": <>
	}

	"spec": {
	// Access Control
	"namespaces": [] // Workloads from this namespaces can access the inbound service
	serviceAccountAccess : {"cluster.local/ns/default/sa/sleep": {"GET": "/static"}} // {istioobject - authorizationPolicy, will be applied for the inbound service}
	
	}
}

RETURN STATUS: 201
RETURN BODY: 
{ 
  "name": "<name>"
  "Message": "inbound service created"
}
Add Clients to inbound service 02
Client 01

POST - traffic intent to add clients for accessing a specific inbound service

POST
URL: /v2/projects/{project-name}/composite-apps/brown-app/{version}/traffic-group-intent/outbound-intents/httpbin/
POST BODY:
{
	"metadata": {
	"name": "<name>" // unique name for each intent
    "description": "connectivity intent add client communication"
	"application": "<app1>",
	"userdata1": <>,
	"userdata2": <>
	}

	spec: {
		"clientServiceName": "sleep", // Name of the client service
		"type": "istio", // options are istio, k8s and external
		"headless": "false", // default is false. Option "True" will generate the required configs for all the instances of headless service
		"targetService": "productpage.k8s.com"
	}
}

RETURN STATUS: 201
RETURN BODY:
{ 
  "name": "<name>"
  "Message": "Client created"
}
Client 02

POST - traffic intent to add clients for accessing a specific inbound service

POST
URL: /v2/projects/{project-name}/composite-apps/brown-app/{version}/traffic-group-intent/outbound-intents/httpbin
POST BODY:
{
	"metadata": {
	"name": "<name>" // unique name for each intent
    "description": "connectivity intent add client communication"
	"application": "<app1>",
	"userdata1": <>,
	"userdata2": <>
	}

	spec: {
		"clientServiceName": "bookinfo-user", // Name of the client service
		"type": "istio", // options are istio, k8s and external
		"headless": "false", // default is false. Option "True" will generate the required configs for all the instances of headless service
		"inboundServiceName": "productpage.k8s.com"		
	}
}

RETURN STATUS: 201
RETURN BODY:
{ 
  "name": "<name>"
  "Message": "Client created"
}

Generate Istio object resources

Name of the ClusterMicroserviceIstio ConfigurationComments
Cluster01
MicroserviceLogicalcloud01Logicalcloud02
Common access for httpbinserviceEntry (httpbin)
sleepdestinationRule
bookinfo-productpage
AuthorizationPolicy, destinationRule 

Cluster02
MicroserviceLogicalcloud01Logicalcloud02
common access for bookinfo-productpageserviceEntry
bookinfo-userdestinationRule
sleepdestinationRule

Cluster01 Resources

1. ServiceEntry - To enable sleep to access to httpbin (logicalcloud01) 

ServiceEntry
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: service-entry-httpbin
  namespace: <> // namespace where the client service are deployed
spec:
  hosts:
  - httpbin.<namespace_of_service>.logicalcloud02 // which is the translation of "httpbin.k8s.com"
  # template for the remote service name - <servicename.namespace.global>
  # Treat remote cluster services as part of the service mesh
  # as all clusters in the service mesh share the same root of trust.
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: DNS
  addresses:
  # the IP address to which httpbin.<namespace>.<logicalcloudname> will resolve to
  # must be unique for each remote service, within a given cluster.
  # This address need not be routable. Traffic for this IP will be captured
  # by the sidecar and routed appropriately.
  - 240.0.0.2
  endpoints:
  # This is the routable address of the istio ingress gateway in cluster02
  # routed to this address.
  - address: 172.25.55.50 // IP of the istio-ingress-gateway
    ports:
      http1: 15443 //Sni. Do not change this

2.  DestinationRule for TLS - sleep (logicalcloud01)

DestinationRule
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sleep-dr
  namespace: <namespace_of_sleep>
spec:
  host: "sleep"
  trafficPolicy:
    tls:
      mode: MUTUAL
      serverCertificate: /etc/certs/cert-chain.pem
      privateKey: /etc/certs/key.pem
      caCertificates: /etc/certs/root-cert.pem

3.  DestinationRule for TLS, Loadbalancing and circuit breaking - productpage (logicalCloud02)

DestinationRule
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: bookinfo-productpage-dr
  namespace: <namespace_of_productpage>
spec:
  host: "productpage"
  trafficPolicy:
    tls:
	  mode: MUTUAL
      serverCertificate: /etc/certs/cert-chain.pem
      privateKey: /etc/certs/key.pem
      caCertificates: /etc/certs/root-cert.pem
    loadbalancer:
      consistentHash:
        httpCookie: "user2"
    connectionPool:
      tcp:
        maxConnections: 10
      http:
        http2MaxRequests: 1000
        maxRequestsPerConnection: 100
    outlierDetection:
      consecutiveErrors: 7
      interval: 5m
      baseEjectionTime: 15m

4. Gateway and Virtual Service resource to allow specific host headers and expose the service outside the cluster

DestinationRule
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
  namespace: namespace02
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*.local"
	- ".*logicalcloud01"
	- ".*logicalcloud02"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
    - "*.local"
	- ".*logicalcloud01"
	- ".*logicalcloud02"
  gateways:
  - bookinfo-gateway
  http:
  - match:
    - uri:
        prefix: /productpage.k8s.com
    route:
    - destination:
        port:
          number: 8000
        host: productpage.namespace02.local

5.  AuthorizationPolicy for bookinfo-productpage - (logicalCloud02)

AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: deny-all
 namespace: <namespace_of_prodfuct-page>
spec:
  selector:
   matchLabels:
     app: <name_used_for_productpage>
  rules:
  - from:
    - source:
        principals: ["cluster.global/ns/default/sa/sleep", "cluster.global/ns/default/sa/bookinfo-user" ]
    to:
    - operation:
        methods: ["GET"]
        paths: ["/static*"]
    - operation:
        methods: ["GET"]
        paths: ["/api/v1/products"]

Cluster 02 Resources

1. ServiceEntry - To enable access to bookinfo-productpage - (logicalCloud01)

ServiceEntry
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: service-entry-bookinfo-productpage
  namespace: namespace01
spec:
  hosts:
  - productpage.namespace01.logicalcloud01 // format is <svc>.<namespace>.<logical_cluster_domain>
  # template for the remote service name - <servicename.namespace.global>
  # Treat remote cluster services as part of the service mesh
  # as all clusters in the service mesh share the same root of trust.
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: DNS
  addresses:
  # the IP address to which httpbin.<namespace>.<logicalcloudname> will resolve to
  # must be unique for each remote service, within a given cluster.
  # This address need not be routable. Traffic for this IP will be captured
  # by the sidecar and routed appropriately.
  - 240.0.0.3
  endpoints:
  # This is the routable address of the istio ingress gateway in cluster02
  # routed to this address.
  - address: 172.25.55.210
    ports:
      http1: 15443 //Sni. Do not change this

2. DestinationRule for TLS - sleep - (logicalCloud01)

DestinationRule
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sleep-dr
  namespace: namespace01
spec:
  host: "sleep"
  trafficPolicy:
    tls:
	  mode: MUTUAL
      serverCertificate: /etc/certs/cert-chain.pem
      privateKey: /etc/certs/key.pem
      caCertificates: /etc/certs/root-cert.pem

3. DestinationRule for bookinfo-user - (logicalCloud01)

DestinationRule
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: sleep-dr
  namespace: namespace01
spec:
  host: "bookinfo-user"
  trafficPolicy:
    tls:
	  mode: MUTUAL
      serverCertificate: /etc/certs/cert-chain.pem
      privateKey: /etc/certs/key.pem
      caCertificates: /etc/certs/root-cert.pem

4.  DestinationRule for simple TLS, Loadbalancing and circuit breaking for httpbin - (logicalCloud02)

DestinationRule
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin-dr
  namespace: namespace02
spec:
  host: "httpbin"
  trafficPolicy:
    tls:
      mode: MUTUAL
	  serverCertificate: /etc/certs/cert-chain.pem
      privateKey: /etc/certs/key.pem
      caCertificates: /etc/certs/root-cert.pem
    loadbalancer:
      consistentHash:
        httpCookie: "user1"
    connectionPool:
      tcp:
        maxConnections: 10
      http:
        http2MaxRequests: 1000
        maxRequestsPerConnection: 100
    outlierDetection:
      consecutiveErrors: 7
      interval: 5m
      baseEjectionTime: 15m

5.  AuthorizationPolicy for httpbin - (logicalCloud02)

AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: deny-all
 namespace: namespace02
spec:
  selector:
   matchLabels:
     app: <app_Name_of_httpbin>
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/default/sa/sleep"]
    to:
    - operation:
        methods: ["GET"]
        paths: ["/status*"]
    - operation:
        methods: ["GET"]
        paths: ["/headers"]

6.  Gateway and Virtual Service resource to allow specific host headers and expose the service outside the cluster

DestinationRule
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
  namespace: namespace02
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*.local"
	- ".*logicalcloud01"
	- ".*logicalcloud02"
	- "onap.k8s.org"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-gateway
spec:
  hosts:
    - "*.local"
	- ".*logicalcloud01"
	- ".*logicalcloud02"
	- "onap.k8s.org"
  gateways:
  - httpbin-gateway
  http:
  - match:
    - uri:
        prefix: /httpbin.k8s.com
    route:
    - destination:
        port:
          number: 8000
        host: httpbin.namespace02.local