Description: - Connect multiple microservices on multiple logical clouds deployed across multiple clusters
Diagram
NOTE - For this scenario, the default mesh wide policy must be set to "PERMISSIVE" on both the clusters. It will not work if the default Mesh Policy is "STRICT"
Important Info - cert-chain.pem
is Envoy’s cert that needs to be presented to the other side. key.pem
is Envoy’s private key paired with Envoy’s cert in cert-chain.pem
. root-cert.pem
is the root cert to verify the peer’s cert. In this example, we only have one Citadel in a cluster, so all Envoys have the same root-cert.pem
.
Add Inbound service 01
POST - traffic intent for the inbound service (service hosted behind the cluster)
URL: /v2/projects/{project-name}/composite-apps/blue-app/{version}/traffic-intent-set/inbound-intents/ POST BODY: { "metadata": { "name": "<>" // unique name for each intent "description": "connectivity intent for inbound communication" "userdata1": <>, "userdata2": <> } "spec": { // update the memory allocation for each field as per OpenAPI standards "application": "<app1>", "servicename": "httpbin" //actual name of the client service - {istioobject - serviceEntry of client's cluster} "externalName": "httpbin.k8s.com" "protocol": "HTTP", "headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service "mutualTLS": "MUTUAL", // default is simple. Option MUTUAL will enforce mtls {istioobject - destinationRule} "port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on "serviceMesh": "istio", // get it from cluster record "sidecar-proxy": "yes", // The features (mTLS, LB, Circuit breaking) are not available to services without istio-proxy. Only inbound routing is possible. // Traffic management fields below are valid only if the sidecar-proxy is set to "yes" "traffic-management-info" : { // Traffic configuration - Loadbalancing is applicable per service. The traffic to this service is distrbuted amongst the pods under it. "loadbalancingType": "ConsistenHash", // "Simple" and "consistentHash" are the two modes - {istioobject - destinationRule} "loadBalancerMode": "httpCookie" // Modes for consistentHash - "httpHeaderName", "httpCookie", "useSourceIP", "minimumRingSize", Modes for simple - "LEAST_CONN", "ROUND_ROBIN", "RANDOM", "PASSTHROUGH" // choices of the mode must be explicit - {istioobject - destinationRule} "httpCookie": "user1" // Name of the cookie to maitain sticky sessions - {istioobject - destinationRule} // Circuit Breaking "maxConnections": 10 //connection pool for tcp and http traffic - {istioobject - destinationRule} "concurrenthttp2Requests": 1000 // concurent http2 requests which can be allowed - {istioobject - destinationRule} "httpRequestPerConnection": 100 // number of http requests per connection. Valid only for http traffic - {istioobject - destinationRule} "consecutiveErrors": 8 // Default is 5. Number of consecutive error before the host is removed - {istioobject - destinationRule} "baseEjectionTime" : 15 // Default is 5, - {istioobject - destinationRule} "intervalSweep": 5m, //time limit before the removed hosts are added back to the load balancing pool. - {istioobject - destinationRule} } // credentials for mTLS. "Servicecertificate" : "" // Present actual certificate here. "ServicePrivateKey" : "" // Present actual private key here. "caCertificate" : "" // present the trusted certificate to verify the client connection, Required only when mtls mode is MUTUAL } } RETURN STATUS: 201 RETURN BODY: { "name": "<name>" "Message": "inbound service created" }
Authorization for Inbound Service 01
URL: /v2/projects/{project-name}/composite-apps/blue-app/{version}/traffic-intent-set/inbound-intents/{serviceName}/authorization-policies POST BODY: { "metadata": { "name": "<>" // unique name for each intent "description": "Authorization Policy for inbound services" "userdata1": <>, "userdata2": <> } "spec": { // Access Control "namespaces": [] // Workloads from this namespaces can access the inbound service - {istioobject - authorizationPolicy} "serviceAccountAccess" : {[ "cluster.local/ns/<Namespace>/sa/sleep": ["GET": "/status"]} // {istioobject - authorizationPolicy, will be applied for the inbound service} } } RETURN STATUS: 201 RETURN BODY: { "name": "<name>" "Message": "Authorizations Policy created" }
Client 01
POST - traffic intent to add clients for accessing a specific inbound service - NOTE - Clients will have the mTLS mode same as the inbound service
URL: /v2/projects/{project-name}/composite-apps/brown-app/{version}/traffic-group-intent/outbound-intents/ POST BODY: { "metadata": { "name": "<name>" // unique name for each intent "description": "connectivity intent for outbound communication" "application": "<app1>", "userdata1": <>, "userdata2": <> } spec: { "ServiceName": "sleep", // Name of the client service "type": "istio", // options are istio, k8s and external "headless": "false", // default is false. Option "True" will generate the required configs for all the instances of headless service "targetServiceName": "httpbin.k8s.com" // FQDN expected since the client belongs to a different composite app } } RETURN STATUS: 201 RETURN BODY: { "name": "<name>" "Message": "Client created" }
Client 02
POST - traffic intent to add clients for accessing a specific inbound service - NOTE - Clients will have the mTLS mode same as the inbound service
URL: /v2/projects/{project-name}/composite-apps/blue-app/{version}/traffic-group-intent/outbound-intents/ POST BODY: { "metadata": { "name": "<name>" // unique name for each intent "description": "connectivity intent add client communication" "application": "<app1>", "userdata1": <>, "userdata2": <> } spec: { "clientServiceName": "sleep", // Name of the client service "type": "istio", // options are istio, k8s and external "headless": "false", // default is false. Option "True" will generate the required configs for all the instances of headless service "targetService": "httpbin.k8s.com" // Both client and service belong to the same composite. This notation is still used for consistency } } RETURN STATUS: 201 RETURN BODY: { "name": "<name>" "Message": "Client created" }
Client 03
POST - traffic intent to add clients for accessing a specific inbound service - NOTE - Clients will have the mTLS mode same as the inbound service
URL: /v2/projects/{project-name}/composite-apps/brown-app/{version}/traffic-group-intent/outbound-intents/ POST BODY: { "metadata": { "name": "<name>" // unique name for each intent "description": "connectivity intent add client communication" "application": "<app1>", "userdata1": <>, "userdata2": <> } spec: { "clientServiceName": "onap.k8s.org", // Name of the client service "type": "external", // options are istio, k8s and external "headless": "false", // default is false. Option "True" will generate the required configs for all the instances of headless service "targetService": "httpbin.k8s.com" } } RETURN STATUS: 201 RETURN BODY: { "name": "<name>" "Message": "Client created" }
Add Inbound service 02
POST
URL: /v2/projects/{project-name}/composite-apps/blue-app/{version}/traffic-intent-set/inbound-intents/ POST BODY: { "metadata": { "name": "<httpbin>" // unique name for each intent "description": "connectivity intent for stateless micro-service to stateless micro-service communication" "userdata1": <>, "userdata2": <> } "spec": { "application": "<app1>", "servicename": "productpage" //actual name of the client service "externalName": "productpage.k8s.com" "protocol": "HTTP", "headless": "false", // default is false. Option "True" will make sure all the instances of the headless service will have access to the client service "mutualTLS": "MUTUAL", // default is simple. Option MUTUAL will enforce mtls "port" : "80", // port on which service is exposed as through servicemesh, not the port it is actually running on "serviceMesh": "istio", // get it from cluster record "sidecar-proxy": "yes", // The features (mTLS, LB, Circuit breaking) are no avaialble to services without istio-proxy. Only inbound routing is possible. // Traffic management fields below are valid only if the sidecar-proxy is set to "yes" traffic-management-info : { // Traffic configuration - Loadbalancing is applicable per service. The traffic to this service is distrbuted amongst the pods under it. "loadbalancingType": "ConsistenHash", // "Simple" and "consistentHash" are the two modes - {istioobject - destinationRule} "loadBalancerMode": "httpCookie" // Modes for consistentHash - "httpHeaderName", "httpCookie", "useSourceIP", "minimumRingSize", Modes for simple - "LEAST_CONN", "ROUND_ROBIN", "RANDOM", "PASSTHROUGH" // choices of the mode must be explicit - {istioobject - destinationRule} "httpCookie": "user2" // Name of the cookie to maitain sticky sessions - {istioobject - destinationRule} // Circuit Breaking "maxConnections": 10 //connection pool for tcp and http traffic - {istioobject - destinationRule} "concurrenthttp2Requests": 1000 // concurent http2 requests which can be allowed - {istioobject - destinationRule} "httpRequestPerConnection": 100 // number of http requests per connection. Valid only for http traffic - {istioobject - destinationRule} "consecutiveErrors": 8 // Default is 5. Number of consecutive error before the host is removed - {istioobject - destinationRule} "baseEjectionTime" : 15 // Default is 5, - {istioobject - destinationRule} "intervalSweep": 5m, //time limit before the removed hosts are added back to the load balancing pool. - {istioobject - destinationRule} } // credentials for mTLS. "Servicecertificate" : "" // Present actual certificate here. "ServicePrivateKey" : "" // Present actual private key here. "caCertificate": "" // Trusted caCertificates used to verify the client // Access Control "namespaces": [] // Workloads from this namespaces can access the inbound service serviceAccountAccess : {"cluster.local/ns/default/sa/sleep": {"GET": "/static"}} // {istioobject - authorizationPolicy, will be applied for the inbound service} } } RETURN STATUS: 201 RETURN BODY: { "name": "<name>" "Message": "inbound service created" }
Add Authorization Policy to the inbound service
URL: /v2/projects/{project-name}/composite-apps/blue-app/{version}/traffic-intent-set/inbound-intents/{serviceName}/authrization-policies POST BODY: { "metadata": { "name": "<httpbin>" // unique name for each intent "description": "Authorization Policy for the client" "userdata1": <>, "userdata2": <> } "spec": { // Access Control "namespaces": [] // Workloads from this namespaces can access the inbound service serviceAccountAccess : {"cluster.local/ns/default/sa/sleep": {"GET": "/static"}} // {istioobject - authorizationPolicy, will be applied for the inbound service} } } RETURN STATUS: 201 RETURN BODY: { "name": "<name>" "Message": "inbound service created" }
Add Clients to inbound service 02
Client 01
POST - traffic intent to add clients for accessing a specific inbound service
URL: /v2/projects/{project-name}/composite-apps/brown-app/{version}/traffic-group-intent/outbound-intents/httpbin/ POST BODY: { "metadata": { "name": "<name>" // unique name for each intent "description": "connectivity intent add client communication" "application": "<app1>", "userdata1": <>, "userdata2": <> } spec: { "clientServiceName": "sleep", // Name of the client service "type": "istio", // options are istio, k8s and external "headless": "false", // default is false. Option "True" will generate the required configs for all the instances of headless service "targetService": "productpage.k8s.com" } } RETURN STATUS: 201 RETURN BODY: { "name": "<name>" "Message": "Client created" }
Client 02
POST - traffic intent to add clients for accessing a specific inbound service
URL: /v2/projects/{project-name}/composite-apps/brown-app/{version}/traffic-group-intent/outbound-intents/httpbin POST BODY: { "metadata": { "name": "<name>" // unique name for each intent "description": "connectivity intent add client communication" "application": "<app1>", "userdata1": <>, "userdata2": <> } spec: { "clientServiceName": "bookinfo-user", // Name of the client service "type": "istio", // options are istio, k8s and external "headless": "false", // default is false. Option "True" will generate the required configs for all the instances of headless service "inboundServiceName": "productpage.k8s.com" } } RETURN STATUS: 201 RETURN BODY: { "name": "<name>" "Message": "Client created" }
Generate Istio object resources
Name of the Cluster | Microservice | Istio Configuration | Comments | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Cluster01 |
| ||||||||||||||
Cluster02 |
|
Cluster01 Resources
1. ServiceEntry - To enable sleep to access to httpbin (logicalcloud01)
apiVersion: networking.istio.io/v1alpha3 kind: ServiceEntry metadata: name: service-entry-httpbin namespace: <> // namespace where the client service are deployed spec: hosts: - httpbin.<namespace_of_service>.logicalcloud02 // which is the translation of "httpbin.k8s.com" # template for the remote service name - <servicename.namespace.global> # Treat remote cluster services as part of the service mesh # as all clusters in the service mesh share the same root of trust. location: MESH_INTERNAL ports: - name: http1 number: 8000 protocol: http resolution: DNS addresses: # the IP address to which httpbin.<namespace>.<logicalcloudname> will resolve to # must be unique for each remote service, within a given cluster. # This address need not be routable. Traffic for this IP will be captured # by the sidecar and routed appropriately. - 240.0.0.2 endpoints: # This is the routable address of the istio ingress gateway in cluster02 # routed to this address. - address: 172.25.55.50 // IP of the istio-ingress-gateway ports: http1: 15443 //Sni. Do not change this
2. DestinationRule for TLS - sleep (logicalcloud01)
apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: sleep-dr namespace: <namespace_of_sleep> spec: host: "sleep" trafficPolicy: tls: mode: MUTUAL serverCertificate: /etc/certs/cert-chain.pem privateKey: /etc/certs/key.pem caCertificates: /etc/certs/root-cert.pem
3. DestinationRule for TLS, Loadbalancing and circuit breaking - productpage (logicalCloud02)
apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: bookinfo-productpage-dr namespace: <namespace_of_productpage> spec: host: "productpage" trafficPolicy: tls: mode: MUTUAL serverCertificate: /etc/certs/cert-chain.pem privateKey: /etc/certs/key.pem caCertificates: /etc/certs/root-cert.pem loadbalancer: consistentHash: httpCookie: "user2" connectionPool: tcp: maxConnections: 10 http: http2MaxRequests: 1000 maxRequestsPerConnection: 100 outlierDetection: consecutiveErrors: 7 interval: 5m baseEjectionTime: 15m
4. Gateway and Virtual Service resource to allow specific host headers and expose the service outside the cluster
apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: bookinfo-gateway namespace: namespace02 spec: selector: istio: ingressgateway # use Istio default gateway implementation servers: - port: number: 80 name: http protocol: HTTP hosts: - "*.local" - ".*logicalcloud01" - ".*logicalcloud02" --- apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: httpbin spec: hosts: - "*.local" - ".*logicalcloud01" - ".*logicalcloud02" gateways: - bookinfo-gateway http: - match: - uri: prefix: /productpage.k8s.com route: - destination: port: number: 8000 host: productpage.namespace02.local
5. AuthorizationPolicy for bookinfo-productpage - (logicalCloud02)
apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: deny-all namespace: <namespace_of_prodfuct-page> spec: selector: matchLabels: app: <name_used_for_productpage> rules: - from: - source: principals: ["cluster.global/ns/default/sa/sleep", "cluster.global/ns/default/sa/bookinfo-user" ] to: - operation: methods: ["GET"] paths: ["/static*"] - operation: methods: ["GET"] paths: ["/api/v1/products"]
Cluster 02 Resources
1. ServiceEntry - To enable access to bookinfo-productpage - (logicalCloud01)
apiVersion: networking.istio.io/v1alpha3 kind: ServiceEntry metadata: name: service-entry-bookinfo-productpage namespace: namespace01 spec: hosts: - productpage.namespace01.logicalcloud01 // format is <svc>.<namespace>.<logical_cluster_domain> # template for the remote service name - <servicename.namespace.global> # Treat remote cluster services as part of the service mesh # as all clusters in the service mesh share the same root of trust. location: MESH_INTERNAL ports: - name: http1 number: 8000 protocol: http resolution: DNS addresses: # the IP address to which httpbin.<namespace>.<logicalcloudname> will resolve to # must be unique for each remote service, within a given cluster. # This address need not be routable. Traffic for this IP will be captured # by the sidecar and routed appropriately. - 240.0.0.3 endpoints: # This is the routable address of the istio ingress gateway in cluster02 # routed to this address. - address: 172.25.55.210 ports: http1: 15443 //Sni. Do not change this
2. DestinationRule for TLS - sleep - (logicalCloud01)
apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: sleep-dr namespace: namespace01 spec: host: "sleep" trafficPolicy: tls: mode: MUTUAL serverCertificate: /etc/certs/cert-chain.pem privateKey: /etc/certs/key.pem caCertificates: /etc/certs/root-cert.pem
3. DestinationRule for bookinfo-user - (logicalCloud01)
apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: sleep-dr namespace: namespace01 spec: host: "bookinfo-user" trafficPolicy: tls: mode: MUTUAL serverCertificate: /etc/certs/cert-chain.pem privateKey: /etc/certs/key.pem caCertificates: /etc/certs/root-cert.pem
4. DestinationRule for simple TLS, Loadbalancing and circuit breaking for httpbin - (logicalCloud02)
apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: httpbin-dr namespace: namespace02 spec: host: "httpbin" trafficPolicy: tls: mode: MUTUAL serverCertificate: /etc/certs/cert-chain.pem privateKey: /etc/certs/key.pem caCertificates: /etc/certs/root-cert.pem loadbalancer: consistentHash: httpCookie: "user1" connectionPool: tcp: maxConnections: 10 http: http2MaxRequests: 1000 maxRequestsPerConnection: 100 outlierDetection: consecutiveErrors: 7 interval: 5m baseEjectionTime: 15m
5. AuthorizationPolicy for httpbin - (logicalCloud02)
apiVersion: security.istio.io/v1beta1 kind: AuthorizationPolicy metadata: name: deny-all namespace: namespace02 spec: selector: matchLabels: app: <app_Name_of_httpbin> rules: - from: - source: principals: ["cluster.local/ns/default/sa/sleep"] to: - operation: methods: ["GET"] paths: ["/status*"] - operation: methods: ["GET"] paths: ["/headers"]
6. Gateway and Virtual Service resource to allow specific host headers and expose the service outside the cluster
apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: httpbin-gateway namespace: namespace02 spec: selector: istio: ingressgateway # use Istio default gateway implementation servers: - port: number: 80 name: http protocol: HTTP hosts: - "*.local" - ".*logicalcloud01" - ".*logicalcloud02" - "onap.k8s.org" --- apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: httpbin-gateway spec: hosts: - "*.local" - ".*logicalcloud01" - ".*logicalcloud02" - "onap.k8s.org" gateways: - httpbin-gateway http: - match: - uri: prefix: /httpbin.k8s.com route: - destination: port: number: 8000 host: httpbin.namespace02.local