...
When instantiating a distributed app with user facing microservices, a DNS Provider Intent is included as part of the Traffic Control Intent Set includes information to automate indicate the update set of external DNS servers which should be updated with appropriate DNS records for the user facing services. Each element of the DNS Provider Intent points to a collection of DNS Provider Information that provides the details for the specific DNS server that is to be updated. DNS Provider Information can be defined per cluster or per logical cloud for a given project. At the Project level, DNS Update Records are supplied to identify the type of DNS Provider to be updated along with a set of parameters required to perform the DNS record update. The 'external-dns' package is used to perform DNS update to a selected set of DNS providers. After the DNS Update Records have been created, they can be associated with specific logical clouds or clusters utilizing the Key Value pair feature of those APIs.
Each instance of a DNS Provider Information record Update Record has a unique name within the Project . The name was be used as a 'key' and the associated 'value' will be a JSON object which contains and is created and managed by the Traffic Controller API. The record will contain all of the parameters needed to invoke 'external-dns' for that specific provider. This key value pair is then added to the key value pair list of the logical cloud or the appropriate cluster. More than one DNS Provider Information record can be associated with a logical cloud or a cluster.
The following example illustrates the DNS PRovider Key Value pair. The idea is that the 'externalDnsParameters' portion will be used to create the parameters list for the 'external-dns' deployment that will be created to handle this DNS Provider.
...
Update Record API.
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| "<dnsprovidername>": {
"description": "description for DNS Provider",
"externalDnsParameters": { // list will be supplied to external-dns as parameters.
"aws-zone-type": "
| |||||||
URL: /v2/project/{project-name}/dns-update-records/ POST BODY: { "metadata": { "name": "name_of_dns_information_record", "description": "description of the DNS information", "userdata1": "some user data", "userdata2": "some different user data" When using the}, AWS provider, filter for zones of this type (optional, options: public, private) "spec": { "provider": "coredns", "awsexternal-zonedns-tagsparameters": "",{ "oneOf": [ When using the AWS provider, filter for zones with these tags "aws-assume-role":"", When using the AWS provider, assume this IAM role. Useful for hosted zones in another AWS account. Specify the full ARN, e.g. `arn:aws:## schema-like (array structure not in body) - one of the following is selected to match the 'provider' - e.g. 'coredns' in this case { "aws-zone-type": "public", "aws-zone-tags": "zone tags", "aws-assume-role": "arn:aws:iam::123455567:role/external-dns` (optional)dns", "aws-batch-change-size":" 1000", When using the AWS provider, set the maximum number of changes that will be applied in each batch. "aws-batch-change-interval":"1s", When using the AWS provider, set the interval between batch changes. "aws-batch-change-interval": "1s", "aws-evaluate-target-health": true, "no-aws-evaluate-target-health":"enabled" true, When using the AWS provider, set whether to evaluate the health of a DNS target (default: enabled, disable with --no-aws-evaluate-target-health) "aws-api-retries": 3, "aws-apiprefer-retriescname":"3", true When using the AWS provider}, set the maximum number of retries for{ API calls before giving up. "awsazure-preferconfig-cnamefile":"disabled" "/etc/kubernetes/azure.json", "azure-resource-group": "resource group", When using the AWS provider, prefer using CNAME instead of ALIAS (default: disabled) "azure-subscription-id": "subscription id", etc. } } |
Alternative approach (?) - DNS Providers becomes an explicit resource (e.g. v2/project/{project-name}/dns-providers/{dns-provider}) which are associated to logical-clouds and clusters via labels.
The following illustrates the API for creating a DNS Provider intent. To associate multiple DNS Providers with a DNS Provider Intent, multiple calls are made.
| Code Block | ||||
|---|---|---|---|---|
| ||||
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/{traffic-intent-set-name}/dnsproviders
POST BODY:
{
"name": "<dnsprovidername>" // values used as a key to find the DNS Provider Information details from the logical-cloud or cluster KV pair list.
}
|
| Code Block | ||||
|---|---|---|---|---|
| ||||
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/{traffic-intent-set-name}/dnsproviders
GET RESPONSE BODY:
[
{
"name": "dnsprovidername1"
},
{
"name": "dnsprovidername2"
},
{
"name": "dnsprovidername3"
},
{
"name": "dnsprovidername4"
}
]
URL: /v2/project/{project-name}/rb/{rb-name}/{rb-version}/traffic-intent-sets/{traffic-intent-set-name}/dnsproviders/dnsprovidername2
GET RESPONSE BODY:
{
"name": "dnsprovidername2"
}
|
DNS Provider Intent Handling
After DNS Provider KV pairs are created and associated with logical cloud and/or clusters and these DNS Providers are added to a traffic control intent set, the traffic intent set will be associated with an intent group, which in turn will be provided as a parameter when a call to prepare for the instantiation of a distribued application is made. Eventually, the multicloud orchestrator will invoke the traffic controller to process the traffic controller intents. One part of that process will be to handle the DNS Provider Intents.
The following sequence diagram illustrates what happens:
The DNS Provider Intent handling consists of two key tasks:
- Prepare manifests for external-dns Deployments which will handle the updating of specific DNS-Providers. There is a separate external-dns Deployment for each DNS Provider (based on current understanding of how external-dns works).
- For a given project, there should only need to be one external-dns Deployment to handle all distributed Apps that are deployed - so, maybe the App Context is not the right location for these Deployment manifests.
- The 'source' for external-dns will always be a DNSendpointCRD for a namespace in the project. An enhancement to external-dns is expected necessary to handle filtering the DNSendpointCRD based on a label to associate specific CRs with a DNS Provider.
- Prepare DNSendpointCRD manifests for user facing services present in the App Context.
The following show an example of a DNS Endpoint CR instance.
| Code Block | ||||
|---|---|---|---|---|
| ||||
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
name: microservice1-dnsrecords
namespace: projectnamespace1
labels:
dnsprovider: dnsprovidername2
spec:
endpoints:
- dnsName: abc.123.com
recordTTL: 180
recordType: A
targets:
- 1.2.3.4
- 5.6.7.8
|
see also: https://github.com/kubernetes-sigs/external-dns/tree/master/docs/contributing/crd-source
To associate these with specific DNS Providers, the proposal is to add a label which will allow external-dns (after modification) to handle only DNS Endpont CRs that match the label (in addition to the namespace) for a given DNS Provider.
The 'name' of the DNS Endpoint CR can be derived from the sub-app (microservice) that is exposing these endponts.
The DNS Provider Intent handler creates the manifests for these DNS Endpoint CRs using the following algorithm:
| Code Block | ||||
|---|---|---|---|---|
| ||||
for each DNS provider intent find the KV pair for this intent in either the logical cloud or a cluster prepare a list of clusters that apply for this intent for each cluster for each sub-app with an external FQDN defined (e.g. in an ISTIO gateway) acquire the IP(s) associated with cluster and scope of the DNS Provider create DNSendpointCRD(s) manifests including a label with the DNSproviderName "azure-user-assigned-identity-client-id": "client id" }, { "coredns-prefix": "skydns" }, { "rfc2136-host": "host.sample.com", "rfc2136-zone": "", "rfc2136-insecure": false, "rfc2136-tsig-keyname": "tsig key", "rfc2136-tsig-secret": "tsig secret", "rfc2136-tsig-secret-alg": "tsig secret alg", "rfc2136-tsig-axfr": "axfr", "request-timeout": "30s" } ], "contour-load-balancer": "heptio-contour/contour", ## from here on, possible parameters used by external-dns - not all required "fqdn-template": "template", "combine-fqdn-annotation": true, "ignore-hostname-annotation": true, "compatibility": "mate", "publish-internal-services": true, "publish-host-ip": true, "service-type-filter": "all", "domain-filter": "example.com", "exclude-domains": "example.com", "zone-id-filter": "zone filter", "tls-ca": "tls ca path", "tls-client-cert": "tls client cert path", "tls-client-cert-key": "tls client cert key path", "policy": "sync", "registry": "txt", "txt-owner-id": "default", "txt-prefix": "custom string", "txt-cache-interval": "30s", "interval": "30s", "once": true, "dry-run": true, "log-format": "text", "metrics-address": ":7979", "log-level": "info" } } } |
The following shows an OpenAPI component definition of the above DNS Update Record structure (note: in progress). Parameters with descriptions are shown in more detail.
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
openapi: 3.0.0
info:
title: DNS update information record
description: Defines information needed to update external DNS
version: 2.0.0
paths:
<...tbd...>
components:
schemas:
dnsrecord:
properties:
metadata:
$ref: '#/components/schemas/metadata'
spec:
$ref: '#/components/schemas/extdns_info'
extdns_info:
type: "object"
properties:
provider:
type: "string"
enum: ["aws", "aws-sd", "azure", "azure-dns", "azure-private-dns", "coredns", "rfc2136"]
example: "coredns"
external-dns-parameters:
allOf:
- $ref: '#/components/schemas/extdns_provider'
- $ref: '#/components/schemas/extdns_general'
metadata:
type: "object"
required:
- name
properties:
name:
type: "string"
example: "name_of_dns_information_record"
description:
type: "string"
example: "description of the DNS information"
userdata1:
type: "string"
example: "some user data"
userdata2:
type: "string"
example: "some different user data"
extdns_general:
type: "object"
required:
- provider
properties:
request-timeout:
type: "string"
example: "30s"
description: "Request timeout when calling Kubernetes APIs. 0s means no timeout"
contour-load-balancer:
type: "string"
example: "heptio-contour/contour"
default: "heptio-contour/contour"
description: "The fully-qualified name of the Contour load balancer service."
fqdn-template:
type: "string"
example: "template"
description: "A templated string that's used to generate DNS names from sources that don't define a hostname themselves, or to add a hostname suffix when paired with the fake source (optional). Accepts comma separated list for multiple global FQDN."
combine-fqdn-annotation:
type: "boolean"
example: true
description: "Combine FQDN template and Annotations instead of overwriting"
ignore-hostname-annotation:
type: "boolean"
example: true
default: false
description: "Ignore hostname annotation when generating DNS names, valid only when using fqdn-template is set (optional, default: false)"
compatibility:
type: "string"
example: "mate"
enum: ["mate", "molecule"]
description: "Process annotation semantics from legacy implementations (optional, options: mate, molecule)"
publish-internal-services:
type: "boolean"
example: true
description: "Allow external-dns to publish DNS records for ClusterIP services (optional)"
publish-host-ip:
type: "string"
example: true
description: "Allow external-dns to publish host-ip for headless services (optional)"
service-type-filter:
type: "string"
example: "all"
default: "all"
enum: ["all", "ClusterIP", "NodePort", "LoadBalancer", "ExternalName"]
description: "The service types to take care about (default: all, expected: ClusterIP, NodePort, LoadBalancer or ExternalName)"
domain-filter:
type: "string"
example: "example.com"
description: "Limit possible target zones by a domain suffix; specify multiple times for multiple domains (optional)"
exclude-domains:
type: "string"
example: "example.com"
description: "Exclude subdomains (optional)"
zone-id-filter:
type: "string"
example: "zone filter"
description: "Filter target zones by hosted zone id; specify multiple times for multiple zones (optional)"
tls-ca:
type: "string"
example: "tls ca path"
description: "When using TLS communication, the path to the certificate authority to verify server communications (optionally specify --tls-client-cert for two-way TLS)"
tls-client-cert:
type: "string"
example: "tls client cert path"
description: "When using TLS communication, the path to the certificate to present as a client (not required for TLS)"
tls-client-cert-key:
type: "string"
example: "tls client cert key path"
description: "When using TLS communication, the path to the certificate key to use with the client certificate (not required for TLS)"
policy:
type: "string"
example: "sync"
default: "sync"
enum: ["sync", "upsert-only", "create-only"]
description: "Modify how DNS records are synchronized between sources and providers (default: sync, options: sync, upsert-only, create-only)"
registry:
type: "string"
example: "txt"
default: "txt"
enum: ["txt", "noop", "aws-sd"]
description: "The registry implementation to use to keep track of DNS record ownership (default: txt, options: txt, noop, aws-sd)"
txt-owner-id:
type: "string"
example: "default"
default: "default"
description: "When using the TXT registry, a name that identifies this instance of ExternalDNS (default: default)"
txt-prefix:
type: "string"
example: "custom string"
description: "When using the TXT registry, a custom string that's prefixed to each ownership DNS record (optional)"
txt-cache-interval:
type: "string"
example: "30s"
description: "The interval between cache synchronizations in duration format (default: disabled)"
interval:
type: "string"
example: "30s"
default: "1m"
description: "The interval between two consecutive synchronizations in duration format (default: 1m)"
once:
type: "boolean"
example: true
description: "When enabled, exits the synchronization loop after the first iteration (default: disabled)"
dry-run:
type: "boolean"
example: true
description: "When enabled, prints DNS record changes rather than actually performing them (default: disabled)"
log-format:
type: "string"
example: "text"
default: "text"
enum: ["text", "json"]
description: "The format in which log messages are printed (default: text, options: text, json)"
metrics-address:
type: "string"
example: ":7979"
description: "Specify where to serve the metrics and health check endpoint (default: :7979)"
log-level:
type: "string"
example: "info"
default: "info"
enum: ["panic", "debug", "info", "warning", "error", "fatal"]
description: "Set the level of logging. (default: info, options: panic, debug, info, warning, error, fatal"
extdns_provider:
oneOf:
- $ref: '#/components/schemas/extdns_provider_aws'
- $ref: '#/components/schemas/extdns_provider_azure'
- $ref: '#/components/schemas/extdns_provider_coredns'
- $ref: '#/components/schemas/extdns_provider_rfc2136'
extdns_provider_aws:
type: "object"
properties:
aws-zone-type:
type: "string"
example: "public"
enum: ["public", "private"]
description: "When using the AWS provider, filter for zones of this type (optional, options: public, private)"
aws-zone-tags:
type: "string"
example: "zone tags"
description: "When using the AWS provider, filter for zones with these tags"
aws-assume-role:
type: "string"
example: "arn:aws:iam::123455567:role/external-dns"
description: "When using the AWS provider, assume this IAM role. Useful for hosted zones in another AWS account. Specify the full ARN, e.g. `arn:aws:iam::123455567:role/external-dns` (optional)"
aws-batch-change-size:
type: "integer"
example: 1000
description: "When using the AWS provider, set the maximum number of changes that will be applied in each batch."
aws-batch-change-interval:
type: "string"
example: "1s"
description: "When using the AWS provider, set the interval between batch changes."
aws-evaluate-target-health:
type: "boolean"
example: true
description: "When using the AWS provider, set whether to evaluate the health of a DNS target (default: enabled, disable with --no-aws-evaluate-target-health)"
no-aws-evaluate-target-health:
type: "boolean"
example: true
description: "When using the AWS provider, set whether to evaluate the health of a DNS target (default: enabled, disable with --no-aws-evaluate-target-health)"
aws-api-retries:
type: "integer"
example: 3
description: "When using the AWS provider, set the maximum number of retries for API calls before giving up."
aws-prefer-cname:
type: "boolean"
example: true
description: "When using the AWS provider, prefer using CNAME instead of ALIAS (default: disabled)"
extdns_provider_azure:
type: "object"
properties:
azure-config-file:
type: "string"
example: "/etc/kubernetes/azure.json"
description: "When using the Azure provider, specify the Azure configuration file (required when --provider=azure"
azure-resource-group:
type: "string"
example: "resource group"
description: "When using the Azure provider, override the Azure resource group to use (required when --provider=azure-private-dns)"
azure-subscription-id:
type: "string"
example: "subscription id"
description: "When using the Azure provider, specify the Azure configuration file (required when -ws-zone-type-provider=azure-private-dns)"
azure-user-assigned-identity-client-id:
type: "string"
example: "client id"
description: "When using the Azure provider, override the client id of user assigned identity in config file (optional)"
extdns_provider_coredns:
type: "object"
properties:
coredns-prefix:
type: "string"
example: "skydns"
description: "When using the CoreDNS provider, specify the prefix name"
extdns_provider_rfc2136:
type: "object"
properties:
rfc2136-host:
type: "string"
example: "host.sample.com"
description: "When using the RFC2136 provider, specify the host of the DNS server"
rfc2136-zone:
type: "string"
example: ""
description: "When using the RFC2136 provider, specify the zone entry of the DNS server to use"
rfc2136-insecure:
type: "boolean"
example: false
description: "When using the RFC2136 provider, specify whether to attach TSIG or not (default: false, requires --rfc2136-tsig-keyname and --rfc2136-tsig-secret)"
rfc2136-tsig-keyname:
type: "string"
example: "tsig key"
description: "When using the RFC2136 provider, specify the TSIG key to attached to DNS messages (required when --rfc2136-insecure=false)"
rfc2136-tsig-secret:
type: "string"
example: "tsig secret"
description: "When using the RFC2136 provider, specify the TSIG (base64) value to attached to DNS messages (required when --rfc2136-insecure=false)"
rfc2136-tsig-secret-alg:
type: "string"
example: "tsig secret alg"
description: "When using the RFC2136 provider, specify the TSIG (base64) value to attached to DNS messages (required when --rfc2136-insecure=false)"
rfc2136-tsig-axfr:
type: "string"
example: "axfr"
description: "When using the RFC2136 provider, specify the TSIG (base64) value to attached to DNS messages (required when --rfc2136-insecure=false)"
|
The Key Value pair list that is added to the logical-clouds and clusters will look like the following:
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
{
"metadata":{
"name":"dns-update-record-list",
"description":"list of the DNS update records associated with this logical-cloud or cluster",
"userData1":"<user data>",
"userData2":"<user data>"
},
"spec":{
"kv":[
{
"key1":"dns-update-record-1"
},
{
"key2":"dns-update-record-47"
}
]
}
}
|
Traffic Control Intent Handling - DNS Update Records
After DNS Update Records are created and associated with logical cloud and/or clusters and associated with the Traffic Control intents of the distributed application, a call is made to instantiate a Profile. Eventually, the multicloud orchestrator will invoke the traffic controller to process the traffic controller intents. One part of that process will be to handle the DNS Update Records associated with the Profile (via the intent).
The following sequence diagram illustrates what happens:
The DNS Update Record handling consists of two key tasks:
- Prepare manifests for external-dns Deployments which will handle the updating of specific DNS-Providers. There is a separate external-dns Deployment for each DNS Provider (based on current understanding of how external-dns works).
- For a given project, there should only need to be one external-dns Deployment to handle all distributed Apps that are deployed - so, maybe the App Context is not the right location for these Deployment manifests.
- The 'source' for external-dns will always be a DNSendpointCRD for a namespace in the project. An enhancement to external-dns is expected necessary to handle filtering the DNSendpointCRD based on a label to associate specific CRs with a DNS Provider.
- Prepare DNSendpointCRD manifests for user facing services present in the App Context.
The following show an example of a DNS Endpoint CR instance.
| Code Block | ||||
|---|---|---|---|---|
| ||||
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
name: microservice1-dnsrecords
namespace: projectnamespace1
labels:
dnsprovider: dnsprovidername2
spec:
endpoints:
- dnsName: abc.123.com
recordTTL: 180
recordType: A
targets:
- 1.2.3.4
- 5.6.7.8
|
see also: https://github.com/kubernetes-sigs/external-dns/tree/master/docs/contributing/crd-source
To associate these with specific DNS Providers, the proposal is to add a label which will allow external-dns (after modification) to handle only DNS Endpont CRs that match the label (in addition to the namespace) for a given DNS Provider.
The 'name' of the DNS Endpoint CR can be derived from the sub-app (microservice) that is exposing these endpoints.
The DNS Update Record handler creates the manifests for these DNS Endpoint CRs using the following algorithm:
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
Get the TC Intent Set - and acquire the Inbound Intents
Get the Generic Placement Intents to find the list of logical-clouds for this composite app
For each inbound intent in the TC set (e.g. each App with an inbound intent)
If externalName and publicDomain (e.g. FQDN: <externalName>.<publicDomain>) are set
Identify the logical-cloud associated with the App
Query the logical-cloud KV for the DNS Update Record list associated with the logical cloud
For all DNS Update Records
Create an external-dns deployment for the Provider information in the DNS Update Record
Source will be DNSendpointCRs labeled with this composite-app/dns-context (e.g. logical cloud context)
For each cluster in which this app is deployed
Obtain the IP address for public network access
Create/Update the DNSendpointCRD with endpoints for this App
If externalName and localDomain (e.g. FQDN: <externalName>.<localDomain>) are set
Query the cluster KV for the DNS Update Record list associated with the cluster
For each logical cloud this app is deployed to
For each cluster this app is deployed to
For all DNS Update Records in the logical cloud 'dns-update-record-list' KV pair
Create an external-dns deployment
Source will be DNSendpointCRs labeled with this composite-app/dns-context (e.g. cluster context)
Create DNSendpointCRD records to be handled by this deployment
|
Notes:
- There will need to be a way to find the appropriate set of IPs to use. Some IPs will be appropriate for a public scope (e.g. update a DNS Provider associated with the logical cloud) and others may be local to the cluster network (e.g. cluster DNS Providers).
...
The following illustrates how the external DNS updates are triggered once the Resource Synchronization of the distributed app is performed.
The external-dns deployment for a given DNS Provider may be utilized for more than one distributed application, so some method for detecting that there is already a deployment running is needed.
...