Experimenting with blue-green cluster migration using Amazon Route 53 and ExternalDNS
At Branch, most of our infrastructure runs on Amazon EKS (Elastic Kubernetes Service). However, a significant challenge is that EKS requires periodic version upgrades, which can bring breaking changes or disruptions to production workloads in the cluster.
While solutions like Argo Rollouts offer support for in-cluster service canary deployments and blue-green migration, there is no simple solution to address the concerns of cross-cluster service migrations. To help solve this, our team has begun experimenting with a new strategy for cross-cluster, blue-green upgrades: using Route 53 weighted routing and ExternalDNS¹. This does not address all the pain points of cluster upgrades, but it does simplify workload and traffic migration without the need for a complex solution like a service mesh².
In this article, we will describe how each component of our blue-green system is configured, and share our learnings.
Side note: what is a blue-green deployment?
In a blue-green deployment, two clusters are maintained. This allows one cluster (running an old version of EKS) to serve production traffic while the other cluster is being upgraded.
If we take an example where the “blue” cluster is serving the production using an old version of EKS, and the “green” cluster is running a new version, deploying the EKS upgrade simply involves migrating the workload from the “blue” cluster to the “green” cluster. After all the workloads are migrated to the “green” cluster, the “blue” cluster is decommissioned.
Then, during the next version upgrade, the process is reversed: the “blue” cluster is upgraded, and the production workload from the “green” cluster is migrated back to the “blue” cluster.
While the migration is in progress, the external service that is identified by the same name (e.g. a URL) is handled proportionally by the upstream from both clusters. After the migration is complete, only one cluster will handle requests to the external service.
The blue-green strategy offers the advantage of quickly rolling back to a previous state if anything goes wrong, simply by routing traffic back to EKS cluster that is still on the old version of Kubernetes.
Thanks for the contribution from our infra team. Special thanks to my colleague, Grzegorz Kolano, for the original idea, research and the initial implementation.
Goals
For the purpose of this article, we will use a fictional “acme” web service as an example. The “acme” service runs on a v1.21 EKS cluster named blue-phoenix (the “blue” cluster). The goal is to upgrade EKS to v1.22 in a new cluster named green-phoenix (the “green” cluster) and move all the “acme” workload from the blue to the green cluster.
From the external user’s perspective, the consumer uses the acme service URL, https://acme.phoenix.branch.io, to access the service. The external URL will remain color-neutral and constant during the migration, and the user does not need to know which exact acme Kubernetes pod is serving the their request.
This means the network traffic can go to the backend hosted in either the blue or green cluster, and the traffic will be split between the two backends from different clusters while the blue-green migration is taking place.
In order to support the blue-green migration, we need:
- The backend service to be configured as a K8s LoadBalancer Service (Ingress should work similarly).
- Functional ExternalDNS in each EKS cluster, with permission to create, modify and delete DNS records in different Route 53 zones (such as blue-phoenix.branch.io and green-phoenix.branch.io for each cluster, and phoenix.branch.io for the user-facing service URL)³.
- The Service or Ingress to use ExternalDNS to manage Route 53 DNS records and assign weights to these records through deployment manifest annotations.
- To use a Route 53 weighted routing policy for the service URL.
Components
Before we get into the details, let’s have a brief discussion of the components involved.
Route 53 weighted routing policy
Based on AWS documentation,
Weighted routing lets you associate multiple resources with a single domain name (example.com) or subdomain name (acme.example.com) and choose how much traffic is routed to each resource. This can be useful for a variety of purposes, including load balancing and testing new versions of software.
At Branch, each EKS cluster has its own private hosted Route 53 zone. An additional color-neutral private hosted zone is also created to host the user-facing service URL, i.e. phoenix.branch.io.
ExternalDNS
When ExternalDNS manages a DNS record, it uses a TXT record to tag (track) the record. The TXT record created by ExternalDNS has a specific value, for example, “heritage=external-dns,external-dns/owner=blue-phoenix,external-dns/resource=ingress/default/acme” indicates that the record is managed by ExternalDNS running in the blue-phoenix cluster. The source resource spec is the acme Ingress resource in the default namespace. ExternalDNS won’t modify any record that is not tagged by it.
Any record is hosted in a dedicated Route 53 hosted zone. In order to assign weight to the record, you need to create multiple records for the same record name + record type combination. According to AWS,
You can’t create non-weighted records that have the same values for Record name and Record type as weighted records.
However, in reality, you can have the same record name, record type but with different record ids. For example, it is perfectly fine to have the following records and ask ExternalDNS to create them for us. The total weight for the same record name can be 255 (in reality, the total weight does not need to be 255, it can be 100). The following records are for the same record name, record type but with different record ids. The weight split is 50–50:
record name record type weight routing policy record id alias value
----------- ----------- ------ -------------- --------- ----- -----
acme.phoenix.branch.io A 50 Weighted blue-phoenix Yes internal-blue.us-west-1.elb.amazonaws.com.
acme.phoenix.branch.io A 50 Weighted green-phoenix Yes internal-green.us-west-1.elb.amazonaws.com.The value of the record can be any valid AWS resource for DNS, e.g. Route 53 Alias, CNAME, or CloudFront. We let ExternalDNS manage the service URLs in three hosted zones: one blue, one green, and the color-neutral.
Here’s a summary of ExternalDNS behavior in the context of Route 53:
- ExternalDNS updates relevant records in different Route 53 zones based on hostname annotation in the Kubernetes Service. For example: external-dns.alpha.kubernetes.io/hostname targets the zone blue-phoenix.branch.io or green-phoenix.branch.io.
- The ExternalDNS hostname annotation applies to either K8s Ingress or Service. The same Service backend can have multiple host names, and the hostname does not need to be in the same zone. For example: the service manifest can have both acme.blue-phoenix.branch.io and acme.phoenix.branch.io pointing to the same K8s ReplicaSet. Since it is the same K8s service, the traffic goes through the same ELB and ExternalDNS will create Route 53 A record in different hosted zone with the value of the same ELB.
- More than one ExternalDNS process can manage the records from the same Route 53 zone. These records can have the same record name (thus URL) and type (A record) as long as they have different record ids. For example: the ExternalDNS from blue-phoenix cluster creates an A record for acme.phoenix.branch.io in phoenix.branch.io zone; The ExternalDNS from green-phoenix cluster creates an A record for the same URL in the same hosted zone. ExternalDNS uses domain filter, “ — domain-filter”, to restrict the scope of its operation, and each process in different clusters has its dedicated domain setting for the filter.
DNS TTL
TTL is only applicable if you’re using a Route 53 A record as an ALIAS. ALIAS is not really a DNS record type, but it is a meaningful entity to Route53. In our case, the A record managed by ExternalDNS is an ALIAS to an ELB. As such, the TTL for the DNS name is ELB’s TTL, which is 30 seconds by default.
In reality, we’ve noticed random TTL behavior on both Linux or MacOS:
- If the weight is distributed 50–50 between two upstream ELBs, the DNS resolution can stick to the same ELB for as long as two minutes, or it can switch to a different ELB immediately after three seconds. However, overall, the 50–50 distribution holds proportionally among the two ELBs.
- If the weight is distributed 100–0 between two upstream ELBs, the DNS resolution deterministically points to the IP of the correct ELB.
If you observe that the traffic split between the blue and green cluster does not happen immediately, it might be because of this TTL delay.
Traffic migration
Now we are ready to work on the configuration of ExternalDNS and K8s manifest. The configuration’s impact is depicted in the following diagram, which we will explain in more detail below:
Configure ExternalDNS
Step 1: Change the configuration for the green-phoenix cluster to enable it to manage both phoenix.branch.io (user-facing service URL) and green-phoenix.branch.io Route 53 domains, by adding the domain-filter to the CLI argument,
--domain-filter=green-phoenix.branch.io
--domain-filter=phoenix.branch.ioStep 2: Similarly, change the configuration for the blue-phoenix cluster to enable it to manage both phoenix.branch.io and blue-phoenix.branch.io Route 53 domains, by adding the domain-filter to the CLI argument,
--domain-filter=blue-phoenix.branch.io
--domain-filter=phoenix.branch.ioStep 3: Set txt-owner-id to the said cluster:
--txt-owner-id=blue-phoenixSet it accordingly in the green cluster too.
Step 4: During the migration, if you want DNS change to take effect fast, you can set the interval CLI argument to a smaller number. For example:
--interval=1mThe default is 5 min, but keep in mind that Route 53 calls can get rate limited by AWS. You can also turn on listening to event by adding — events to the argument.
The following is a copy of all the CLI arguments we set to ExternalDNS in blue-phoenix:
containers:
- args:
- --metrics-address=:7979
- --log-level=debug
- --log-format=text
- --events
- --domain-filter=blue-phoenix.branch.io
- --domain-filter=phoenix.branch.io
- --policy=sync
- --provider=aws
- --registry=txt
- --interval=5m
- --txt-owner-id=blue-phoenix
- --source=service
- --source=ingress
- --aws-api-retries=3
- --aws-zone-type=private
- --aws-batch-change-size=1000
- --aws-batch-change-interval=30s
- --aws-zones-cache-duration=24hConfigure acme K8s Service (initial state)
At the start of the blue-green migration, the acme K8s Service in the blue cluster serves 100% of the traffic. Here is an example of the blue cluster configuration:
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
external-dns.alpha.kubernetes.io/hostname: acme.blue-phoenix.branch.io,acme.phoenix.branch.io # (a)
external-dns.alpha.kubernetes.io/aws-weight: '100' # (b)
external-dns.alpha.kubernetes.io/set-identifier: blue-phoenix # (c)
labels:
app: acme
name: acme
namespace: default
spec:
type: LoadBalancer
ports:
- port: 80
protocol: TCP
targetPort: 8080
selector:
app: acme # (d)- Notice that there are two host names that reside in two different Route 53 zones. ExternalDNS listens to the K8s service API change, it then creates or modifies Route 53 records in the Route 53 zone accordingly.
- We start directing all the traffic, full weight of 100, to acme service in the blue cluster. The same acme k8s service in the green cluster is assigned with the weight of 0 as the starting point.
- The identifier gets directly translated into the “Record Id” for the Route 53 record.
- The service backend is the pods for the acme service.
After the successful deployment of the above service manifest, you can verify that the ELB is created:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/acme LoadBalancer 10.100.20.85 internal-blue.us-west-1.elb.amazonaws.com 80:32265/TCP 6mNotice the ELB URL is not the actual URL. It is changed here for the sake of the article.
For the acme K8s Service in the green cluster, we start with no traffic at all. Here is an example of the green cluster configuration:
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
external-dns.alpha.kubernetes.io/hostname: acme.green-phoenix.branch.io,acme.phoenix.branch.io # (a)
external-dns.alpha.kubernetes.io/aws-weight: '0' # (b)
external-dns.alpha.kubernetes.io/set-identifier: green-phoenix # (c)
labels:
app: acme
name: acme
namespace: default
spec:
type: LoadBalancer
ports:
- port: 80
protocol: TCP
targetPort: 8080
selector:
app: acme # (d)- Notice that there are two host names that reside in two different Route 53 zones;
- We start directing no traffic to the green cluster, weight of 0. The weight will increase as the migration moves forward. Eventually, all the traffic will go to the green cluster, while no traffic goes to the blue cluster;
- The identifier gets directly translated into the “Record Id” for the Route 53 record;
- The service backend is the pods for the acme service.
After successful deploy of the above service manifest, you can verify that the ELB is created:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/acme LoadBalancer 10.101.12.13 internal-green.us-west-1.elb.amazonaws.com 80:32265/TCP 4mWe then check the Route 53 data in the three different Route 53 zones. We omit the result from the zones with colors, and only present the relevant records that are created by ExternalDNS in the user-facing zone. They are the ones that matter to the client accessing the service:
Record name Type Routing policy Differentiator Value/Route traffic to Record ID
----------- ---- ------- ------ -------------- ---------------------- ---------
cname-acme.phoenix.branch.io TXT Weighted 100 "heritage=external-dns,external-dns/owner=blue-phoenix,external-dns/resource=service/default/acme" blue-phoenix
cname-acme.phoenix.branch.io TXT Weighted 0 "heritage=external-dns,external-dns/owner=green-phoenix,external-dns/resource=service/default/acme" green-phoenix
acme.phoenix.branch.io A Weighted 100 internal-blue.us-west-1.elb.amazonaws.com. blue-phoenix
acme.phoenix.branch.io A Weighted 0 internal-green.us-west-1.elb.amazonaws.com. green-phoenix
acme.phoenix.branch.io TXT Weighted 100 "heritage=external-dns,external-dns/owner=blue-phoenix,external-dns/resource=service/default/acme" blue-phoenix
acme.phoenix.branch.io TXT Weighted 0 "heritage=external-dns,external-dns/owner=green-phoenix,external-dns/resource=service/default/acme" green-phoenix- The TXT records are tags for ExternalDNS to track the names managed by it.
- The external-dns/owner field has two different values: IDs for the blue and green clusters. These identify the actual ExternalDNS process from either cluster as the owner of the Route 53 record. The external-dns/resource field points to the same service.
- Each of the A record is created by the ExternalDNS process from either the blue or green cluster. The weight of the A record indicates that the acme service traffic all goes to ELB internal-blue.us-west-1.elb.amazonaws.com, which fronts the service pods from the blue cluster.
Configure acme K8s Service (in migration)
Let’s say ExternalDNS in both clusters works properly. The backend pods in the green cluster are ready to serve the traffic.
We start with routing a small portion, i.e. 10%, of https://acme.phoenix.branch.io traffic to the green cluster. If the acme service does not work in the green cluster due to whatever reason caused by the Kubernetes upgrade, we can roll the traffic back to the blue cluster.
In order to make 10% of the traffic to the green cluster, we need to:
- Change service annotation external-dns.alpha.kubernetes.io/aws-weight in the blue cluster to the weight of ‘90’.
- Change service annotation external-dns.alpha.kubernetes.io/aws-weight in the green cluster to the weight of ‘10’.
- Re-deploy the acme service in both clusters.
After 1-2 minutes, based on the ExternalDNS’ “interval” startup setting, the Route 53 records are changed to the following values in the phoenix.branch.io Route 53 zone:
Record name Type Routing policy Differentiator Value/Route traffic to Record ID
----------- ---- ------- ------ -------------- ---------------------- ---------
cname-acme.phoenix.branch.io TXT Weighted 90 "heritage=external-dns,external-dns/owner=blue-phoenix,external-dns/resource=service/default/acme" blue-phoenix
cname-acme.phoenix.branch.io TXT Weighted 10 "heritage=external-dns,external-dns/owner=green-phoenix,external-dns/resource=service/default/acme" green-phoenix
acme.phoenix.branch.io A Weighted 90 internal-blue.us-west-1.elb.amazonaws.com. blue-phoenix
acme.phoenix.branch.io A Weighted 10 internal-green.us-west-1.elb.amazonaws.com. green-phoenix
acme.phoenix.branch.io TXT Weighted 90 "heritage=external-dns,external-dns/owner=blue-phoenix,external-dns/resource=service/default/acme" blue-phoenix
acme.phoenix.branch.io TXT Weighted 10 "heritage=external-dns,external-dns/owner=green-phoenix,external-dns/resource=service/default/acme" green-phoenixFrom now on, the network traffic to https://acme.phoenix.branch.io is split 90–10 between the blue and green cluster. If all works well for acme in the green cluster, which runs the newer version of Kubernetes, we can reduce the percentage of the traffic to the blue cluster to 0, while increasing the percentage of the traffic to the green cluster to 100.
Network traffic configured through a K8s Ingress proxy
ExternalDNS supports network traffic configured through a K8s Ingress proxy, too. A similar approach can be used for the migration to the new cluster. The example annotation is shown below.
For the ingress running in the blue cluster:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
external-dns.alpha.kubernetes.io/aws-weight: '100'
external-dns.alpha.kubernetes.io/set-identifier: blue-phoenix
kubernetes.io/ingress.class: nginx-internal
name: acme
namespace: default
spec:
rules:
- host: acme.blue-phoenix.branch.io
http:
paths:
- backend:
service:
name: acme
port:
number: 80
path: /
pathType: Prefix
- host: acme.phoenix.branch.io
http:
paths:
- backend:
service:
name: acme
port:
number: 80
path: /
pathType: PrefixFor the ingress running in the green cluster:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
external-dns.alpha.kubernetes.io/aws-weight: '0'
external-dns.alpha.kubernetes.io/set-identifier: green-phoenix
kubernetes.io/ingress.class: nginx-internal
name: acme
namespace: default
spec:
rules:
- host: acme.green-phoenix.branch.io
http:
paths:
- backend:
service:
name: acme
port:
number: 80
path: /
pathType: Prefix
- host: acme.phoenix.branch.io
http:
paths:
- backend:
service:
name: acme
port:
number: 80
path: /
pathType: PrefixA word on AWS traffic policy
The approach described here is based on the Route 53 (weighted) record routing policy and ExternalDNS, which provides a means to shape traffic through K8s manifests directly.
If you have used the Route 53 Traffic Policy to migrate traffic, you may have noticed some similarities to what we have described here. Traffic policies provide additional features such as failovers, and they are likely the best strategy for complex routing.
Here are some pros and cons to consider when comparing the Route 53 Traffic Policy to what we described:
Pros
- Allows you to have an additional unique host name to be shared with external customers.
- Clear versioning.
- Nice visual editor.
- Does not need ExternalDNS to make the change; changes are not delayed by an ExternalDNS sync cycle.
Cons
- Extra cost.
- Upstream CNAME, normally an ELB endpoint, requires manual matching. This can be avoided by using ExternalDNS, where you only need to set the hostname annotation and let it match the ELB endpoint behind the scene.
- Can only interact with it through the AWS console. You cannot use Terraform or Helm chart to manage it.
Summary
To summarize, the Branch team has developed a simple way to simplify EKS upgrades without using a service mesh⁴.
We still have work to do, and have discovered a long list of improvements we want to make to the system, but we believe this experiment will help us meaningfully simplify our K8s management and improve reliability for our customers.
If you found what you read interesting, Branch’s engineering organization is hiring! Thanks for reading, and please leave any questions or comments below.
[1] ExternalDNS version 0.12.0 is used.
[2] Many service meshes offer service discovery and L7 traffic management, allowing users to create rules to direct network traffic as needed, such as load balancing, traffic splitting, dynamic failover, and custom resolvers. The approach presented here leverages Route 53, ELB, and ExternalDNS to achieve service discovery, load balancing, and traffic splitting.
[3] For your use case, the separation of the three Route 53 zones is not a requirement. The access to the hosted zone from which the client-facing URL is served is important.
[4] At the time of the writing, there is a blog post on AWS which uses ArgoCD and ExternalDNS to solve the similar problem. It also leverages Route 53 weighted routing.
