DNS Load Balancing & Failover

DNS load balancing distributes and withdraws traffic by changing which addresses an authoritative nameserver hands out, which makes it the cheapest way to move an entire endpoint between regions and the bluntest instrument you own for anything finer than that. This guide covers the routing policies each managed provider exposes, the health-check machinery behind them, the resolver-cache physics that cap how fast a DNS-layer failover can possibly complete, and where the responsibility should instead sit with an edge or anycast layer.

Key implementation principles:

• The authoritative server decides what answer to return, never where a request goes; once a resolver caches the answer, the client is committed for the remainder of the TTL.
• Health checks convert an endpoint’s liveness into record membership, so failover speed is detection interval plus record TTL plus resolver honesty — never a single number.
• Routing policies key on the resolver’s network unless EDNS Client Subnet is in play, so geolocation and latency steering are approximations of user location, not measurements of it.
• DNS failover is an endpoint-granularity tool: use it for whole-region evacuation and pair it with edge load balancing for per-request decisions.

What the DNS layer can and cannot steer

An authoritative nameserver answers a question. That is the whole of its power. When a resolver asks for app.example.com A, the server returns a resource record set (RRset) and a TTL, and the conversation ends. Nothing in the protocol lets the server observe whether the client subsequently connected, whether the connection succeeded, or how much traffic that one answer ultimately carried. Every load-balancing behavior you get at this layer is therefore an inference: hand out address X to some share of queries and hope the resulting traffic split resembles the share you intended.

That inference breaks in predictable ways. A single answer served to a large corporate resolver may fan out to fifty thousand desktops; the same answer served to a home router covers one household. Weighted DNS distributes queries, not sessions, and query volume correlates with session volume only when your client population is large and homogeneous. Below a few thousand distinct resolvers you should expect the observed traffic split to wander several percentage points either side of the configured weights.

The second constraint is caching. Classic round-robin — publishing several A records in one RRset and rotating their order per response — looks like load balancing but delegates the actual choice to the stub resolver and the client library. Some clients try addresses in the order received; glibc’s getaddrinfo applies RFC 6724 sorting and may reorder them; browsers running Happy Eyeballs (RFC 8305) race an IPv4 and an IPv6 candidate and keep whichever handshake completes first. Publishing three addresses guarantees only that clients can reach three endpoints, not that a third of them will. Read Propagation & Caching Basics for the cache-hierarchy detail that governs how long any of these answers survives in the wild.

The third constraint is that removing an address from an RRset does not remove it from a resolver’s memory. This is the entire reason DNS failover is a coarse tool. You can flip the authoritative answer in under a second; you cannot reach into a resolver in a different autonomous system and invalidate what it already stored. Failover completion time is therefore bounded below by the record TTL, and in practice by the largest effective TTL any resolver in your traffic mix chose to enforce.

Finally, connection reuse means DNS has no influence at all over traffic that is already flowing. An HTTP/2 client holding an open connection to the failing origin keeps using it until the socket errors or the connection idle timeout fires, regardless of what the zone now says. DNS-layer failover only redirects new name resolutions, which is why a well-instrumented failover drill measures error rate, not just answer content.

Routing policies and what each one keys on

Managed DNS providers expose a small, remarkably consistent family of policies. The names differ; the mechanics do not.

Simple / round robin returns every record in the set, leaving the choice to the client. It is not load balancing so much as redundancy advertisement, and it has no concept of health — a dead address stays in the answer until you remove it.

Weighted attaches an integer weight to each record set behind a shared name. The authoritative server selects one set per query with probability proportional to its weight. Weight 0 withdraws a record without deleting it, which is the standard mechanism for staged canary rollouts and for draining a region before maintenance.

Latency-based (Route 53) or dynamic latency steering (Cloudflare) resolves against a continuously updated map of round-trip times between the querying network and each candidate region. The key subtlety: the measurement is to the resolver, not the user. A user on a mobile network in Lisbon whose handset is configured to use a public resolver anycast into Madrid gets the Madrid answer, which is usually fine, and occasionally is not.

Geolocation and geoproximity map the querying network to a continent, country, or subdivision and answer accordingly. Geolocation is a lookup table; geoproximity is a distance calculation with a bias knob that lets you expand or shrink a region’s catchment. Both need an explicit default record for queries that match nothing, and both are only as good as the IP-to-location database behind them.

Multivalue answer returns up to eight healthy records, each independently health-checked, in one response. It is round robin with liveness — the best available approximation of “give the client several working options and let it choose.”

Failover is a strict either/or: the PRIMARY record set is served while its health check passes, and the SECONDARY otherwise. This is the policy people mean when they say “DNS failover,” and it is the subject of the dedicated walkthrough on configuring Route 53 health checks with DNS failover.

EDNS Client Subnet (RFC 7871) partially repairs the resolver-versus-user gap by forwarding a truncated client prefix — typically a /24 for IPv4 — alongside the query. Google Public DNS and OpenDNS send it; Cloudflare’s 1.1.1.1 deliberately does not, on privacy grounds. That single fact means a measurable slice of your users will always be geo-routed by resolver location rather than their own.

Provider implementations

AWS Route 53

Route 53 splits the problem into two objects: a health check (an independent resource with its own ID, billed per check) and a record set that references it. A record with a failing health check is simply omitted from the candidate pool, and if every record in a set is unhealthy Route 53 falls back to answering with all of them — the “all unhealthy means all healthy” rule that surprises people during a total outage.

The Terraform below builds a weighted pair with health checks and a matching failover pair. Note failure_threshold and request_interval: three consecutive failures at a 30-second interval is the default, giving roughly 90 seconds of detection latency before the record is withdrawn.

resource "aws_route53_health_check" "us_east" {
  fqdn              = "us-east.origin.example.com"
  port              = 443
  type              = "HTTPS_STR_MATCH"
  resource_path     = "/healthz"
  search_string     = "\"status\":\"ok\""
  request_interval  = 30
  failure_threshold = 3
  measure_latency   = true
  regions           = ["us-east-1", "eu-west-1", "ap-southeast-1"]

  tags = {
    Name = "us-east-healthz"
  }
}

resource "aws_route53_record" "app_us" {
  zone_id         = aws_route53_zone.main.zone_id
  name            = "app.example.com"
  type            = "A"
  ttl             = 60
  records         = ["203.0.113.10"]
  set_identifier  = "us-east"
  health_check_id = aws_route53_health_check.us_east.id

  weighted_routing_policy {
    weight = 80
  }
}

resource "aws_route53_record" "app_eu" {
  zone_id         = aws_route53_zone.main.zone_id
  name            = "app.example.com"
  type            = "A"
  ttl             = 60
  records         = ["198.51.100.10"]
  set_identifier  = "eu-west"
  health_check_id = aws_route53_health_check.eu_west.id

  weighted_routing_policy {
    weight = 20
  }
}

The same pair expressed as a raw change batch, which is what you reach for during an incident when Terraform state is not something you want to touch:

{
  "Comment": "app.example.com PRIMARY/SECONDARY failover pair",
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "app.example.com.",
        "Type": "A",
        "SetIdentifier": "primary-us-east",
        "Failover": "PRIMARY",
        "TTL": 60,
        "ResourceRecords": [{ "Value": "203.0.113.10" }],
        "HealthCheckId": "b1f4c2de-1f0e-4b6a-9d13-2b7d9e5a1c34"
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "app.example.com.",
        "Type": "A",
        "SetIdentifier": "secondary-eu-west",
        "Failover": "SECONDARY",
        "TTL": 60,
        "ResourceRecords": [{ "Value": "198.51.100.10" }]
      }
    }
  ]
}

aws route53 change-resource-record-sets \
  --hosted-zone-id Z2FDTNDATAQYW2 \
  --change-batch file://failover.json

Alias records to an ELB, CloudFront distribution, or S3 website endpoint can set EvaluateTargetHealth: true instead of carrying their own health check ID, which delegates liveness to the target’s own health signal. That interacts directly with apex naming, so review CNAME flattening and ALIAS behavior before you put a failover pair on the zone apex.

Cloudflare Load Balancing

Cloudflare models the same problem as monitors, pools, and load balancers. A monitor is a reusable probe definition; a pool is a named set of origins that share a monitor and a minimum_origins floor; a load balancer binds pools to a hostname with a steering policy. When proxied = true, the DNS answer is a Cloudflare anycast address and the actual origin selection moves to the edge on every request — which is why Cloudflare’s product blurs the DNS/edge boundary discussed later on this page.

resource "cloudflare_load_balancer_monitor" "https_healthz" {
  account_id       = var.account_id
  type             = "https"
  method           = "GET"
  path             = "/healthz"
  expected_codes   = "200"
  expected_body    = "ok"
  interval         = 60
  retries          = 2
  timeout          = 5
  follow_redirects = false

  header {
    header = "Host"
    values = ["app.example.com"]
  }
}

resource "cloudflare_load_balancer_pool" "us_east" {
  account_id      = var.account_id
  name            = "us-east"
  monitor         = cloudflare_load_balancer_monitor.https_healthz.id
  minimum_origins = 1
  check_regions   = ["ENAM", "WNAM"]

  origins {
    name    = "us-east-1"
    address = "203.0.113.10"
    enabled = true
    weight  = 1
  }
}

resource "cloudflare_load_balancer" "app" {
  zone_id          = var.zone_id
  name             = "app.example.com"
  default_pool_ids = [cloudflare_load_balancer_pool.us_east.id]
  fallback_pool_id = cloudflare_load_balancer_pool.eu_west.id
  proxied          = true
  steering_policy  = "dynamic_latency"
  session_affinity = "cookie"
  ttl              = 30
}

Setting proxied = false turns the same configuration into pure DNS steering: Cloudflare answers with the origin IP directly and the ttl field becomes meaningful, because the client now talks to your origin rather than to Cloudflare’s edge.

Azure Traffic Manager and GCP Cloud DNS

Azure Traffic Manager is a DNS-only product — it never sits in the data path. A profile owns the probe configuration and the routing method (Priority, Weighted, Performance, Geographic, MultiValue, Subnet), and endpoints are attached to it. Because it answers with CNAMEs pointing at *.trafficmanager.net, the profile TTL is the number that governs failover speed.

az network traffic-manager profile create \
  --name app-tm --resource-group prod \
  --routing-method Priority --unique-dns-name app-prod-example \
  --ttl 30 --protocol HTTPS --port 443 --path /healthz \
  --interval 10 --timeout 5 --max-failures 3

az network traffic-manager endpoint create \
  --name us-east --profile-name app-tm --resource-group prod \
  --type externalEndpoints --target 203.0.113.10 \
  --priority 1 --endpoint-status Enabled

az network traffic-manager endpoint create \
  --name eu-west --profile-name app-tm --resource-group prod \
  --type externalEndpoints --target 198.51.100.10 \
  --priority 2 --endpoint-status Enabled

GCP Cloud DNS attaches routing policies directly to a record set. Weighted round robin and geo policies work against any address; health-checked failover is only available when the targets are Google Cloud internal load balancers, because that is the only liveness signal Cloud DNS can consume.

gcloud dns record-sets create app.example.com. \
  --zone=prod-zone --type=A --ttl=30 \
  --routing-policy-type=WRR \
  --routing-policy-data="80.0=203.0.113.10;20.0=198.51.100.10"

gcloud dns record-sets create geo.example.com. \
  --zone=prod-zone --type=A --ttl=30 \
  --routing-policy-type=GEO \
  --routing-policy-data="us-east1=203.0.113.10;europe-west1=198.51.100.10"

Self-hosted: BIND and PowerDNS

BIND has no health-checking capability whatsoever. It serves the zone it was given, so liveness has to come from outside — a supervisor that probes endpoints and rewrites the record via dynamic update. Keep the update TSIG-signed and keep it idempotent, because a flapping probe that rewrites the zone every ten seconds will bump the SOA serial into a notify storm across your secondaries.

nsupdate -k /etc/bind/keys/failover.key <<'EOF'
server ns1.example.com
zone example.com.
update delete app.example.com. A
update add app.example.com. 30 A 198.51.100.10
send
EOF

PowerDNS solves it in-band with Lua records, which evaluate at query time. ifportup returns only the addresses whose TCP port is currently answering, and pickclosest adds latitude/longitude proximity selection — both driven by a background health-check thread inside the authoritative server itself.

app.example.com.  30  IN  LUA  A  "ifportup(443, {'203.0.113.10', '198.51.100.10'})"
geo.example.com.  30  IN  LUA  A  "pickclosest({'203.0.113.10', '198.51.100.10'})"

Enable it with enable-lua-records=yes in pdns.conf. The trade-off is that your authoritative servers now perform outbound health checks from wherever they happen to be hosted, so their view of “up” is narrower than a managed provider probing from a dozen regions.

Standing up a health-checked failover pair

1. Pick the health endpoint before anything else. It must exercise the dependencies that matter — database reachability, cache connectivity, the ability to sign a token — and it must return quickly. A /healthz that only proves nginx is listening will keep a broken region in rotation indefinitely.

2. Lower the TTL and wait out the old one. Drop the record to 60 seconds at least 24 hours before you rely on failover, so every resolver holding the previous long TTL has expired it. The mechanics are worked through in Mastering TTL Strategies.

3. Create the health checks and confirm they are green before attaching them to records. A record referencing a health check that has never passed will be treated as unhealthy from the moment it is created.

   aws route53 get-health-check-status --health-check-id b1f4c2de-1f0e-4b6a-9d13-2b7d9e5a1c34 \
     --query 'HealthCheckObservations[].{Region:Region,Status:StatusReport.Status}' --output table

4. Apply the record pair with terraform apply or the change batch above, then poll the change until it reaches INSYNC across the provider’s fleet.

   aws route53 wait resource-record-sets-changed --id /change/C2682N5HXP0BZ4

5. Verify the authoritative answer directly, bypassing every cache, by querying the zone’s own nameservers.

   for ns in $(dig +short NS example.com); do
     printf '%-32s %s\n' "$ns" "$(dig +short @"$ns" app.example.com A | paste -sd, -)"
   done

6. Verify what the public internet sees across several independent resolvers, checking the remaining TTL rather than just the address.

   for r in 1.1.1.1 8.8.8.8 9.9.9.9 208.67.222.222; do
     dig +noall +answer @"$r" app.example.com A
   done

7. Force a failure and time it. Break the health endpoint deliberately (return 503, or block the checker source ranges) and record the wall-clock gap between the first failed probe and the first changed answer.

   while :; do
     printf '%s %s\n' "$(date -u +%H:%M:%S)" "$(dig +short @8.8.8.8 app.example.com A | paste -sd, -)"
     sleep 5
   done

8. Restore and confirm the failback, which is the half everyone forgets to rehearse. Failback is usually slower than failover because health checks typically require several consecutive successes before a record is readmitted.

TTL, caching, and how long a failover really takes

Total failover time is the sum of three intervals, and only the first is under tight control.

Detection is request_interval × failure_threshold plus a variable settling period while checkers in different regions reach consensus. At Route 53 defaults that is roughly 90 seconds; at Azure Traffic Manager’s fastest setting, roughly 30. Tightening these numbers trades detection speed for flap sensitivity — a 10-second interval with a threshold of 1 will withdraw a region because of one dropped packet.

Publication is near-instantaneous on managed providers, since the policy engine evaluates health at query time rather than rewriting a zone. On a self-hosted stack it is dynamic update plus IXFR to every secondary, which is seconds if your notify path is healthy and minutes if it is not.

Cache drain is the record TTL — in theory. In practice resolvers apply their own floors and ceilings. Many enterprise forwarders clamp very low TTLs upward to protect themselves from query load, and a nontrivial share of consumer CPE simply ignores TTL and caches until reboot. Setting a 5-second TTL does not produce 5-second failover; it produces higher query bills and a distribution of client migration times whose slowest stragglers are unchanged. Sixty seconds is the pragmatic floor for most production zones, and 30 is defensible when the failover budget genuinely justifies double the query volume.

Negative caching deserves a separate mention. If your failover configuration ever produces a NODATA or NXDOMAIN answer — a geolocation policy with no default record is the classic route to this — that negative answer is cached according to the SOA minimum field, and clients will keep failing for its full duration even after you fix the zone. Keep the SOA minimum modest (300 seconds is a common choice) precisely so that this class of mistake is recoverable in minutes.

Composing DNS steering with the edge layer

The two layers are not alternatives; they operate at different granularities and different speeds. DNS chooses an entry point once per cache lifetime. An anycast network chooses a point of presence per packet, invisibly, via BGP. An edge load balancer chooses an origin per request, with full visibility of the HTTP method, path, headers, and the outcome of the previous attempt.

The practical division of labor: use DNS to select which anycast platform or which cloud provider serves a hostname, and to evacuate an entire region when that platform is unreachable. Use the edge to pick among origins, apply weights for canaries, enforce stickiness, and retry a failed request against a second origin before the user ever sees an error. Once traffic is proxied through an anycast edge, DNS becomes your provider-outage escape hatch rather than your day-to-day traffic tool — the trade-off is compared in detail in DNS Failover versus Edge Load Balancing, and the edge-side mechanics are covered in Configuring Edge Health Checks and Automatic Failover.

Troubleshooting and rollback

The rollback protocol is deliberately simple, because rollback happens under pressure:

1. Set the failing record’s weight to 0, or disassociate its health check — do not delete the record. Deleting loses the configuration you will need in ten minutes.
2. Confirm the authoritative answer changed with a direct dig @ns against every nameserver in the NS set, not just one.
3. Announce a wait equal to the TTL plus a margin before declaring the rollback complete, and keep the old origin serving traffic — degraded is better than refused — for that entire window.
4. Only after the drain window closes should you scale down or terminate the withdrawn endpoint.

Edge cases and gotchas

• The “all unhealthy” fallback. Route 53 answers with every record when none are healthy, on the theory that a possibly-broken answer beats no answer. During a genuine total outage this looks identical to health checks not working at all.
• Health checks that test the wrong thing. An endpoint returning 200 OK from a static file while the application layer is throwing 500s will never trigger failover. String matching against a body that a real dependency produces is the cheap fix.
• Alias records cannot carry a TTL. Route 53 alias records inherit the TTL of the target, so your carefully chosen 60 seconds may not be what resolvers actually receive.
• DNSSEC and dynamic answers. Providers that compute answers per query sign on the fly. If you operate DNSSEC yourself, a supervisor rewriting records must re-sign the zone, and a stale RRSIG is a hard resolution failure rather than a degraded answer.
• Health checks cost real money at scale. Route 53 bills per health check per month, with a premium for HTTPS string matching and for latency measurement. Hundreds of per-endpoint checks add up faster than teams expect.
• Query-time policies do not appear in a zone file export. Weighted, latency, and failover record sets are provider metadata. A zone export for a migration will not carry them, which turns a “simple” provider move into a re-implementation — see DNS Zone Management.
• Flapping is worse than a clean failure. A health check oscillating on a 90-second cycle produces a zone whose answer changes faster than caches drain, so different users get different answers indefinitely and your error rate stays elevated in both regions.

Frequently Asked Questions

How low should I set the TTL for DNS failover? Sixty seconds is the practical floor for most production zones, and 30 seconds is defensible when the failover budget justifies roughly double the query volume. Going lower buys almost nothing, because many resolvers clamp very short TTLs upward and some client-side caches ignore TTL entirely, so the slow end of your client migration curve barely moves.

Does DNS failover move traffic that is already connected? No. DNS only affects new name resolutions, so a client holding an open HTTP/2 or WebSocket connection to the failing endpoint stays there until the socket errors or an idle timeout fires. Shed those connections at the origin — close them, or stop accepting — if you need them to move within the failover window.

Why does my traffic split not match the weights I configured? Weighted DNS distributes queries, not sessions, and one answer served to a large corporate or public resolver can fan out to tens of thousands of users. The split converges on the configured weights only when you have many distinct resolvers behind your traffic; below that, several points of drift is normal and not a bug.

Do I still need DNS failover if my traffic is proxied through an anycast edge? Yes, but for a different job. The edge handles origin failure per request, so DNS failover stops being your day-to-day tool and becomes the escape hatch for the edge provider itself being unreachable — a second provider’s hostname held ready behind a low TTL that you can activate manually.

Related

• Configuring Route 53 Health Checks with DNS Failover
• DNS Failover versus Edge Load Balancing
• Mastering TTL Strategies
• Load Balancing at the Edge
• Propagation & Caching Basics

Back to DNS Fundamentals & Advanced Record Configuration