Edge & DNS Ops Guide

DNS Failover versus Edge Load Balancing

After working through this guide you will be able to decide, for a specific failure mode, whether the recovery belongs at the DNS layer or at the edge — and you will be able to prove the choice with measured failover times rather than vendor claims, using a repeatable dig and curl procedure that produces a number you can put in a runbook.

The two mechanisms get compared as if they were competing products. They are not. DNS failover changes what answer a resolver receives, once per cache lifetime, at endpoint granularity. Edge load balancing changes which origin serves this request, on every request, with the previous attempt’s outcome available as input. They fail differently, cost differently, and protect against different things — and the reason nearly every mature setup runs both is that neither covers the other’s blast radius. This guide measures both against the same induced outage and then lays out the hybrid arrangement that falls out of the numbers.

Key implementation objectives:

  • Measure real end-to-end failover time for both layers with a client-side loop, not with provider dashboards.
  • Map each failure mode — single origin, whole region, whole edge provider — to the layer that can actually recover it.
  • Understand where stickiness, cost, and observability differ, and why those usually decide the architecture before latency does.
  • Assemble the hybrid pattern: edge steering for origins, DNS held in reserve for the edge itself.

Prerequisites and environment setup

You need dig (bind-utils / dnsutils 9.16+), curl 7.75+ for the --retry-all-errors and write-out fields used below, and administrative access to both a managed DNS zone with health checks and an edge load balancer. A two-origin deployment is enough: two addresses in different regions, each serving a /healthz endpoint you can break on demand.

dig -v                # DiG 9.16 or later
curl --version | head -1

Set the DNS side up first, because it has the longest lead time. The record must have been serving a 60-second TTL for at least a day before you measure anything, or you will be timing the drain of an old, longer TTL instead of the one you configured — the setup is covered step by step in Configuring Route 53 Health Checks with DNS Failover. On the edge side, configure a pool with both origins and a health monitor, as described in Configuring Edge Health Checks and Automatic Failover.

The same failure, seen at three moments A two by three status matrix. Under DNS failover the client is still failing at five and sixty seconds and only recovers by one hundred and eighty seconds; under edge load balancing the client has already recovered at five seconds and is steady thereafter. t = 5 s t = 60 s t = 180 s DNS failover TTL 60 s, 3 x 30 s FAILING check not yet tripped FAILING 2 of 3 probes failed RECOVERED caches drained Edge steering per-request retry RECOVERED retried on peer origin STEADY pool rebalanced STEADY origin drained One origin failure, sampled at three moments after it began. The edge absorbs it inside a single request; DNS needs detection plus a full cache drain.

Step-by-step measurement procedure

Step 1 — Instrument the client, not the dashboard

Provider consoles report when they changed state, which is a fraction of what a user experiences. Measure from a client with a one-second sampling loop that records the HTTP status and where the response came from. Run it from a host whose resolver you control and understand — a public cloud VM using the platform resolver is representative; your laptop behind a corporate forwarder is not.

run_probe() {
  local host="$1" out="$2" secs="${3:-600}"
  local end=$(( SECONDS + secs ))
  : > "$out"
  while [ "$SECONDS" -lt "$end" ]; do
    printf '%s %s\n' "$(date -u +%H:%M:%S)" \
      "$(curl -s -o /dev/null --max-time 5 -w '%{http_code} %{time_total}' "https://$host/healthz")" \
      >> "$out"
    sleep 1
  done
}

The --max-time 5 matters: without it a hung origin produces one 130-second sample and hides the outage shape entirely.

Step 2 — Measure DNS-layer failover

Start the probe against the DNS-steered hostname, then break the primary origin’s health endpoint. In parallel, sample the answer every five seconds so you can separate when the record changed from when the client stopped erroring.

run_probe app-dns.example.com dns-failover.log 600 &
while :; do
  printf '%s %s\n' "$(date -u +%H:%M:%S)" "$(dig +short @8.8.8.8 app-dns.example.com A | paste -sd, -)"
  sleep 5
done | tee dns-answers.log

After the run, extract the error window — the number that actually belongs in your runbook:

awk '$2 != "200" { if (!s) s = $1; e = $1; n++ } END { print "errors:", n, "from", s, "to", e }' \
  dns-failover.log

Expected shape: an unbroken run of non-200 samples starting the moment you broke the origin and continuing for detection time plus cache drain. With a 30-second interval, a threshold of 3, and a 60-second TTL, a typical result is 120 to 170 seconds of continuous errors — and crucially, a tail where a few later samples still fail because a forwarder held the old answer longer.

Step 3 — Measure edge-layer failover

Repeat against the hostname proxied through the edge load balancer, breaking the same origin the same way. Add a header that reveals which origin served the request so you can see the switch rather than infer it.

run_probe app-edge.example.com edge-failover.log 300 &
curl -sI https://app-edge.example.com/healthz | grep -iE 'x-served-by|cf-ray|x-origin-region'

Expected shape: either zero failed samples — because the edge retried the request against a healthy origin inside the same client request — or a single failed sample, if the failure mode is one the proxy cannot retry safely (a non-idempotent POST, or a response whose headers were already flushed). The DNS answer does not change at all during this test, which is the whole point: the name resolves to an anycast address that never moves.

Step 4 — Compare on every axis, not just latency

Latency is the axis that gets quoted and the least likely to decide the design. Put the measured numbers next to the properties that constrain your architecture.

Dimension DNS failover Edge load balancing
Measured recovery time 120-170 s typical (detection + TTL drain), plus stragglers 0-1 failed request; sub-second for retryable methods
Decision granularity Whole endpoint, once per cache lifetime Per request, with method, path, and headers in scope
Control loop input Out-of-band probe result only Live request outcomes plus probe results
Stickiness None — a re-resolve can land anywhere Cookie or IP affinity, honored per request
Traffic split accuracy Approximate; weights distribute queries, not sessions Exact; every request is counted and steered
Cost model Per health check per month, plus query volume Per request, plus proxied bandwidth
Observability Answer content and query counts; no request outcomes Full request logs, per-origin error rates, retry counts
Protects against Region loss, provider loss, edge platform loss Origin loss, origin degradation, capacity imbalance
Fails when The failure is faster than the TTL, or resolvers ignore it The edge platform itself is unreachable
Config change speed Instant at the authoritative server, minutes at the client Seconds, globally, no cache to drain

Two rows deserve emphasis. Stickiness is where DNS is not merely slower but structurally incapable: a resolver re-query mid-session can hand a client a different address, and nothing at the DNS layer can prevent it. If your application needs a user pinned to a region — for write affinity, for a local session store, for anything stateful — that pinning has to live at the edge, as covered in Session Affinity and Sticky Routing at the Edge.

Observability is the quieter differentiator. DNS tells you what answer you gave; it cannot tell you whether the resulting connection succeeded. Edge logs give you per-origin error rates, retry counts, and latency percentiles, so a partially degraded origin — the failure mode a binary health check misses — shows up as a rising 5xx rate long before any probe threshold trips.

Which layer should recover this failure A decision tree: if the blast radius is a single origin with healthy peers still available, steer at the edge; if a whole region or provider is unreachable, fail over at the DNS layer. Where is the blast radius? one origin, or the whole region? narrow blast radius wide blast radius One origin or instance healthy peers still available Whole region or provider no healthy peer on the platform Steer at the edge sub-second, per request Fail over at the DNS layer minutes, per endpoint Pool weights, request retries, session affinity, instant drain Health-checked PRIMARY set, SECONDARY behind a 60 s TTL

The hybrid pattern most teams actually run

Once the numbers are on the table the architecture writes itself. Traffic resolves to an anycast edge, the edge picks among origins on every request, and DNS is held in reserve for the one failure the edge cannot recover from — the edge platform itself. That last case is rare and severe, which is exactly the profile a slow, coarse, cheap mechanism suits.

Hybrid: edge for origins, DNS for the edge A client resolves one hostname; DNS holds a primary answer pointing at the active edge platform and a secondary pointing at a standby platform, and each edge platform independently steers requests across origins. Client one hostname app.example.com PRIMARY: edge A anycast SECONDARY: edge B, TTL 60 s steady state provider outage only Edge platform A (active) per-request origin choice + retries origins: us-east, eu-west, ap-south Edge platform B (standby) same origins, config kept in sync activated by a DNS change, not a probe DNS answers once per TTL; the edge decides on every single request.

Three details make the pattern work in practice. First, the secondary edge platform must be kept warm and configured — a standby whose routing rules were last synced six months ago is not a standby. Second, the DNS record fronting it stays at a low TTL permanently, so the escape hatch is always ready; the cost of that is query volume, which is trivially cheap compared to the outage it covers. Third, resist the temptation to automate the platform-level failover with a health check. Automatic provider-to-provider cutover is exactly the mechanism most likely to trigger spuriously and cause the incident it was meant to prevent — a human decision with a documented command is the right trade for a failure this rare.

Where the two layers genuinely overlap is traffic shaping: weighted rollouts, regional canaries, and capacity balancing. Do those at the edge, where a weight change takes effect in seconds and every request is counted, rather than at DNS where weights distribute queries and the split drifts. Weighted Load Balancing Across Multi-Region Origins covers that side; the DNS-side policies and their limits are catalogued in the parent guide, DNS Load Balancing & Failover.

Verification

A good measurement run produces three artifacts. Confirm you have all three before drawing conclusions.

awk '$2 != "200" { n++ } END { printf "dns  : %d failed samples\n", n }' dns-failover.log
awk '$2 != "200" { n++ } END { printf "edge : %d failed samples\n", n }' edge-failover.log
dns  : 148 failed samples
edge : 1 failed samples

Second, confirm the DNS answer actually changed during the DNS run and did not change during the edge run — if the edge hostname’s answer moved, you accidentally measured DNS twice:

sort -u -k2 dns-answers.log | head
dig +short app-edge.example.com A

Third, confirm the tail. Filter the DNS log for failures occurring more than one TTL after the answer changed; those samples are resolvers that ignored or clamped your TTL, and their existence is the argument against treating DNS as a fast failover mechanism.

awk '$2 != "200" && $1 > "10:04:30" { print }' dns-failover.log | wc -l

Troubleshooting

The DNS run shows zero errors

Your probe host cached nothing because it re-resolved on every request, or its resolver honored a much shorter TTL than the public average. Pin the measurement to a specific resolver and confirm the client is actually caching:

dig +noall +answer @8.8.8.8 app-dns.example.com A
dig +noall +answer @8.8.8.8 app-dns.example.com A   # TTL should have decremented

If the second query returns the full TTL again, you are talking to a different anycast instance of the same resolver on each query, and your measurement is not representative of a single user’s experience.

The edge run shows a long error window

That means the edge did not retry — either the health monitor had not marked the origin down and no request-level retry policy was configured, or the failed method was non-idempotent. Check the monitor’s state and the retry configuration; a pool with minimum_origins set to the total origin count will refuse to serve at all rather than degrade to the remaining origin.

Both layers recover but users still report errors

The recovery you measured applies to new connections. Clients holding an established HTTP/2 connection to the broken origin stay there until the socket errors or an idle timeout fires, and neither layer can reach into that. Close the connections at the origin — stop accepting, or send GOAWAY — and re-run the measurement to see the difference.

The measured numbers vary wildly between runs

Health check thresholds are evaluated per checker region and aggregated, so the exact moment of the flip depends on which probe cycle your outage landed in. Run each measurement at least three times and quote the worst case in your runbook, not the median. If the spread exceeds the probe interval, something upstream — a load balancer draining, a CDN caching the health endpoint — is adding its own delay.

Frequently Asked Questions

Which layer should I build first? Build the edge layer first. It covers the failure modes that occur weekly — one origin degrading, one region losing capacity — and it does so without a cache drain. DNS failover covers a much rarer, much larger failure, and it depends on a low TTL that takes a day to establish, so it is naturally the second phase.

Can DNS failover ever be as fast as edge steering? No, and shortening the TTL does not change that. The floor is detection time plus cache drain, and a meaningful share of resolvers and client-side caches will not honor a very short TTL anyway. The fastest realistic DNS failover lands around 30 to 60 seconds; the edge answers in the same request.

Does using an edge load balancer mean I can ignore TTLs? The opposite. Because the anycast address rarely changes, teams let the TTL drift upward to hours — and then discover during a provider outage that their escape hatch takes half a day to take effect. Keep the record fronting the edge at 60 seconds precisely because you almost never change it.

How do I test edge failover without breaking production? Use a dedicated hostname bound to the same pool, or mark a single origin as disabled in the pool configuration rather than breaking the origin itself. Both exercise the steering logic; only the second also exercises the health monitor, so run each at least once.

Is anycast BGP reconvergence a third layer I should measure? It is a real layer but not one you control. When a point of presence withdraws its route, clients reconverge on a neighbor in seconds without any DNS or configuration change. Treat it as a property of your edge provider, note it in the runbook, and measure it only if you operate the anycast network yourself.

Back to DNS Load Balancing & Failover