Edge & DNS Ops Guide

Configuring Route 53 Health Checks with DNS Failover

After working through this guide you will be able to build an HTTPS health check that inspects the response body, bind it to a PRIMARY/SECONDARY record pair, alarm on its state through CloudWatch, rehearse a forced failover with a stopwatch, and roll the whole thing back without deleting configuration you will need again in ten minutes.

Route 53 keeps health and records as separate resources on purpose. A health check is an independent object with its own ID and its own bill; a record set merely references one. That separation is what lets a single check drive several record sets, and it is also the source of the most common misconfiguration — a perfectly green health check that no record actually consults. This guide wires the two together end to end against a two-region deployment: us-east.origin.example.com as primary and eu-west.origin.example.com as standby, both fronted by app.example.com.

Key implementation objectives:

  • Create an HTTPS_STR_MATCH health check that fails when the application is broken, not merely when the web server is listening.
  • Attach it to a PRIMARY record set and pair that with a SECONDARY set carrying no health check.
  • Alarm on HealthCheckStatus in us-east-1 so a failover is announced rather than discovered.
  • Rehearse the failover, measure it against the TTL, and rehearse the failback too.

Prerequisites and environment setup

You need AWS CLI v2.15 or newer, Terraform 1.6+ if you follow the declarative path, and an IAM principal holding route53:CreateHealthCheck, route53:ChangeResourceRecordSets, route53:GetHealthCheckStatus, and cloudwatch:PutMetricAlarm. Confirm your toolchain and locate the hosted zone before touching anything:

aws --version                       # aws-cli/2.15.x or later
terraform version                   # v1.6.0 or later
aws route53 list-hosted-zones-by-name \
  --dns-name example.com --query 'HostedZones[0].{Id:Id,Name:Name}' --output table

Two preparation items matter more than the tooling. First, both endpoints must be reachable by name or IP from the public internet — Route 53’s checkers run outside your VPC and cannot reach a private endpoint. Second, drop the TTL on app.example.com to 60 seconds at least a day before you depend on failover, so that every resolver holding the previous value has expired it. The reasoning is worked through in Mastering TTL Strategies; skipping it means your first real incident is also your first TTL drain.

From probe to answer Route 53 checkers probe the origin endpoint every thirty seconds; the response is matched against a search string to set the health check status, which is attached to the PRIMARY record set and decides whether a resolver query is answered with the primary or the secondary address. From probe to answer Route 53 checkers 15+ global locations HTTPS GET /healthz Origin endpoint origin.example.com {"status":"ok"} Health check HTTPS_STR_MATCH status: HEALTHY probe every 30 s 2xx + string match attached to the PRIMARY set Resolver query app.example.com A PRIMARY set 203.0.113.10 served while healthy SECONDARY set 198.51.100.10 served only when PRIMARY fails A failing check removes the PRIMARY set from the candidate pool. Resolvers holding the old answer keep it until their TTL expires.

Step-by-step procedure

Step 1 — Make the health endpoint tell the truth

The single most common reason failover never fires is a health endpoint that returns 200 OK from a static file while the application behind it is dead. Serve a body that can only be produced by exercising the dependencies you actually care about, and keep the handler fast enough to answer well inside the checker’s four-second connection budget.

app.get("/healthz", async (req, res) => {
  try {
    await db.query("SELECT 1");          // proves the primary datastore is reachable
    await cache.ping();                  // proves the session store is reachable
    res.status(200).json({ status: "ok", region: process.env.REGION });
  } catch (err) {
    res.status(503).json({ status: "degraded", error: err.code });
  }
});

Expected side effect: a dependency failure now flips the body from {"status":"ok",...} to {"status":"degraded",...}, which is what the string match will key on. Exclude /healthz from any authentication middleware, rate limiter, or WAF rule — a 403 counts as a failure exactly like a 503, and you will spend an afternoon discovering it.

Step 2 — Create the HTTPS health check with string matching

HTTPS_STR_MATCH combines two conditions: the response status must be 2xx or 3xx and the search string must appear in the first 5120 bytes of the body. That second condition is what distinguishes an application-aware check from a port scan.

aws route53 create-health-check \
  --caller-reference "us-east-healthz-$(date +%s)" \
  --health-check-config '{
    "Type": "HTTPS_STR_MATCH",
    "FullyQualifiedDomainName": "us-east.origin.example.com",
    "Port": 443,
    "ResourcePath": "/healthz",
    "SearchString": "\"status\":\"ok\"",
    "RequestInterval": 30,
    "FailureThreshold": 3,
    "MeasureLatency": true,
    "EnableSNI": true,
    "Regions": ["us-east-1", "us-west-2", "eu-west-1", "ap-southeast-1", "ap-northeast-1"]
  }'

The command prints the new check’s Id — capture it, because every later step needs it. The same resource in Terraform:

resource "aws_route53_health_check" "primary" {
  fqdn              = "us-east.origin.example.com"
  port              = 443
  type              = "HTTPS_STR_MATCH"
  resource_path     = "/healthz"
  search_string     = "\"status\":\"ok\""
  request_interval  = 30
  failure_threshold = 3
  enable_sni        = true
  measure_latency   = true
  regions           = ["us-east-1", "us-west-2", "eu-west-1", "ap-southeast-1", "ap-northeast-1"]

  tags = {
    Name = "app-primary-healthz"
  }
}

RequestInterval and FailureThreshold are the only two knobs that control detection latency, and they multiply. Thirty seconds times three failures is roughly 90 seconds before the status flips; a 10-second interval with a threshold of 3 cuts that to about 30 seconds but costs more per check and makes the check far more sensitive to transient packet loss. Each checker region votes independently, and the aggregate status flips when more than 18% of checkers report failure.

Health check state transitions A health check sits in the healthy state until a probe fails, moves through an intermediate failing state while the failure count is below the threshold, and reaches unhealthy on the third consecutive failure; recovery requires the full consecutive success count. Failure threshold 3, interval 30 s HEALTHY record set in the pool FAILING 1-2 still answered UNHEALTHY record set withdrawn fail pass 3rd failback needs the full consecutive success count Roughly 90 seconds elapse between the first failed probe and the withdrawn record, and the same again on the way back - failback is never faster than failover.

Step 3 — Attach the failover record pair

The PRIMARY set carries the health check; the SECONDARY set deliberately does not, so it remains answerable when everything else is on fire. Both share the same name and type and are distinguished by SetIdentifier.

resource "aws_route53_record" "app_primary" {
  zone_id         = data.aws_route53_zone.main.zone_id
  name            = "app.example.com"
  type            = "A"
  ttl             = 60
  records         = ["203.0.113.10"]
  set_identifier  = "primary-us-east"
  health_check_id = aws_route53_health_check.primary.id

  failover_routing_policy {
    type = "PRIMARY"
  }
}

resource "aws_route53_record" "app_secondary" {
  zone_id        = data.aws_route53_zone.main.zone_id
  name           = "app.example.com"
  type           = "A"
  ttl            = 60
  records        = ["198.51.100.10"]
  set_identifier = "secondary-eu-west"

  failover_routing_policy {
    type = "SECONDARY"
  }
}

If your primary is an ALB, CloudFront distribution, or another AWS-hosted target, use an alias record with evaluate_target_health = true instead of a standalone health check — the target’s own health signal is more accurate and costs nothing. Be aware that alias records inherit the target’s TTL rather than declaring one, which interacts with apex naming as described in CNAME Flattening Explained.

Step 4 — Alarm on the health check state

Route 53 publishes HealthCheckStatus to CloudWatch in us-east-1 regardless of where your workload runs. Use the Minimum statistic so a single reporting period containing a zero trips the alarm, and treat missing data as breaching so a checker outage does not silently mute you.

aws cloudwatch put-metric-alarm --region us-east-1 \
  --alarm-name r53-app-primary-unhealthy \
  --namespace AWS/Route53 --metric-name HealthCheckStatus \
  --dimensions Name=HealthCheckId,Value=b1f4c2de-1f0e-4b6a-9d13-2b7d9e5a1c34 \
  --statistic Minimum --period 60 --evaluation-periods 2 \
  --threshold 1 --comparison-operator LessThanThreshold \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:dns-oncall

Add a second alarm on HealthCheckPercentageHealthy with a threshold around 50 to catch partial failures — a state where a third of checker regions cannot reach your endpoint, which usually means a regional network problem rather than an application one, and which will not flip the aggregate status.

Step 5 — Rehearse the forced failover

Break the endpoint on purpose and time the result. The cleanest lever is a feature flag that makes /healthz return 503 without touching real traffic; blocking the checker source ranges at the firewall also works but tests a different failure path.

curl -sS -X POST https://us-east.origin.example.com/admin/healthz/force-degraded \
  -H "Authorization: Bearer $ADMIN_TOKEN"

start=$(date +%s)
while :; do
  answer=$(dig +short @8.8.8.8 app.example.com A | paste -sd, -)
  printf '%4ss  %s\n' "$(( $(date +%s) - start ))" "$answer"
  [ "$answer" = "198.51.100.10" ] && break
  sleep 5
done

Expected output — the primary address for roughly 90 seconds of detection plus up to the full TTL of cache drain, then a clean switch:

   0s  203.0.113.10
  ...
  95s  203.0.113.10
 130s  198.51.100.10

Then restore and watch the failback, which is the half nobody rehearses. Because recovery requires the same number of consecutive successes, expect it to take at least as long as the failover did.

Forced failover drill timeline Five stages of a failover drill with elapsed times: forcing a 503 response, the first failed probe, the health check flipping to unhealthy, the authoritative answer changing to the secondary address, and public resolvers converging after the TTL. What a drill looks like with a stopwatch t+0 s Force /healthz to return 503 on the primary origin t+30 s First probe fails; status still HEALTHY, record still answered t+90 s Third consecutive failure; check flips UNHEALTHY, CloudWatch alarm fires t+92 s Authoritative nameservers now answer with 198.51.100.10 t+150 s Public resolvers have drained the 60 s TTL and converged Record the real numbers from your own drill; they are the ones your runbook should quote.

Step 6 — Roll back cleanly

Rollback means returning to a known-good answer, not deleting resources. Disassociate the health check from the record set — leaving both objects intact — so the PRIMARY set is unconditionally answered again:

aws route53 change-resource-record-sets --hosted-zone-id Z2FDTNDATAQYW2 --change-batch '{
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "app.example.com.",
      "Type": "A",
      "SetIdentifier": "primary-us-east",
      "Failover": "PRIMARY",
      "TTL": 60,
      "ResourceRecords": [{ "Value": "203.0.113.10" }]
    }
  }]
}'

Keep the failing endpoint serving degraded responses for one full TTL after the change; a degraded origin still answering is strictly better than an origin that has already been terminated while resolvers still point at it.

Verification

Query the zone’s own nameservers to see the authoritative answer with no caching in the way, then compare against public resolvers to see what users get.

for ns in $(dig +short NS example.com); do
  printf '%-34s %s\n' "$ns" "$(dig +short @"$ns" app.example.com A)"
done
ns-1024.awsdns-00.org.             203.0.113.10
ns-1536.awsdns-00.co.uk.           203.0.113.10
ns-256.awsdns-32.com.              203.0.113.10
ns-768.awsdns-32.net.              203.0.113.10

Confirm the TTL that resolvers are actually caching — the number in the second column is what bounds your failover, and a value larger than 60 means something upstream overrode you:

dig +noall +answer @1.1.1.1 app.example.com A
app.example.com.	60	IN	A	203.0.113.10

Read the health check’s own view, including per-region observations, which is the fastest way to distinguish “my app is broken” from “one region cannot reach my app”:

aws route53 get-health-check-status --health-check-id b1f4c2de-1f0e-4b6a-9d13-2b7d9e5a1c34 \
  --query 'HealthCheckObservations[].{Region:Region,IP:IPAddress,Status:StatusReport.Status}' \
  --output table

aws route53 get-health-check-last-failure-reason \
  --health-check-id b1f4c2de-1f0e-4b6a-9d13-2b7d9e5a1c34 \
  --query 'HealthCheckObservations[0].StatusReport.Status' --output text

Troubleshooting

Health checker IP ranges blocked by a firewall

Symptom: every checker region reports failure at the same instant, and curl from your laptop succeeds. Diagnose with get-health-check-status — uniform failure across geographically unrelated regions is a network policy, not an application fault. Route 53 publishes its checker ranges in the ROUTE53_HEALTHCHECKS service entry of the IP ranges document, which you can pull and turn into a security group or WAF allowance:

curl -s https://ip-ranges.amazonaws.com/ip-ranges.json \
  | jq -r '.prefixes[] | select(.service=="ROUTE53_HEALTHCHECKS") | .ip_prefix' | head

Fix by allowing those ranges on port 443 and exempting /healthz from WAF managed rules and from any rate limiter. If you already run rate limiting at the edge, remember that a checker hitting you every 30 seconds from 15 locations is 30 requests a minute from a rotating set of addresses.

Health check passes while the application is broken

Symptom: users are getting 500s, the check is stubbornly green. This is almost always a HTTP/HTTPS type check that only validates the status line, or an HTTPS_STR_MATCH whose search string appears in an error page too. Verify what the checker sees, in the checker’s own terms:

curl -sS -o /dev/null -w '%{http_code}\n' https://us-east.origin.example.com/healthz
curl -sS https://us-east.origin.example.com/healthz | head -c 5120 | grep -c '"status":"ok"'

A grep -c result of 0 while the status code is 200 means the string match is doing its job and the type is wrong. Switch to HTTPS_STR_MATCH and pick a string that only a healthy dependency graph can produce.

Failover not observed because of resolver caching

Symptom: the authoritative servers return the secondary address, but a specific office or user still lands on the dead primary. Compare authoritative and cached views side by side, and check the remaining TTL rather than only the address:

dig +noall +answer @ns-256.awsdns-32.com app.example.com A
dig +noall +answer @8.8.8.8 app.example.com A
dig +noall +answer @9.9.9.9 app.example.com A

If the public resolvers show a TTL larger than the one you configured, an intermediate forwarder is clamping it. There is no remote flush for someone else’s cache — the fix is to wait, and to shed the connections at the origin so clients are forced to resolve again. The methodology for tracking this across many vantage points is in Debugging DNS Propagation Delays Across Global Resolvers.

Calculated and CloudWatch-metric checks misbehaving

A CALCULATED check aggregates child checks with a HealthThreshold; a CLOUDWATCH_METRIC check tracks an alarm state. Both fail in the same way: Inverted left on from an experiment, or InsufficientDataHealthStatus defaulting to LastKnownStatus so a brand-new check that has never reported is treated as healthy forever. Inspect the configuration rather than guessing:

aws route53 get-health-check --health-check-id 9c22a5f1-77b3-4e0e-8a41-6d0f2c1b4e77 \
  --query 'HealthCheck.HealthCheckConfig.{Type:Type,Inverted:Inverted,Threshold:HealthThreshold,Insufficient:InsufficientDataHealthStatus}'

Set InsufficientDataHealthStatus to Unhealthy for anything gating a failover, so an absence of signal is treated as bad news.

Alias versus non-alias failover records

Symptom: terraform plan refuses the configuration, or failover works but the TTL you set is ignored. Alias records cannot carry a ttl and cannot carry a health_check_id together with evaluate_target_health in the way people expect — for an alias to an AWS target, set evaluate_target_health = true and drop the standalone check. For a non-alias A record pointing at a raw IP, the reverse holds: you must attach a health check ID, because there is no target health to evaluate. Mixing an alias PRIMARY with a non-alias SECONDARY is legal and common; just remember the alias half reports the TTL of its target, not yours.

Frequently Asked Questions

How fast can Route 53 DNS failover actually complete? Detection is RequestInterval × FailureThreshold, so about 90 seconds at the defaults and about 30 seconds with a 10-second interval. Add the record TTL for cache drain, and expect the observed end-to-end time to land between two and three minutes with a 60-second TTL — not the sub-second failover an edge load balancer gives you.

Should the SECONDARY record have its own health check? Usually not. The SECONDARY is the answer of last resort, and attaching a check to it means both sets can be unhealthy at once, at which point Route 53 falls back to answering with everything anyway. Attach a check to the secondary only when you have a third tier — a static maintenance page, for example — for it to fail over to.

Why did my check go unhealthy when the site was clearly up? The most common causes are a WAF or rate limiter treating the checkers as abusive, an expired or misconfigured certificate with EnableSNI set incorrectly, or a /healthz handler that occasionally exceeds the connection timeout under load. Check get-health-check-last-failure-reason first — it names the failure class directly.

Do health checks work with private endpoints inside a VPC? No. Route 53 health checkers run on the public internet and cannot reach a private address. Use a CLOUDWATCH_METRIC health check driven by an alarm on a metric your private workload already publishes, or expose a dedicated public health endpoint that proxies the internal check.

What does it cost to run these checks? Route 53 bills per health check per month, with surcharges for HTTPS, string matching, and latency measurement. A handful of checks is negligible; several hundred per-endpoint checks across many services is a line item worth reviewing, and is a good argument for checking a small number of aggregate endpoints rather than every instance individually.

Back to DNS Load Balancing & Failover