Edge Observability & Debugging

Edge observability is the practice of reconstructing what a request did inside a runtime you cannot log into, from telemetry that was sampled at the point of production, buffered in a colocation facility you will never visit, and delivered minutes after the incident you are trying to explain.

Every debugging habit built on centralized servers breaks at the edge. There is no host to SSH into, no /var/log to tail -f, no single process whose memory you can inspect. Your code runs in several hundred points of presence simultaneously, each with its own isolate pool, its own cache, and its own view of the network. Worse, a large fraction of the requests you care about never touch your origin at all — the edge function terminated them with a redirect, a cached response, or a 403. If the edge does not tell you about those requests, nothing else will.

Key implementation points:

• Treat the four edge signals — live tail, structured logs, metrics, and traces — as four different instruments with different retention, latency and cost, not as four names for the same thing.
• Emit one structured JSON line per request from inside the function, carrying a request identifier you can join against the origin’s access log.
• Ship logs off the isolate with event.waitUntil() (or the platform’s equivalent) so the shipping call survives the response returning, without consuming your CPU budget.
• Budget log volume explicitly: sample by outcome rather than uniformly, and decide up front which fields are redacted before they leave the PoP.

Why edge debugging is a different discipline

Start with the property that causes the most confusion: your code is not running in one place. A deploy pushes the same script to every PoP, but each PoP serves a different population, holds a different cache, and reaches your origin over a different network path. A bug that manifests in Frankfurt and nowhere else is not a rare bug — it is a bug in the interaction between your code and one PoP’s conditions, and the only way to see it is to make your telemetry carry the PoP identity on every line.

The second property is that the isolate is disposable. A Worker or edge function has no filesystem, no long-lived process, and no guarantee that the isolate serving request N will still exist for request N+1. Anything you want to keep must leave the isolate over the network before the invocation ends. That single constraint explains most of the design of edge logging: logs are pushed, never pulled, and the push has to be scheduled in a way that does not block the response you are trying to keep fast.

The third property is the one that catches teams late. At the edge, a growing share of your traffic is terminated before it reaches the origin — cached responses, country redirects, WAF blocks, rate-limit rejections, and auth failures. Those requests produce no origin access-log entry at all. If your dashboards are built on origin logs, then improving your edge configuration makes your dashboards emptier, and an incident affecting only edge-terminated traffic is invisible. The better your edge gets, the more your observability has to move to the edge with it.

Finally, everything you see is sampled by default at some layer, and the sampling is usually applied before the filtering. That inverts the intuition you built on origin logs, where you filter first and the survivors are complete. At the edge you filter a sample, and the difference matters enormously when the thing you are chasing is rare.

The four signals and what each one can answer

Live tail is a websocket from your terminal to the running script. It is the only signal with no ingestion delay, which makes it the right tool for a deploy you are watching or a bug you can reproduce on demand. It is also the only signal with zero retention: close the terminal and the evidence is gone. Streaming a tail is covered end to end in Streaming Worker Logs with wrangler tail.

Structured logs are one JSON object per request, emitted from inside the function and shipped to storage you control. This is the signal that answers “what happened at 04:12 last Tuesday”, and it is the only one that supports arbitrary post-hoc queries. It is also the expensive one — see Shipping Edge Logs with Logpush for the delivery mechanics and batching behavior.

Metrics and analytics are counters and distributions computed at write time. Because the aggregation happens before storage, you can afford to record every request, which is exactly the property sampled logs lack. The trade is that you cannot drill into an individual request; you get “how many” and “how slow”, never “which one”. Cloudflare’s Analytics Engine, CloudFront’s per-distribution metrics, and Vercel’s usage analytics all sit here.

Traces are spans stitched together by a correlation identifier that travels in a request header. A trace is the only signal that shows you the shape of a request across hops — edge function, subrequest, origin, downstream service — with real durations attached. The technique for propagating the identifier across those hops is the subject of Tracing Requests Across the Edge with Correlation Headers.

A healthy edge deployment uses all four, and uses them for different questions. Alerting fires off metrics because metrics are complete and cheap. Triage starts in structured logs because logs are queryable. Root cause usually needs a trace. And the fix gets verified on a live tail while you push the canary. Reaching for the wrong instrument is the most common reason an edge incident takes three hours instead of twenty minutes.

Sampling, and the statistics that ruin your afternoon

Every edge platform samples. Live tail samples once a script exceeds roughly a hundred requests per second, because the stream cannot keep up. Trace vendors sample at ingest to control cardinality. Log pipelines sample because you configured a head sampling rate to keep the bill sane. The danger is not sampling itself — it is reasoning about a sample as though it were the population.

Work the arithmetic once and you will never forget it. Suppose you serve 10,000 requests per minute and 0.12% of them return 502. That is twelve failures a minute, a genuine and page-worthy problem. Now apply a 1% head sampling rate. Your sample contains 100 requests and, in expectation, 0.12 failures. The probability that a given minute’s sample contains zero 502s is roughly 89%. You will stare at a clean log stream while customers open tickets, and you will conclude the problem is intermittent or already fixed.

Two corrections cover almost every case. The first is tail-based or outcome-based sampling: decide whether to keep a record after you know how the request ended. Keep every exception and every 5xx, keep every request that exceeded a latency threshold, and keep a small percentage of the boring ones for baseline comparison. Cloudflare’s Workers observability config exposes a head sampling rate, but you can implement outcome-based retention yourself by only calling your log-shipping function on interesting requests. The second correction is count everything, log a sample: push a counter into an analytics dataset on every single request, and let the sampled log lines serve only as examples. The counter tells you the failure exists; the sample gives you something to read.

One subtlety that bites: head sampling that is decided per request produces an unbiased estimate of rates but destroys traces, because the edge span and the origin span may make different keep/drop decisions. If you need coherent traces, the sampling decision must be made once, at the first hop, and propagated in the header alongside the trace identifier.

What an edge runtime can and cannot observe

The V8-isolate runtimes used by Cloudflare Workers, Vercel Edge functions and Deno Deploy deliberately expose a small surface. Knowing where the walls are saves hours of chasing tools that will never work there.

There is no filesystem. fs does not exist, so there is nowhere to buffer a log file and nothing to rotate. Every log line must go out over fetch() or through a platform-native sink.

CPU time is metered separately from wall-clock time. Waiting on a subrequest is usually free; JSON-stringifying a large object is not. A logging helper that serializes the entire request body on every invocation is a genuine way to hit the CPU ceiling and start returning 1102-class errors under load. Keep serialization proportional to what you actually need.

event.waitUntil() (or ctx.waitUntil()) is the escape hatch for shipping. It extends the lifetime of the invocation past the moment the response is returned, so a fetch() to your log collector can complete after the user already has their bytes. Without it, the runtime is entitled to cancel the in-flight request the instant the response stream closes, and your logs vanish silently — no error, no partial write, nothing.

export default {
  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    const started = Date.now();
    const requestId = crypto.randomUUID();

    const response = await fetch(request);

    // Build the record AFTER the outcome is known, so sampling can be outcome-based.
    const record = {
      ts: new Date().toISOString(),
      request_id: requestId,
      colo: (request as any).cf?.colo ?? "unknown",
      method: request.method,
      path: new URL(request.url).pathname,
      status: response.status,
      duration_ms: Date.now() - started,
    };

    const interesting = response.status >= 500 || record.duration_ms > 1000;
    if (interesting || Math.random() < 0.01) {
      // waitUntil keeps the invocation alive until the POST finishes.
      ctx.waitUntil(
        fetch("https://collector.example.com/v1/edge", {
          method: "POST",
          headers: { "content-type": "application/json" },
          body: JSON.stringify(record),
        })
      );
    }

    const out = new Response(response.body, response);
    out.headers.set("x-request-id", requestId);
    return out;
  },
};

console.log() is not free and is not durable. On most platforms it feeds the live tail and, if you have enabled it, the platform’s own log store. It is not a transport to your own systems, and on some runtimes a very large log line is truncated or dropped entirely rather than chunked.

You cannot attach a debugger to production. There is no remote inspector on a live PoP. The closest equivalents are running the isolate locally against real edge infrastructure (wrangler dev --remote) and replaying a captured request against a preview deployment.

Correlating an edge log line with an origin log line

The single highest-value piece of edge instrumentation is a request identifier that appears in both the edge record and the origin record. Without it, joining the two datasets means guessing from timestamps and URLs, which fails precisely when you need it most — during a burst, when a thousand requests share the same second and the same path.

Generate the identifier at the edge, because the edge is the first hop that runs your code. Put it on the outbound subrequest as a header, echo it back to the client on the response, and log it on both sides. On Cloudflare you get cf-ray for free and it is genuinely useful — its suffix encodes the colo — but it is Cloudflare’s identifier, not yours, and it does not survive a multi-CDN or multi-provider path. Emit your own alongside it.

On the origin side, make the identifier part of the access-log format so the join is a column comparison rather than a regex over free text. For nginx that is a one-line change:

log_format edgejoin '$remote_addr $http_x_request_id "$request" '
                    '$status $body_bytes_sent $request_time '
                    '"$http_cf_ray" "$http_cf_ipcountry"';

access_log /var/log/nginx/access.log edgejoin;

Two failure modes are worth naming. First, if a request is served from cache the origin never sees it, so a missing origin row for a given identifier is information, not a data-quality bug — it tells you the edge terminated the request. Second, some proxies strip unknown request headers; if your identifiers stop arriving at the origin after an infrastructure change, check the header allowlist before you suspect the edge code.

Provider implementations

Cloudflare

Cloudflare gives you the fullest set of the four signals, at four different price points.

wrangler tail opens the live stream. It attaches to a deployed script by name, supports server-side filters, and emits either a human-readable format or newline-delimited JSON:

# Human-readable, errors only
npx wrangler tail --name edge-router --status error --format pretty

# JSON for machine triage, one specific environment
npx wrangler tail --name edge-router --env production --format json \
  | jq -c '{outcome, url: .event.request.url, msg: .logs[0].message}'

Workers Logs turns console.log() output into a queryable store with a configurable head sampling rate, enabled declaratively:

{
  "name": "edge-router",
  "main": "src/index.ts",
  "compatibility_date": "2025-01-15",
  "observability": {
    "enabled": true,
    "head_sampling_rate": 0.1
  },
  "analytics_engine_datasets": [
    { "binding": "EDGE_METRICS", "dataset": "edge_router_metrics" }
  ]
}

Logpush pushes the workers_trace_events dataset — one record per invocation, including Outcome, Exceptions, Logs and the script name — to R2, S3 or an HTTP endpoint on a batching interval:

curl -sX POST \
  "https://api.cloudflare.com/client/v4/accounts/${ACCOUNT_ID}/logpush/jobs" \
  -H "Authorization: Bearer ${CF_API_TOKEN}" \
  -H "content-type: application/json" \
  -d '{
        "name": "worker-trace-events",
        "dataset": "workers_trace_events",
        "destination_conf": "r2://edge-logs/workers?account-id='"${ACCOUNT_ID}"'&access-key-id='"${R2_KEY}"'&secret-access-key='"${R2_SECRET}"'",
        "output_options": {
          "field_names": ["EventTimestampMs","ScriptName","Outcome","Exceptions","Logs","Event"],
          "timestamp_format": "rfc3339"
        },
        "enabled": true
      }'

Analytics Engine is the write-cheap counter store. Every request can afford a data point, which is what makes it immune to the sampling trap described above:

env.EDGE_METRICS.writeDataPoint({
  blobs: [request.method, new URL(request.url).pathname, (request as any).cf?.colo ?? "??"],
  doubles: [response.status, durationMs],
  indexes: [String(response.status)],
});

Finally, a Tail Worker (declared as a tail_consumers entry) receives the trace events of another Worker as a normal tail() handler, which lets you redact, reshape and route logs in code before they leave your account — the cleanest place to implement redaction policy.

AWS: Lambda@Edge and CloudFront

Lambda@Edge has one behavior that surprises nearly everyone the first time: its CloudWatch log groups are created in the AWS region closest to the edge location that executed the function, not in the region where you defined the function. A viewer-request function triggered in Sydney writes to ap-southeast-2. Hunting for logs in us-east-1 and finding an empty log group is the single most common Lambda@Edge support question.

# Enumerate the per-region log groups your function has actually written to
for region in us-east-1 eu-west-1 ap-southeast-2 sa-east-1; do
  aws logs describe-log-groups --region "$region" \
    --log-group-name-prefix "/aws/lambda/us-east-1.edge-router" \
    --query 'logGroups[].logGroupName' --output text
done

For request-level data independent of your function code, CloudFront real-time logs stream selected fields into a Kinesis Data Stream within seconds, with a configurable sampling rate:

aws cloudfront create-realtime-log-config \
  --name edge-router-rtl \
  --sampling-rate 100 \
  --fields timestamp c-ip sc-status cs-uri-stem x-edge-location \
           x-edge-result-type time-taken x-edge-request-id \
  --end-points StreamType=Kinesis,KinesisStreamConfig="{RoleARN=${ROLE_ARN},StreamARN=${STREAM_ARN}}"

The x-edge-request-id field is CloudFront’s own correlation key, and it is returned to the client in the x-amz-cf-id response header — the practical equivalent of cf-ray.

Vercel

Vercel exposes runtime logs in the dashboard and through the CLI, with a short retention window that depends on plan tier. That makes them a triage tool rather than an archive:

vercel logs https://my-app.vercel.app --json

For anything you need to keep, configure a log drain, which POSTs newline-delimited JSON to an endpoint you own. Inside Vercel Edge Middleware the shipping pattern mirrors Cloudflare’s, using waitUntil from the platform helpers:

import { waitUntil } from '@vercel/functions';
import { NextRequest, NextResponse } from 'next/server';

export function middleware(req: NextRequest) {
  const res = NextResponse.next();
  const requestId = req.headers.get('x-vercel-id') ?? crypto.randomUUID();
  res.headers.set('x-request-id', requestId);

  waitUntil(
    fetch('https://collector.example.com/v1/edge', {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({
        request_id: requestId,
        path: req.nextUrl.pathname,
        region: process.env.VERCEL_REGION ?? 'dev',
        country: req.headers.get('x-vercel-ip-country') ?? 'XX',
      }),
    })
  );

  return res;
}

The x-vercel-id header carries the region identifier of the PoP that served the request, which is the field you filter on when a problem is regional.

Fastly

Fastly’s model is the most explicit: you declare named logging endpoints, then write to them from VCL or from Compute code. Because the write is a first-class statement rather than a side effect of console.log(), the log format is entirely yours and the delivery is real time.

sub vcl_log {
#FASTLY log
  log {"syslog "} req.service_id {" edge_json :: "}
    {"{"}
      {""ts":""} time.start.sec {"","}
      {""req_id":""} req.http.X-Request-ID {"","}
      {""pop":""} server.datacenter {"","}
      {""status":"} resp.status {","}
      {""cache":""} fastly_info.state {"","}
      {""ttfb_ms":"} time.elapsed.msec
    {"}"};
}

fastly_info.state distinguishes HIT, MISS, PASS and HIT-STALE, which makes Fastly logs unusually good at answering caching questions without a separate dataset — useful alongside stale-while-revalidate and resilient caching work.

Platform comparison

Read the last column as what you lose when the pipeline breaks. Every one of these systems is deliberately fail-open: logging never blocks traffic. That is the correct trade for availability and a genuinely dangerous one for compliance, because a silent gap in your audit trail looks identical to a period of no traffic. If you have retention obligations, alert on log volume falling below a floor, not only on it spiking.

A debugging playbook

When a report arrives, resist the urge to open a dashboard. Work the sequence below; it converges far faster than exploratory querying.

1. Reproduce before you investigate. Capture the exact URL, method, headers and body from the report, and replay them yourself with curl. Half of all “edge bugs” are a client sending something you did not expect — a stale Accept-Encoding, a cookie from a previous release, a Range header. Note the cf-ray, x-vercel-id or x-amz-cf-id of your own reproduction; you will need it in step three.

2. Isolate the point of presence. Compare the identifier suffix on a failing request with one on a working request. If they differ, you have a regional problem: a PoP-local cache entry, a route to a degraded origin, or a geo rule. If they match, the problem is in your code. Provider status pages are worth thirty seconds here, since a partially degraded PoP produces exactly this signature and no amount of code reading will fix it.

3. Replay against a stream you are watching. Open a live tail filtered to your own address, then send the reproduction. Now every console.log() in the code path lands in front of you in real time, in order, for a request you control. This is the closest the edge gets to a breakpoint, and it is the step most teams skip.

4. Bisect the middleware chain. Edge handlers accumulate stages: security headers, auth, geo, rewrite, cache, origin fetch. Add a single log line at the entry and exit of each stage with the stage name and elapsed milliseconds, deploy to a preview environment, and replay. The stage where the elapsed time jumps or the exit log never appears is your suspect. A chain that logs its own boundaries turns a multi-hour hunt into a single read.

5. Verify the fix under a canary. Deploy the new version to a small traffic percentage, keep the tail open filtered to --status error, and watch for the class of failure to disappear while total error volume stays flat. Promote only after both conditions hold — see the deployment mechanics alongside edge health checks and automatic failover.

Redaction, retention and the cost of logging everything

Edge logs are the most privacy-sensitive data in your stack, because the edge sees the raw request before any application-level scrubbing. A naive “log the whole request” helper will capture Authorization headers, session cookies, password fields in form bodies, and full query strings containing reset tokens. Once that data is in an object store it is subject to every retention and access rule your compliance regime imposes, and deleting it selectively from compressed batch files is miserable.

Redact at the edge, before the record leaves the PoP. An allowlist beats a denylist every time — enumerate the headers you keep rather than the ones you strip, because the next SDK release will add a header you did not think to block.

const HEADER_ALLOWLIST = [
  "user-agent",
  "accept",
  "accept-encoding",
  "content-type",
  "referer",
  "x-request-id",
];

const VOLATILE_QUERY = /^(token|code|reset|session|sig|signature)$/i;

function safeHeaders(h: Headers): Record<string, string> {
  const out: Record<string, string> = {};
  for (const key of HEADER_ALLOWLIST) {
    const value = h.get(key);
    if (value) out[key] = value.slice(0, 256);
  }
  return out;
}

function safeUrl(raw: string): string {
  const url = new URL(raw);
  for (const [k] of [...url.searchParams]) {
    if (VOLATILE_QUERY.test(k)) url.searchParams.set(k, "[redacted]");
  }
  return url.toString();
}

Truncating each value (slice(0, 256) above) does double duty: it caps the blast radius of an unexpectedly large header and it caps your per-record size, which is what actually drives cost. A single JSON line of 1.2 KB at 20,000 requests per second is about 2 TB per day before compression. Trimming that record to 400 bytes and sampling successful requests at 5% takes the same traffic to well under 100 GB per day. The levers, in descending order of effect, are: sample successes aggressively, drop fields you have never queried, truncate long values, and only then negotiate storage pricing.

Set a retention policy per dataset rather than one policy for everything. Exception records and 5xx samples justify long retention because they feed post-incident review. Full-fidelity 2xx records rarely justify more than a week — the metrics derived from them are what you actually query at month three.

Delivery latency, batching and propagation

Observability at the edge has its own version of TTL, and confusing the windows is a reliable way to misread an incident.

Live tail is sub-second but lossy. Events reach your terminal almost immediately, but the stream is sampled above roughly a hundred requests per second and the connection can be reset by the platform. Absence of an event in a tail is never evidence that the event did not happen.

Batch pipelines add minutes. Logpush and comparable systems accumulate records per PoP and flush on whichever comes first — a byte threshold or a time interval, commonly somewhere between five seconds and five minutes. During a global incident this means the last few minutes of data are still in flight, so a graph that appears to show recovery may simply be showing an unflushed tail. Always check the maximum event timestamp in your store before concluding anything about “now”.

Records arrive out of order. Different PoPs flush independently, so a batch from Singapore may land after a later batch from Dublin. Never build alerting on “the newest row in the table”; partition and query by event time, not ingestion time.

Sampling changes propagate on the deploy cadence. Turning a head sampling rate from 1% to 100% requires a redeploy on most platforms, and that deploy propagates globally in seconds — but the records already sampled away are gone forever. When you suspect you are under-sampling during an incident, raise the rate immediately rather than after you finish the current line of investigation.

Analytics aggregates have their own granularity. A counter store that rolls up into one-minute buckets cannot answer a question about a fifteen-second spike, no matter how long you retain it. Choose the bucket width against your alerting requirements before you commit.

Troubleshooting and rollback

Rollback protocol. Observability changes deserve the same rollback discipline as routing changes, because a logging bug can take down the function it instruments. If a new logging path is implicated, revert the deployment first (wrangler rollback, or repointing the CloudFront behavior at the previous function version) and investigate afterwards. Disable a Logpush job rather than deleting it — deletion loses the field configuration you tuned. And keep at least one always-on signal that does not depend on your own code: provider-native analytics keep reporting even when your custom logging is the thing that is broken.

Edge cases and gotchas

• A return before your logging call means no log line. Early returns for redirects, 304s and auth failures are exactly the paths you most want visibility into, and exactly the paths that skip a logger placed at the bottom of the handler. Wrap the handler and log in a finally, or log from a single exit point.
• try/catch around your entire handler converts exceptions into successes. The platform’s outcome field will report ok because nothing propagated out of the isolate. Re-log the caught error explicitly with a distinguishing field, or you will have a silent failure class that no dashboard counts.
• Lambda@Edge log groups appear in the region nearest the executing edge location. Budget for cross-region log aggregation from day one rather than discovering it during an incident.
• console.log inside a streaming response may execute after headers are sent. The ordering you see in a tail is emission order, not necessarily the order you reason about when reading the code top to bottom.
• Sampling applied before filtering. A tail filtered to --status error still only sees errors that survived the sample. Filters narrow what you are shown; they do not un-sample.
• Multiple tail sessions on one script compete. Platforms cap concurrent tail consumers (commonly around ten), and each additional session increases the chance of sampling. Coordinate during incidents rather than having six engineers each open their own stream.
• Preview and production emit into the same store unless you separate them. Tag every record with the environment name and the deployment version, or your preview traffic will contaminate the baselines you alert on.
• Clock skew between edge and origin is small but non-zero. Never join records on timestamps alone; that is what the request identifier is for.

Frequently Asked Questions

Why can I not just SSH into an edge node and read the logs? Edge runtimes are multi-tenant sandboxes with no shell, no filesystem and no per-customer host — your code shares an isolate pool with thousands of other scripts on hardware you have no account on. Everything you learn about a request has to be emitted by your own code or by the platform’s instrumentation and shipped off the node over the network.

Is wrangler tail enough on its own for production debugging? No. Tail is sub-second and unfiltered by retention policy, which makes it excellent for a bug you can reproduce right now, but it stores nothing and it samples once a script gets busy. Pair it with a batch pipeline such as Logpush for anything you need to query after the fact.

How do I match an edge log line to the corresponding origin log line? Generate a request identifier in the edge function, forward it to the origin as a request header, echo it on the response, and log it on both sides. Add the header to your origin’s access-log format so the join is a column match; a missing origin row then reliably means the edge terminated that request.

What sampling rate should I run in production? Sample by outcome rather than picking one uniform number: keep 100% of exceptions, 5xx responses and requests over your latency budget, and keep a small percentage — 1% to 5% is typical — of successful requests for baseline comparison. Push a counter for every request into an analytics dataset so your rate calculations stay accurate regardless of what the log sample contains.

Related

• Streaming Worker Logs with wrangler tail
• Shipping Edge Logs with Logpush
• Tracing Requests Across the Edge with Correlation Headers
• Cloudflare Workers Routing
• Tuning WAF False Positives with Log Mode

Back to Edge Routing & Serverless Function Architecture