Edge & DNS Ops Guide

Tracing Requests Across the Edge with Correlation Headers

After working through this guide you will be able to design a correlation scheme that survives every hop a request takes — client, edge function, rewrite, subrequest, origin — so that one identifier links an edge log line, an origin log line and a customer support ticket. You will mint W3C trace context at the first edge hop, take the sampling decision once, echo a readable id back to the caller, and keep that id out of your cache key.

Correlation is a design problem before it is a coding problem. Any single hop can generate an id; what makes tracing useful is that every hop agrees on the same id, and that agreement has to be designed in, because the default behaviour of almost every proxy, framework and HTTP client is to invent a fresh one. The failure you are engineering against is not a missing log line — it is four systems that each logged the request perfectly under four different names.

Key implementation objectives:

  • Accept an inbound traceparent when it is well formed, and mint one at the edge when it is not.
  • Propagate trace context unchanged through rewrites, subrequests and the origin fetch, minting a new span id per hop.
  • Take the sampled decision once, at the first edge hop, so edge and origin never disagree about whether to record.
  • Echo a human-quotable X-Request-Id on the response without letting it reach the cache key.

The two identifiers, and why you need both

traceparent is machine-facing and rigid: a fixed-length, hex-encoded field defined by the W3C Trace Context recommendation, designed so intermediaries can parse and forward it without understanding your system. X-Request-Id is human-facing: short enough that a customer can read it off an error page and paste it into a ticket. They carry the same correlation, in two shapes, for two audiences.

Anatomy of a traceparent header The traceparent header split into its version, trace id, parent span id and flags fields, each annotated with what it means and which hop is allowed to change it. traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 00 version 4bf9…4736 trace-id · 16 bytes · 32 hex 00f0…02b7 parent-id · 8 bytes 01 trace-flags fixed always 00 never changes one per request chain changes per hop the caller's span decided once 01 = sampled tracestate carries vendor key/value pairs alongside it; X-Request-Id carries a short readable form. All three travel as request headers only — none of them belongs in a URL, a cookie or a cache key.

An all-zero trace id or parent id is invalid by specification, as is a version of ff. Treat any malformed value as absent and mint a fresh one rather than forwarding garbage — a corrupt trace id poisons the join for every downstream system that trusts it.

Prerequisites

You need Wrangler 3.60 or newer for the Worker examples, Node 18+ for the Vercel middleware, and an origin whose logging format you can change. Confirm your toolchain and that crypto.randomUUID is available in your runtime:

node --version            # v18.x or later
npx wrangler --version    # 3.60.0 or later
npx wrangler whoami

The origin does not need a tracing SDK for this to be useful. Writing the trace id into an existing structured log line is enough to make the join work; a full OpenTelemetry pipeline is a later upgrade, not a prerequisite.

Step-by-step procedure

Step 1 — Accept or mint at the first edge hop

The first hop that touches the request owns the decision. Parse any inbound traceparent; if it is well formed, adopt its trace id and treat its parent id as your parent. If it is missing or malformed, generate both. Generating 16 random bytes for the trace id and 8 for the span id gives a collision probability low enough to ignore at any realistic volume.

const TRACEPARENT = /^([\da-f]{2})-([\da-f]{32})-([\da-f]{16})-([\da-f]{2})$/;

function hex(bytes: number): string {
  const b = new Uint8Array(bytes);
  crypto.getRandomValues(b);
  return [...b].map((n) => n.toString(16).padStart(2, "0")).join("");
}

type Ctx = { traceId: string; parentId: string; spanId: string; sampled: boolean; inherited: boolean };

function contextFrom(request: Request, sampleRate: number): Ctx {
  const m = TRACEPARENT.exec(request.headers.get("traceparent") ?? "");
  const valid =
    m && m[1] !== "ff" && !/^0+$/.test(m[2]) && !/^0+$/.test(m[3]);

  if (valid) {
    return {
      traceId: m[2],
      parentId: m[3],
      spanId: hex(8),                       // this hop gets its own span
      sampled: (parseInt(m[4], 16) & 0x01) === 1,
      inherited: true,
    };
  }
  return {
    traceId: hex(16),
    parentId: "0000000000000000",
    spanId: hex(8),
    sampled: Math.random() < sampleRate,    // decided here, once
    inherited: false,
  };
}

The side effect worth noticing: sampled is only computed on the inherited: false branch. Once a decision exists upstream it is inherited verbatim, which is what stops the edge recording a span whose origin counterpart was dropped.

Step 2 — Propagate through every outbound fetch

Request objects in Workers are immutable, so propagation means constructing a new Request with an amended header set for each subrequest — including the origin fetch, and including any fetch you make after a rewrite. This is the step people skip, and it is why traces so often stop at the edge.

function withTrace(request: Request, ctx: Ctx, requestId: string): Request {
  const headers = new Headers(request.headers);
  headers.set("traceparent", `00-${ctx.traceId}-${ctx.spanId}-${ctx.sampled ? "01" : "00"}`);
  headers.set("tracestate", `edge=colo:${(request as any).cf?.colo ?? "unknown"}`);
  headers.set("x-request-id", requestId);
  return new Request(request, { headers });
}

export default {
  async fetch(request: Request, env: Env, exeCtx: ExecutionContext): Promise<Response> {
    const ctx = contextFrom(request, Number(env.TRACE_SAMPLE_RATE ?? "0.05"));
    const requestId = request.headers.get("x-request-id") ?? crypto.randomUUID();

    const url = new URL(request.url);
    url.hostname = "origin.example.com";              // a rewrite, not a new trace
    const upstream = withTrace(new Request(url, request), ctx, requestId);

    const response = await fetch(upstream);

    // capture the trace fields *before* handing work to waitUntil
    const { traceId, spanId, sampled } = ctx;
    exeCtx.waitUntil(
      recordSpan(env, { traceId, spanId, sampled, status: response.status })
    );

    const out = new Response(response.body, response);
    out.headers.set("x-request-id", requestId);
    out.headers.set("traceresponse", `00-${traceId}-${spanId}-${sampled ? "01" : "00"}`);
    return out;
  },
};

Note that the rewrite reuses the same ctx. A URL or path rewrite at the edge changes where the request goes, not which request it is — minting a new trace there is the single most common way a trace gets cut in half.

What changes at each hop Four hops in a row, each showing the trace context values it carries, illustrating that the trace id is constant while the span id changes at every hop. Client browser or SDK Edge Worker first hop, PoP Origin fetch after rewrite Origin app logs the join key no traceparent sent trace 4bf9…4736 span 00f0…02b7 trace 4bf9…4736 span a1b2…c3d4 trace 4bf9…4736 joins on this Trace id constant · span id new at every hop · sampled flag decided at hop 2 and never revisited A hop that mints a new trace id instead of a new span id silently splits the request into two traces.

Step 3 — The same design in Vercel middleware

Vercel’s middleware runs before routing and can rewrite request headers for the downstream function, which is the equivalent injection point. The shape is different but the rules are identical.

import { NextRequest, NextResponse } from "next/server";

export const config = { matcher: ["/((?!_next/static|favicon.ico).*)"] };

export function middleware(request: NextRequest) {
  const inbound = request.headers.get("traceparent");
  const traceId = inbound?.split("-")[1] ?? crypto.randomUUID().replace(/-/g, "");
  const spanId = crypto.randomUUID().replace(/-/g, "").slice(0, 16);
  const sampled = inbound ? inbound.split("-")[3] === "01" : Math.random() < 0.05;
  const requestId = request.headers.get("x-request-id") ?? crypto.randomUUID();

  const headers = new Headers(request.headers);
  headers.set("traceparent", `00-${traceId}-${spanId}-${sampled ? "01" : "00"}`);
  headers.set("x-request-id", requestId);

  const response = NextResponse.next({ request: { headers } });
  response.headers.set("x-request-id", requestId);
  return response;
}

NextResponse.next({ request: { headers } }) is the part that matters — passing headers on the request object is what makes them visible to the route handler. Setting them only on the response gives you an echo with nothing behind it. The same trap applies to any authentication performed in Vercel middleware, where the identity you resolved has to reach the handler the same way.

Step 4 — Keep the id out of the cache key

This deserves its own step because it is the bug that turns a tracing rollout into an incident. A correlation id is unique per request by definition. If it reaches the cache key — through a cache rule that includes all headers, a Vary: X-Request-Id emitted by a well-meaning origin, or an id appended to the URL as a query parameter — then every request becomes a unique object and your hit ratio goes to zero within one TTL.

The correlation id as a cache buster A before and after comparison showing two identical requests missing the cache when the request id is part of the key, and hitting it when the id is excluded. Before — id in the cache key GET /p?x=1 · X-Request-Id: a1 MISS → origin GET /p?x=1 · X-Request-Id: b2 MISS → origin again Hit ratio collapses to zero every request is a unique object After — id excluded from the key GET /p?x=1 · X-Request-Id: a1 MISS → origin GET /p?x=1 · X-Request-Id: b2 HIT at the edge Hit ratio unchanged the id rides in a header only The same rule applies to Vary — never name a correlation header there, and never put the id in the query string.

The defensive measures are small. Never place the id in the URL. Audit the cache key explicitly rather than assuming, using the technique in customizing cache keys to improve hit ratio, and strip any Vary value naming a correlation header on the response path before it is stored.

Step 5 — Log the same key at the origin

The join only works if both sides emit the field under a name you can predict. Emit structured JSON with the trace id as a first-class key, not buried in a message string.

{
  "ts": "2026-07-28T14:15:29.882Z",
  "level": "error",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "a1b2c3d4e5f60718",
  "parent_span_id": "00f067aa0ba902b7",
  "sampled": true,
  "request_id": "6f1a0c74-2b1d-4f0e-9c6a-3d21f8b7e409",
  "route": "/api/orders",
  "status": 502,
  "duration_ms": 4812,
  "msg": "upstream inventory service timed out"
}

With edge logs landing through the pipeline described in shipping edge logs with Logpush, the join is one query:

SELECT e.EdgeColoCode        AS pop,
       e.EdgeResponseStatus  AS edge_status,
       o.status              AS origin_status,
       o.duration_ms,
       o.msg
FROM   edge_logs  e
JOIN   origin_logs o USING (trace_id)
WHERE  e.EdgeResponseStatus >= 500
  AND  e.EdgeStartTimestamp >= now() - INTERVAL 1 HOUR
ORDER BY o.duration_ms DESC
LIMIT 50;
Joining two log streams on one key An edge log stream and an origin log stream converge on a shared trace id to produce a joined view containing both the edge outcome and the origin cause. Edge log trace_id, RayID, colo status, cache outcome Origin log trace_id, span_id duration_ms, msg Joined view which PoP saw the 502 and what the origin was doing at that moment join on trace_id Index trace_id, never group by it — one row per value makes it a lookup key, not a dimension. Aggregate on colo, route and status; use the trace id only to pull the single request back out.

Verification

Send a request with no trace context and confirm the edge minted one and echoed it:

curl -sv https://www.example.com/api/orders -o /dev/null 2>&1 \
  | grep -iE '^< (x-request-id|traceresponse)'
< x-request-id: 6f1a0c74-2b1d-4f0e-9c6a-3d21f8b7e409
< traceresponse: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Now supply your own context and confirm the trace id is adopted, not replaced, while the span id changes:

curl -sv https://www.example.com/api/orders \
  -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-a1b2c3d4e5f60718-01' \
  -o /dev/null 2>&1 | grep -i '^< traceresponse'

The returned trace id must match byte for byte; a different value means step 1 is minting instead of adopting. Finally, replay one traced request against the origin directly, bypassing the edge, to establish which layer owns a fault:

curl -s -o /dev/null -w '%{http_code} %{time_total}\n' \
  --resolve origin.example.com:443:203.0.113.20 \
  -H 'traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-b7c8d9e0f1a2b3c4-01' \
  https://origin.example.com/api/orders

Troubleshooting

Ids regenerated at each hop

Symptom: every log line has a trace id, and none of them match. Diagnose by echoing the inbound value at each layer — add a temporary x-debug-inbound-traceparent response header carrying what that hop actually received. The usual culprit is a framework middleware that unconditionally calls its id generator rather than checking for an existing header, or a new Request(url) built without spreading the original init, which silently discards every inbound header.

Duplicate ids from a client retry

A client that retries with the same X-Request-Id produces two rows that look like one request, so latency and error counts double-count. Treat X-Request-Id as an idempotency hint rather than a unique key: keep the retried value for correlation, but mint a fresh span id per attempt and record an attempt counter in tracestate. When counting requests, count distinct span ids, never distinct request ids.

Header stripped by an intermediate proxy

Corporate proxies, older load balancers and some managed gateways drop headers they do not recognise. Diagnose with a request through the suspect path echoing all received headers from a debug endpoint; if traceparent is present at the edge and absent at the origin, the hop between them is stripping it. Fixes, in order of preference: allowlist the header in the proxy config; fall back to X-Request-Id, which is far more widely allowed; or, as a last resort, mint at the origin and correlate on RayID instead.

Trace context lost across waitUntil

waitUntil runs after the response has been returned, and the request object it closes over may already have been recycled. Reading request.headers inside a deferred task therefore returns nothing or throws. Capture the trace fields into plain local variables before passing the promise to waitUntil, exactly as the Step 2 example does. The same rule applies to any background write, including the metric emission that a rate limiter at the edge performs after responding.

Cardinality blowing up the log index

Trace ids are unique per request, so indexing them as a searchable dimension creates one index entry per request and can dwarf the log data itself. Keep them as a high-cardinality lookup field: index for exact-match retrieval, exclude from any auto-discovered facet list, and never allow a dashboard to group by them. If your platform bills by indexed field cardinality, this single misconfiguration can outweigh every other logging cost you have.

Frequently Asked Questions

Do I need both traceparent and X-Request-Id? They serve different readers. traceparent is a fixed-format field that intermediaries and tracing backends parse automatically, while X-Request-Id is a short readable value a customer can quote in a support ticket and an on-call engineer can paste into a search. Carrying both costs about 90 bytes per request.

Where should the sampling decision be made? At the first hop that sees the request, and nowhere else. If each layer samples independently you get partial traces — an edge span with no origin span, or the reverse — which are worse than no trace at all, because they look complete until you try to use them.

What happens if a CDN strips unknown headers? The trace stops at that hop and every system past it starts a new one, so you get two disjoint traces for one request. Allowlist traceparent and tracestate in the intermediary’s configuration; if that is impossible, fall back to the more widely tolerated X-Request-Id and accept the loss of parent-child structure.

Can the correlation id break my cache? Yes, and it is the most expensive mistake in this design. A per-request unique value that reaches the cache key or a Vary header makes every object unique and drives the hit ratio to zero, so keep the id in a request header, out of the URL, and explicitly excluded from the cache key.

How do I replay one specific traced request? Take the trace id from the ticket or the log line, pull the full request shape out of your edge logs, then reissue it with the same traceparent and a fresh span id — once through the edge and once directly against the origin with --resolve. Comparing the two answers tells you immediately which layer owns the fault.

Back to Edge Observability & Debugging