Aggregating Microservice Responses at the Edge

After working through this guide you will be able to build a fan-out/fan-in endpoint inside a single Cloudflare Worker invocation: issue parallel subrequests to several internal services, bound each one with its own AbortSignal.timeout, collect the results with Promise.allSettled, reshape them into one client-facing payload, and return a useful partial response with a warnings array when an upstream misbehaves instead of collapsing the whole call into a 500.

Composition is the job a mobile client cannot do for itself. A product page needs catalog metadata, live pricing, and a review summary; done from the handset that is three TLS handshakes and three round trips over a lossy radio link. Done inside an edge API gateway, it is one client request, three warm subrequests over the provider’s backbone, and one response whose latency is set by the slowest upstream rather than the sum of all of them. The engineering work is entirely in the failure and timing behavior — a naive Promise.all composition is strictly worse than three separate calls, because any single upstream hiccup takes the whole page down.

Key implementation objectives:

• Fan out to every upstream at once and fan back in with Promise.allSettled, so one rejection never cancels the rest.
• Give each upstream its own deadline with AbortSignal.timeout and classify it as required or optional.
• Emit a partial 200 carrying a warnings array when an optional fragment is missing, and reserve 502 for genuinely required data.
• Choose a cache policy deliberately — per-fragment or per-composition — and stream the first bytes when a slow upstream would otherwise hold the response.

Prerequisites and environment setup

You need Node 18 or newer, Wrangler 3.60+, and a Cloudflare account. Composition itself works on the Free plan, but the subrequest ceiling is much lower there, so a production aggregator belongs on Workers Paid. Confirm the toolchain first:

node --version           # v18.x or later
npx wrangler --version   # 3.60.0 or later
npx wrangler whoami      # account + auth check

The upstream services must be reachable from the Worker over HTTPS — either public hostnames restricted by a shared secret, Cloudflare Tunnel hostnames, or service bindings to other Workers. There is no TCP socket and no connection pool at the edge, so every upstream call is an HTTP fetch().

{
  "name": "product-composer",
  "main": "src/index.ts",
  "compatibility_date": "2024-09-23",
  "vars": {
    "CATALOG_URL": "https://catalog.internal.example.com",
    "PRICING_URL": "https://pricing.internal.example.com",
    "REVIEWS_URL": "https://reviews.internal.example.com",
    "UPSTREAM_BUDGET_MS": "250"
  },
  "observability": { "enabled": true }
}

npx wrangler secret put UPSTREAM_SHARED_SECRET

Put this Worker behind the same auth layer you already run — validate the caller once with JWT validation at the edge, then forward the verified subject to each upstream as a header so no service re-parses the token. Because one client call now costs three upstream calls, keep the aggregator behind rate limiting as well; an unthrottled composed endpoint is a traffic multiplier pointed at your own backends.

Budget the latency before writing code

Composition only pays off when the fan-out is genuinely parallel. Sequential chaining — fetch the catalog, then use its ID to fetch pricing, then fetch reviews — adds every hop together and is almost always the result of an accidental await inside a loop rather than a real data dependency. Measure both shapes before you commit to a budget.

Write the budget down as a table and treat the total as a service-level objective, not an aspiration. The numbers below are from a three-upstream product endpoint with the services in the same region and Argo smart routing enabled between the PoP and the origin region.

The p99 column is the one that matters. Without a deadline, the composed p99 is unbounded: it becomes whatever the worst upstream does on its worst day. AbortSignal.timeout converts that open-ended tail into a fixed, known cost.

Step-by-step procedure

Step 1 — Declare the upstream map with per-upstream deadlines

Describe each upstream declaratively: where it lives, how long it may take, and whether the response is usable without it. A required fragment that fails produces an error; an optional one produces a warning. Encoding this in data rather than in branching code keeps the merge step honest as the endpoint grows.

interface Env {
  CATALOG_URL: string;
  PRICING_URL: string;
  REVIEWS_URL: string;
  UPSTREAM_BUDGET_MS: string;
  UPSTREAM_SHARED_SECRET: string;
}

interface Upstream {
  name: string;
  url: (id: string, env: Env) => string;
  timeoutMs: number;
  required: boolean;
}

const UPSTREAMS: Upstream[] = [
  { name: "catalog", url: (id, e) => `${e.CATALOG_URL}/items/${id}`,      timeoutMs: 150, required: true  },
  { name: "pricing", url: (id, e) => `${e.PRICING_URL}/price/${id}`,      timeoutMs: 250, required: true  },
  { name: "reviews", url: (id, e) => `${e.REVIEWS_URL}/summary/${id}`,    timeoutMs: 250, required: false },
];

Set each deadline slightly above that upstream’s own p99, not at the composed budget. A deadline generous enough to never fire is decoration; one tighter than the upstream’s normal tail turns a healthy service into a permanent warning.

Step 2 — Fan out with AbortSignal.timeout and fan in with Promise.allSettled

The critical detail is that every fetch() is started before the first await. Calling fetch returns a promise immediately, so building the array of promises fires all three requests inside the same tick; the single await Promise.allSettled(...) then waits once for the slowest. allSettled never short-circuits, so an aborted or rejected upstream still lets the others land.

type Fragment =
  | { name: string; ok: true;  data: unknown; ms: number }
  | { name: string; ok: false; error: string; ms: number };

async function callUpstream(u: Upstream, id: string, env: Env, req: Request): Promise<Fragment> {
  const started = Date.now();
  try {
    const res = await fetch(u.url(id, env), {
      // one deadline per upstream, enforced by the runtime
      signal: AbortSignal.timeout(u.timeoutMs),
      headers: {
        "x-upstream-secret": env.UPSTREAM_SHARED_SECRET,
        "x-auth-sub": req.headers.get("x-auth-sub") ?? "",
        "x-request-id": req.headers.get("x-request-id") ?? crypto.randomUUID(),
        accept: "application/json",
      },
      cf: { cacheTtl: 30, cacheEverything: true },
    });
    if (!res.ok) {
      return { name: u.name, ok: false, error: `upstream_status_${res.status}`, ms: Date.now() - started };
    }
    return { name: u.name, ok: true, data: await res.json(), ms: Date.now() - started };
  } catch (err) {
    const aborted = (err as Error).name === "TimeoutError" || (err as Error).name === "AbortError";
    return {
      name: u.name,
      ok: false,
      error: aborted ? "upstream_timeout" : "upstream_unreachable",
      ms: Date.now() - started,
    };
  }
}

async function fanOut(id: string, env: Env, req: Request): Promise<Fragment[]> {
  // every fetch is in flight before the first await
  const settled = await Promise.allSettled(UPSTREAMS.map((u) => callUpstream(u, id, env, req)));
  return settled.map((s, i) =>
    s.status === "fulfilled"
      ? s.value
      : { name: UPSTREAMS[i].name, ok: false as const, error: "handler_threw", ms: 0 },
  );
}

callUpstream already converts every failure into a resolved Fragment, so in practice allSettled and all would behave identically here. Keep allSettled anyway: it is the safety net for a bug in your own mapping code, and it costs nothing.

Step 3 — Degrade on purpose: partial bodies and a warnings array

Now decide what each outcome means. A missing optional fragment should not cost the client its whole page — return 200 with that key absent and a machine-readable entry in warnings. A missing required fragment is a real failure, and 502 with Retry-After is the honest answer.

function compose(fragments: Fragment[]): { status: number; body: Record<string, unknown> } {
  const byName = new Map(fragments.map((f) => [f.name, f]));
  const warnings = fragments
    .filter((f): f is Extract<Fragment, { ok: false }> => !f.ok)
    .map((f) => ({ upstream: f.name, code: f.error, elapsed_ms: f.ms }));

  const missingRequired = UPSTREAMS.filter((u) => u.required && !byName.get(u.name)?.ok);
  if (missingRequired.length > 0) {
    return {
      status: 502,
      body: { error: "upstream_unavailable", upstreams: missingRequired.map((u) => u.name), warnings },
    };
  }

  const catalog = byName.get("catalog") as Extract<Fragment, { ok: true }>;
  const pricing = byName.get("pricing") as Extract<Fragment, { ok: true }>;
  const reviews = byName.get("reviews");

  // reshape: the client sees one flat product, not three service payloads
  const c = catalog.data as any;
  const p = pricing.data as any;
  const body: Record<string, unknown> = {
    id: c.sku,
    title: c.display_name,
    description: c.long_description,
    price: { amount: p.amount_minor, currency: p.currency, valid_until: p.expires_at },
    reviews: reviews?.ok ? { score: (reviews.data as any).avg, count: (reviews.data as any).n } : null,
  };
  if (warnings.length > 0) body.warnings = warnings;
  return { status: 200, body };
}

Reshaping is not cosmetic. Returning the three upstream payloads verbatim under three keys leaks your internal service boundaries into the public contract, and every later refactor becomes a breaking API change. Flatten once, at the edge.

Step 4 — Choose a cache placement for the composition

There are two defensible placements and one common mistake. Cache each fragment independently (each subrequest carries its own cf.cacheTtl, so pricing can hold for 30 seconds while catalog holds for an hour) and the composition is rebuilt per request from mostly-warm parts. Or cache the composed response itself under a single key, which gives one clean hit but forces the shortest-lived fragment’s TTL onto the whole document.

The mistake is caching the composition without accounting for partial responses. A body that carries warnings describes a degraded moment, and storing it means every later visitor is served that outage until the TTL expires. Set no-store whenever warnings is non-empty, and only apply a shared TTL to complete responses. If the shape of your Cache-Control header is unfamiliar, the reasoning is laid out in setting Cache-Control headers for static and dynamic content; pairing a short s-maxage with stale-while-revalidate keeps a composed page fast even while one upstream is slow.

function cacheHeaders(status: number, hasWarnings: boolean): HeadersInit {
  if (status !== 200 || hasWarnings) {
    return { "cache-control": "no-store", "x-composed": "partial" };
  }
  return {
    "cache-control": "public, s-maxage=30, stale-while-revalidate=120",
    "x-composed": "complete",
  };
}

Step 5 — Stream the first bytes with TransformStream

When one fragment is slow but the rest are ready, buffering the whole document means the client waits for the laggard before it sees anything. A TransformStream lets you write the fast keys immediately and append the slow one when it lands, so time to first byte tracks the fastest upstream instead of the slowest. This changes the wire contract, so gate it behind a query flag or a content type your client opts into.

function streamCompose(id: string, env: Env, req: Request): Response {
  const { readable, writable } = new TransformStream();
  const writer = writable.getWriter();
  const enc = new TextEncoder();

  (async () => {
    try {
      await writer.write(enc.encode('{"id":' + JSON.stringify(id) + ',"fragments":['));
      let first = true;
      // resolve each upstream independently and emit it the moment it lands
      const jobs = UPSTREAMS.map((u) =>
        callUpstream(u, id, env, req).then(async (f) => {
          await writer.write(enc.encode((first ? "" : ",") + JSON.stringify(f)));
          first = false;
        }),
      );
      await Promise.allSettled(jobs);
      await writer.write(enc.encode("]}"));
    } finally {
      await writer.close();
    }
  })();

  return new Response(readable, {
    headers: { "content-type": "application/json", "transfer-encoding": "chunked", "cache-control": "no-store" },
  });
}

Two constraints come with streaming. A streamed body cannot be cached as one unit, so it is no-store by definition, and you cannot change the status code once the first byte is written — a late failure has to be reported inside the payload rather than as a 502.

Subrequest limits, CPU budget, and avoiding N+1 fan-out

Each fetch() from a Worker consumes one subrequest. The Free plan allows 50 per invocation and Workers Paid allows 1000, which sounds generous until a list endpoint loops over 200 items and calls pricing per item — that is the N+1 shape, and it will exhaust the budget, saturate the upstream, and blow the CPU limit in one go. The fix is a batch endpoint upstream: one POST /price/batch with 200 IDs replaces 200 subrequests with one.

CPU time is a separate budget from wall time. Waiting on fetch() costs no CPU, so a 250 ms fan-out is nearly free; JSON.parse and JSON.stringify over large payloads are not. Trim upstream responses with sparse fieldsets (?fields=sku,display_name) rather than parsing megabytes at the edge and discarding most of it.

Keep fan-out width bounded in code as well as by policy — a hard cap that returns 400 above, say, 20 IDs is far kinder than discovering the ceiling in production.

Verification

Confirm the composed timing with curl’s timing variables. time_starttransfer is the number to watch: on a buffered composition it tracks the slowest upstream, and on the streaming variant it should collapse toward the fastest.

curl -s -o /dev/null \
  -w "dns=%{time_namelookup} connect=%{time_connect} ttfb=%{time_starttransfer} total=%{time_total}\n" \
  -H "Authorization: Bearer $TOKEN" \
  https://api.example.com/v1/product/42

dns=0.014 connect=0.031 ttfb=0.128 total=0.131

Inspect the body and headers together to confirm the degradation contract holds while one upstream is down:

curl -s -D - -H "Authorization: Bearer $TOKEN" \
  https://api.example.com/v1/product/42 | head -20

HTTP/2 200
cache-control: no-store
x-composed: partial
{"id":"42","title":"Field Notebook","price":{"amount":1290,"currency":"USD"},
 "reviews":null,"warnings":[{"upstream":"reviews","code":"upstream_timeout","elapsed_ms":250}]}

Watch which upstream is actually slowest across real traffic rather than guessing:

npx wrangler tail --format json | jq -r '.logs[].message[] | select(type=="string")'

compose id=42 catalog=41ms pricing=88ms reviews=250ms(timeout) total=96ms
compose id=17 catalog=38ms pricing=201ms reviews=64ms total=209ms

Log the per-upstream elapsed time on every request, not just failures — the slow upstream that never times out is the one quietly setting your p99. Ship those lines off the platform so you can chart the distribution per upstream over weeks, and propagate the same x-request-id you set in Step 2 using the patterns in tracing requests across the edge with correlation headers.

Troubleshooting

One slow upstream drags every response to its deadline

If composed p99 sits exactly on your longest timeoutMs, one upstream is timing out routinely rather than occasionally. Confirm with a per-upstream histogram from the tail logs above. The fix is rarely a longer deadline: either the upstream needs a cache in front of it, or the fragment should stop being fetched inline and move to a second client request. A deadline that fires on more than about 1% of calls is a capacity problem wearing a timeout costume.

Too many subrequests under load

The error surfaces as an exception on the fetch() call once the invocation crosses the plan’s subrequest ceiling. Diagnose by counting calls per invocation — log UPSTREAMS.length * ids.length before fan-out. Almost always the cause is a loop that fans out per item. Replace it with a batch upstream call, cap the accepted list length, and remember that a redirect followed by the runtime counts as an additional subrequest.

Aborted requests still charge the upstream

AbortSignal.timeout stops your Worker waiting, but the upstream may have already started expensive work and will finish it regardless. Under a partial outage you can therefore time out at the edge while still driving the failing service into the ground. Pair the client deadline with a server-side deadline (a x-deadline-ms header the upstream honors) and, for repeat failures, a circuit breaker keyed in a Durable Object that skips the call entirely for a cooling-off window.

The composed response is cached in a degraded state

A warnings payload served from cache long after recovery is the classic composition bug. Check x-composed: partial against cf-cache-status on a response you know should be complete; if you see HIT with partial, your cache decision ran before the warnings were known. Compute Cache-Control from the final composed body, never from the request, and purge the affected paths after an incident.

TypeError: Illegal invocation or a hung stream

Writing to a TransformStream writer after close(), or forgetting await writer.close() in the finally, leaves the response hanging until the client gives up. Reproduce locally with npx wrangler dev and a curl --max-time 5; a body that stops mid-JSON confirms it. Always close in a finally, and never call writer.write from two branches without awaiting the previous write.

Frequently Asked Questions

Why use Promise.allSettled instead of Promise.all? Promise.all rejects as soon as any subrequest rejects, discarding the results that already succeeded and forcing a 500 for a single flaky upstream. allSettled waits for every promise to settle and hands you the outcome of each, which is what partial degradation requires.

Does an aborted subrequest still count against the subrequest limit? Yes. The subrequest is counted when it is issued, not when it completes, so a fan-out that times out consumes exactly the same budget as one that succeeds. Bound your fan-out width in code rather than relying on timeouts to keep the count down.

Should I cache each fragment or the composed response? Cache fragments when their TTLs differ widely or when the same fragment is reused by several composed endpoints, because each piece keeps its natural lifetime. Cache the composition when all fragments share a similar TTL and the endpoint is hot, since one key then serves the whole document with a single lookup.

How do I keep a partial response out of the shared cache? Compute the Cache-Control header from the composed body: emit no-store whenever the warnings array is non-empty or the status is not 200, and only attach s-maxage to complete responses. That single rule prevents a momentary upstream failure from being replayed to everyone for the length of the TTL.

When is streaming with TransformStream worth the complexity? When one fragment is reliably slower than the others and the client can render progressively — a product page that can paint the title and price before reviews arrive. If the client parses the whole document before rendering anything, streaming adds complexity, forfeits caching, and buys nothing.

Related

• Rate Limiting API Requests at the Edge
• JWT Validation at the Edge with Cloudflare Workers
• Edge API Gateway versus Regional API Gateway
• Tracing Requests Across the Edge with Correlation Headers

Back to API Gateway at the Edge