Shipping Edge Logs with Logpush

Q: Does Logpush replace wrangler tail?

No — they answer different questions. wrangler tail shows you live invocations with sub-second latency but keeps nothing, while Logpush lands complete batches on disk 30 seconds to 5 minutes later and keeps them for as long as your retention policy says. Use tail while deploying and Logpush when investigating.

Q: Why is my batch cadence irregular?

Batches are cut by whichever of max_upload_bytes, max_upload_records or max_upload_interval_seconds triggers first, so a traffic spike produces several files in one second while a quiet period produces one file per interval. Judge health by last_complete advancing, never by a steady file count.

After working through this guide you will be able to stand up a production Logpush pipeline that writes structured edge logs to object storage or an HTTPS collector, carries only the fields you actually query, survives the destination ownership handshake, and lands in a query engine that can answer an operational question in seconds. You will also know what each knob costs you in bytes and in retention spend.

Logpush is the batch half of edge observability. It does not replace live streaming with wrangler tail — that is for watching a deploy in the moment. Logpush is what you reach for at 03:00 when someone asks which point of presence started returning 502 an hour ago, and the answer has to come from data that was already on disk before anybody noticed. The pipeline is deliberately dumb: the edge selects fields, batches records, compresses them, and drops a file. Everything clever happens downstream.

Key implementation objectives:

Pick the right dataset for the question you are answering, and keep the field list narrow enough that retention stays affordable.
Complete the ownership challenge so the destination proves you control it, then create the job with the API.
Tune batching, compression, filename layout, filters and sampling so files arrive at a usable cadence and size.
Load the output into a query engine and answer a concrete incident question from it.

Prerequisites and environment setup

You need a Cloudflare zone on a plan that includes Logpush (Enterprise for the full http_requests dataset; Workers Paid for workers_trace_events), an API token scoped with Logs: Edit plus Account Settings: Read, and a writable bucket. The examples use R2 because it removes egress from the equation, but S3 and GCS work identically apart from the destination string.

export CF_API_TOKEN="…"
export ZONE_ID="023e105f4ecef8ad9ca31a8372d0c353"
export ACCOUNT_ID="9a7806061c88ada191ed06f989cc3dac"

# confirm the token can see the zone and has Logs scope
curl -s -H "Authorization: Bearer $CF_API_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/datasets/http_requests/fields" \
  | jq 'keys | length'

That last call returns the number of available fields — currently well over 170 for http_requests. If it returns a 403, the token lacks the Logs scope; fix that before going further, because every later step fails the same way.

For the query side, install DuckDB 1.0 or newer and jq 1.7. DuckDB reads gzipped NDJSON straight out of a bucket, which means no ETL job stands between you and an answer.

Step-by-step procedure

Step 1 — Choose the dataset

http_requests is zone-scoped and records one row per HTTP request that the edge served: status, cache outcome, colo code, timings, TLS details. It answers questions about traffic. workers_trace_events is account-scoped and records one row per Worker invocation: Outcome, CPUTimeMs, WallTimeMs, and the array of console.log lines and exceptions your code produced. It answers questions about code.

They are not interchangeable. A Worker that throws still produces an http_requests row with EdgeResponseStatus: 500 and no clue why; the stack trace lives only in workers_trace_events. Most teams end up running both jobs, because correlating them is how you get from “the edge returned 500” to “the JWT parser threw on a malformed token”. Wiring a shared identifier into both is covered in tracing requests with correlation headers.

Step 2 — Pick the field set that earns its bytes

The default temptation is to select everything. Resist it. Every field is written on every record, forever, and the fields with the highest cardinality — user agent, full referer, raw header maps — are both the biggest and the least useful for aggregate questions. A thirteen-field record compresses to roughly 90 bytes; the everything-record is closer to 900.

The dropped column is also your privacy control. Logpush has no transform stage — whatever you name in field_names leaves the edge exactly as recorded — so redaction is a selection decision made before the job exists. Where you genuinely need a per-user key, hash it inside the Worker and emit the hash through console.log, so workers_trace_events carries the pseudonym and never the identifier.

Step 3 — Pass the ownership challenge

Before Cloudflare will write to a destination it makes you prove you can read from it. You post the destination string, Cloudflare drops a small challenge file into that location, and you echo the token back when creating the job.

curl -s -X POST \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/ownership" \
  -d '{
    "destination_conf": "r2://edge-logs/http/{DATE}?account-id=9a7806061c88ada191ed06f989cc3dac&access-key-id=AKIA_EXAMPLE&secret-access-key=SECRET_EXAMPLE"
  }' | jq .

The response names the file that was written:

{
  "result": { "filename": "http/20260728/ownership-challenge-9f3c1d2b.txt", "valid": true },
  "success": true, "errors": [], "messages": []
}

Read that object out of the bucket and keep its contents — a short opaque token, valid for about an hour.

Step 4 — Create the job with batching, compression and filenames

Now the real payload. Field selection, output shape, batching thresholds, the filter and the sample rate all live in one request.

{
  "name": "edge-http-5xx",
  "dataset": "http_requests",
  "enabled": true,
  "destination_conf": "r2://edge-logs/http/{DATE}?account-id=9a7806061c88ada191ed06f989cc3dac&access-key-id=AKIA_EXAMPLE&secret-access-key=SECRET_EXAMPLE",
  "ownership_challenge": "9f3c1d2b7a5e4408",
  "output_options": {
    "field_names": [
      "EdgeStartTimestamp", "RayID", "EdgeColoCode", "EdgeResponseStatus",
      "OriginResponseStatus", "OriginResponseDurationMs", "CacheCacheStatus",
      "ClientRequestHost", "ClientRequestPath", "ClientRequestMethod",
      "ClientCountry", "WorkerStatus", "EdgeResponseBytes"
    ],
    "output_type": "ndjson",
    "timestamp_format": "rfc3339",
    "record_delimiter": "\n",
    "sample_rate": 1.0,
    "CVE-2021-44228": true
  },
  "filter": "{\"where\":{\"or\":[{\"key\":\"EdgeResponseStatus\",\"operator\":\"gt\",\"value\":499},{\"key\":\"WorkerStatus\",\"operator\":\"neq\",\"value\":\"ok\"}]}}",
  "max_upload_bytes": 5000000,
  "max_upload_records": 100000,
  "max_upload_interval_seconds": 30,
  "frequency": "high"
}

Post it to POST /client/v4/zones/$ZONE_ID/logpush/jobs. Three details matter more than they look:

{DATE} in the destination path is expanded by Logpush into YYYYMMDD, which gives you a date-partitioned prefix your query engine can prune on. Without it every file lands in one flat prefix and a one-hour question scans a year of data.
Batching is a three-way OR: a file is cut when it hits max_upload_bytes, max_upload_records, or max_upload_interval_seconds, whichever comes first. Thirty seconds is a good default; five minutes produces fewer, larger, cheaper files but delays your incident answer by five minutes.
CVE-2021-44228: true escapes ${ sequences in the output so a log line cannot be interpreted as a lookup expression by a downstream parser. Leave it on.

Files arrive named for the window they cover, which is what makes them selectable without opening them:

edge-logs/http/20260728/20260728T141500Z_20260728T141530Z_a41f9c7b.log.gz
                        └── batch start ─┘└── batch end ──┘└─ unique ─┘

Compression is gzip and is not optional on object-storage destinations. For an HTTPS destination the same batches are posted as a gzipped body, and you control auth by embedding a header in the destination string:

https://logs.internal.example.com/ingest?header_Authorization=Bearer%20REDACTED&tags=service:edge

Step 5 — Manage the job as code

A job created by hand is a job nobody can review. The Terraform resource carries the same fields, which also makes the field list diffable — the single most useful property when the log bill jumps and nobody remembers who added RequestHeaders.

resource "cloudflare_logpush_job" "edge_http_5xx" {
  zone_id             = var.zone_id
  name                = "edge-http-5xx"
  dataset             = "http_requests"
  enabled             = true
  destination_conf    = "r2://edge-logs/http/{DATE}?account-id=${var.account_id}&access-key-id=${var.r2_access_key_id}&secret-access-key=${var.r2_secret_access_key}"
  ownership_challenge = var.ownership_challenge

  filter = jsonencode({
    where = { and = [{ key = "EdgeResponseStatus", operator = "gt", value = 499 }] }
  })

  max_upload_bytes            = 5000000
  max_upload_records          = 100000
  max_upload_interval_seconds = 30

  output_options {
    field_names      = var.logpush_fields
    output_type      = "ndjson"
    timestamp_format = "rfc3339"
    sample_rate      = 1.0
  }
}

Keep logpush_fields in a variable file next to the query definitions that consume them. The same discipline applies to any zone managed as code with Terraform: the state file is the record of what production actually looks like.

Step 6 — Filter and sample before the bytes exist

A filter drops records at the edge, so a filtered record costs nothing anywhere downstream. sample_rate keeps a random fraction — 0.01 is one percent — and is the right tool for a high-volume success stream you only need for ratios, never for individual lookups. Combine them: ship every error, sample the successes.

Retention is then a bucket lifecycle rule rather than a Logpush setting. Ninety days of the filtered stream at this volume is under 7 GB, which is small enough that nobody argues about it — and that is the point of doing the arithmetic before turning the job on.

Step 7 — Load it and answer a question

The concrete question: which PoP returned 5xx in the last hour? With jq, straight off the downloaded batch files:

aws s3 cp --recursive --endpoint-url "$R2_ENDPOINT" \
  "s3://edge-logs/http/20260728/" ./logs/ --exclude "*" --include "20260728T14*"

zcat ./logs/*.log.gz \
  | jq -r 'select(.EdgeResponseStatus >= 500)
           | [.EdgeColoCode, (.EdgeResponseStatus|tostring)] | @tsv' \
  | sort | uniq -c | sort -rn | head -10

   1841	FRA	502
    377	AMS	502
     46	SIN	503
      9	IAD	500

For anything larger than a spot check, point DuckDB at the prefix and let it prune partitions:

SELECT EdgeColoCode                                  AS pop,
       count(*)                                      AS errors,
       round(avg(OriginResponseDurationMs), 1)       AS avg_origin_ms,
       count(*) FILTER (WHERE CacheCacheStatus='dynamic') AS uncached
FROM read_ndjson_auto('logs/20260728/*.log.gz')
WHERE EdgeResponseStatus >= 500
  AND EdgeStartTimestamp >= now() - INTERVAL 1 HOUR
GROUP BY pop
ORDER BY errors DESC
LIMIT 10;

A single PoP dominating that list points at a regional origin path, not at your code — the same signal you would chase through edge health checks and automatic failover.

Verification

Confirm the job exists, is enabled, and has no recorded error:

curl -s -H "Authorization: Bearer $CF_API_TOKEN" \
  "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/logpush/jobs" \
  | jq '.result[] | {id, name, enabled, last_complete, last_error, error_message}'

{
  "id": 4118,
  "name": "edge-http-5xx",
  "enabled": true,
  "last_complete": "2026-07-28T14:15:30Z",
  "last_error": null,
  "error_message": null
}

last_complete moving forward on each poll is the single best health signal. Then confirm files are actually landing and that a record parses:

aws s3 ls --endpoint-url "$R2_ENDPOINT" "s3://edge-logs/http/20260728/" | tail -3
zcat ./logs/20260728T141500Z_*.log.gz | head -1 | jq .

Troubleshooting

Job created but no files appear

Almost always the filter, not the pipeline. A job filtered to EdgeResponseStatus > 499 on a healthy zone legitimately writes nothing. Prove the pipeline works by temporarily creating a second job with no filter and sample_rate: 0.001; if that produces files within a minute, the original job is correct and your zone simply has no errors. The other cause is a zone with genuinely low traffic and max_upload_interval_seconds at its maximum — nothing is wrong, the batch has just not filled.

Ownership challenge keeps failing

ownership_challenge must be the contents of the challenge file, not its filename, and the token expires in roughly an hour — a CI pipeline that requests the challenge in one stage and creates the job three stages later will fail every time. Request and consume it in the same step. If the API reports it cannot write the challenge at all, the credentials in destination_conf lack PutObject on that prefix, which is a bucket policy problem rather than a Logpush one.

Fields missing from the output

A field absent from every record was rejected at job-creation time or does not exist in that dataset — ScriptName belongs to workers_trace_events and is silently unavailable to http_requests. Re-read the live field list with the /datasets/{dataset}/fields endpoint rather than trusting a copied field list. A field present but always null usually means it does not apply to that traffic: OriginResponseDurationMs is null on a cache HIT because no origin fetch happened.

Timestamps and filenames disagree

Filenames carry the batch window in UTC; EdgeStartTimestamp carries the request time. A request served at 14:14:59 can land in the T141500Z file, so an hour-boundary query that selects files by name alone loses records at the edges. Always select one file window wider than you need and filter on the timestamp field inside the query. Set timestamp_format: "rfc3339" so string comparison and date parsing both behave; the unixnano format is compact but invites off-by-a-thousand bugs.

Silent drops when the destination rejects a batch

HTTPS destinations are the risky ones. If your collector returns anything other than 2xx, Logpush retries a bounded number of times and then drops the batch — the data is gone, and the only trace is last_error on the job. Poll that field into your monitoring, alert when last_complete stops advancing for more than three batch intervals, and prefer an object-storage destination for anything you must not lose. A 413 from the collector means max_upload_bytes exceeds its body limit; lower it to match rather than hoping the retries succeed.

Frequently Asked Questions

Does Logpush replace wrangler tail? No — they answer different questions. wrangler tail shows you live invocations with sub-second latency but keeps nothing, while Logpush lands complete batches on disk 30 seconds to 5 minutes later and keeps them for as long as your retention policy says. Use tail while deploying and Logpush when investigating.

How much does a field really cost? Cost scales with compressed bytes multiplied by records multiplied by retention days. Dropping ClientRequestUserAgent and the raw header maps typically cuts a record from roughly 900 bytes to 90, so the same retention window costs a tenth as much and every scan runs about ten times faster.

Can I redact personal data inside Logpush? There is no transform stage, so redaction is purely a field-selection decision made before the job is created. Omit ClientIP, cookies and raw header maps, use coarse substitutes such as ClientIPClass and ClientCountry, and if you need a per-user key, hash it in the Worker and emit the hash rather than the identifier.

Why is my batch cadence irregular? Batches are cut by whichever of max_upload_bytes, max_upload_records or max_upload_interval_seconds triggers first, so a traffic spike produces several files in one second while a quiet period produces one file per interval. Judge health by last_complete advancing, never by a steady file count.

Should I run one job or several? Several, split by dataset and by intent. A tightly filtered error job with a short batch interval gives fast incident answers, while a heavily sampled full-traffic job gives ratios cheaply — combining both goals into one job forces you to over-collect for one of them.

Back to Edge Observability & Debugging