One oversized file, one global outage: what the November 2025 Cloudflare incident says about configuration

On 18 November 2025 a large part of the web returned error pages for hours. The cause, according to Cloudflare's own account, was not an attack or a hardware failure. A routine change to database permissions made a generated file twice as large as expected, and the software that read it could not cope. If your reports depend on generated files, definitions or settings, this one is for you.

Trinetro Labs · Published 25 October 2026 · 4 min read

What happened

Cloudflare sits in front of a very large share of websites and services, so when it failed, sites including X and ChatGPT showed errors. In its own account of the incident, Cloudflare explained that a change to permissions in one of its ClickHouse database clusters caused a query to return duplicate rows. That query builds a feature file used by its Bot Management system, and the duplicates made the file roughly double the size it should have been. The software that reads the file had a fixed limit on how many features it would accept. The oversized file went past that limit and the software crashed. The file is regenerated every few minutes and pushed across the network, so for a while parts of the network flipped between working and failing as good and bad versions arrived. Cloudflare said the problem was not caused by malicious activity. The failure began at about 11:20 UTC, most traffic was flowing normally again by mid-afternoon, and full recovery was announced at 17:06 UTC.

Why a small change looked safe

The change was housekeeping: it altered who could see which tables. Nobody editing permissions expects it to change the output of an unrelated query. But the query had been written on the assumption that it would only ever see one set of tables, and once it saw more, it quietly returned more rows. The code reading the result trusted the size of its input. This is the pattern worth remembering. The change and the failure were in different systems, written by different people, linked only by a file in the middle.

Three habits that would have limited the damage

• Validate what you generate, not only what you write. A file produced by a job deserves the same checks as one produced by a person: expected size, expected number of rows, expected shape. A file twice the normal size is a signal, not a payload.
• Fail soft on limits. A limit that crashes the program turns a bad input into an outage. A limit that rejects the new file and keeps using the last good one turns it into a log line.
• Roll out in stages, with a way back. Generated changes that reach every machine within minutes need a quick, rehearsed way to stop and revert.

What it means for data teams

Analytics stacks run on generated files too: exported schemas, metric definitions, lookup tables, scheduled extracts, permission lists. A change to who can read a table can alter what a report shows, and a changed definition of "revenue" can move a number with no error anywhere. Two habits help. First, treat definitions as versioned things. In our engine, metric definitions are versioned and immutable, so a change is a new version rather than an edit in place, and an answer can be traced to the version that produced it. Second, compare before and after: when a permission or definition changes, run the questions that matter most and check that the answers still match. The semantic layer explainer covers why a single, versioned set of definitions makes that check possible.

Keep reading

• A year on from the AWS outage: what a DNS fault taught data teams about hidden dependencies
• How to read a cloud incident report: eight questions for finance and data leaders
• What Is a Semantic Layer? A Plain-English Guide for Business Teams
• How Trinetro Guards Your Data: System Invariants and Data Governance

Start free for 30 days · Book a demo · Try the guided demo