Chapter 4 — The Data Problem

What goes in

We ended the last part of this book with a sentence that was also a promise: the difficulty of aligning these machines begins exactly where the mirror does — in the data, which is to say, in us. This chapter makes good on that promise by looking, as concretely as the evidence allows, at what actually goes into a language model and what that input does to what comes out. It is the least glamorous chapter in the book and, for exactly that reason, one of the most important. The dramatic worries about artificial intelligence — the runaway superintelligence, the machine that turns on its makers — are downstream of a problem so mundane it is easy to overlook: these systems are made of what we feed them, and what we feed them is a mess.

I want to handle this chapter with particular care, because the data problem is where AI commentary is at its most overheated in both directions. One camp treats every model output it dislikes as proof of a corrupt and dangerous technology; the other waves away documented harms as teething troubles that scale will fix. Neither is honest. What the evidence actually shows is narrower, stranger, and more useful than either story, and I am going to try to give it to you straight — with the same discipline of marking what is established, what is argued, and what is still being worked out.

There are three things that go wrong with the data, and they form a natural sequence. The first is bias: the model inherits the slants and prejudices latent in human text. The second is contamination: the corpus is polluted in ways that corrupt what the model learns. The third — the newest and, in some ways, the most unsettling — is collapse: what happens when models begin to train on the output of other models, and the mirror starts reflecting itself.

Bias: the corpus, faithfully reproduced

We have, in a sense, already proved that language models will be biased. It was the burden of Chapter 1: a system optimized to predict human text must absorb the statistical structure of human text, and that structure includes every prejudice that left a residue in the written record. A mirror does not editorialize. So the existence of bias in these systems is not a surprising empirical discovery; it is a structural certainty, predictable from the mechanism alone. The only open questions are how it manifests, how severe it is, and what — if anything — can be done about it.

The manifestations are by now well documented, and the honest move is to point to the documentation rather than to gesture at the problem in the abstract. [VERIFIED — algorithmic and dataset bias in language models is extensively documented in the peer-reviewed fairness literature; cite specific representative studies in the verification pass, e.g. work on gender and occupational association in embeddings, and demographic disparities in model outputs.] The patterns that recur: occupational stereotypes, where the model associates professions with the gender or ethnicity that dominates the training text; representational harms, where some groups are described in systematically narrower or more negative terms than others; and disparities in performance, where the system simply works less well for the people who were underrepresented in its corpus. [VERIFIED — these three categories — stereotyping, representational harm, and performance disparity — are standard, documented categories in the algorithmic-fairness literature. Verify and cite a representative study for each.]

Notice what all three have in common. None of them is the machine inventing a prejudice of its own. Each is the machine reproducing, with the fidelity of a well-fitted statistical model, a pattern that was already present in what we wrote. The occupational stereotype is in the corpus because it is in our writing about occupations. The model did not author the bias; it inherited it, the way it inherited grammar — and, crucially, by the same mechanism, which is why it cannot simply be told to stop. This is the first hard lesson of the data problem: you cannot instruct a mirror to be fairer than the thing it reflects.

And yet — here the discipline of giving both halves matters — it would be wrong to conclude that nothing can be done. A mirror cannot un-see what is in front of it, but you can change what is in front of it, and you can polish the glass. There is real, ongoing work on reducing these harms: curating training data to be more representative, fine-tuning models against documented biases, building evaluation suites that measure disparity so it can at least be tracked. [VERIFIED — bias mitigation via data curation, fine-tuning, and evaluation benchmarks is an active and partially effective area of practice. Verify current state; do not overstate efficacy.] These methods help. They do not solve the problem, because the underlying corpus remains what it is, and because — as we will see in the alignment chapter — "less biased" requires a standard of fairness that we ourselves have never fully agreed on. But the situation is not hopeless, and a book that left you thinking bias is an unalterable doom would be as dishonest as one that pretended it was already fixed.

Contamination: when the corpus is poisoned

Bias is what the corpus contains because we are biased. Contamination is something narrower and more deliberate: the ways a training corpus can be polluted, accidentally or intentionally, so that the model learns things its makers never intended.

The accidental form is simply the internet being the internet. The web is not a clean library; it is a vast, unsorted heap containing spam, scams, propaganda, conspiracy theories, and every variety of confident falsehood, all of it written in fluent prose that the model has no way to distinguish from careful truth. We met this already as the root of hallucination-in-waiting: to the optimizer, a well-written falsehood and a well-written fact are the same kind of object — plausible text — and both exert their pull on the model's parameters. [VERIFIED — training corpora scraped from the open web contain large quantities of low-quality, false, and adversarial content; this is well established.] A model trained on the open web does not just learn our knowledge. It learns our credulity, our conspiracy theories, and our lies, because they are written in the same language as everything else.

The deliberate form is more troubling, and it has a name: data poisoning. Because models learn from text scraped at scale, an adversary who can plant content where it will be scraped can, in principle, influence what the model learns — seeding it with particular associations, backdoors, or failure modes. [VERIFIED — data poisoning is a documented and actively researched class of attack on machine-learning systems; verify the current state of demonstrated real-world poisoning of large-model training corpora specifically, and characterize precisely how feasible it currently is rather than overstating.] I want to be careful here, because this is exactly the kind of claim that gets inflated into a thriller plot. The honest statement is that data poisoning is a real and studied vulnerability, that its feasibility against the largest models depends on details of how their corpora are assembled and filtered, and that it represents a genuine security concern without being, today, a demonstrated mechanism of widespread catastrophe. Mark it as a real risk under active study, not as a present-tense disaster.

Both forms of contamination point at the same defensive idea, and it is one the field has converged on independently: that the provenance and quality of training data matter enormously, and that the era of indiscriminately scraping the whole web and hoping for the best is giving way to far more careful curation. [VERIFIED — the trend toward curated, filtered, and provenance-aware training data is real and current; verify specifics.] This is not a moral preference. It is an engineering response to documented failure. And it sets up the third and strangest part of the data problem, the one Chapter 2 promised we would return to.

Collapse: the mirror reflecting itself

Recall the quiet danger named at the end of the thermodynamics chapter. When thinking becomes cheap, the world fills not only with more good thinking but with an ocean of automatically generated text — plausible, fluent, and produced simply because it can be. I said then that the energy story and the quality story were the same story seen from two sides. Here is the other side.

For the entire history of language models until very recently, one assumption held: the training corpus was human. The text scraped from the web was written by people. That assumption is now breaking, and breaking fast, because the web is filling with text written by models. And this raises a question that would have been purely hypothetical a few years ago and is now urgently practical: what happens when a model is trained, in significant part, on the output of other models? What happens when the mirror is aimed at a wall of other mirrors?

The answer, established in a striking line of recent research, is degradation — a phenomenon now generally called model collapse. [VERIFIED — model collapse is a real, recent, peer-reviewed finding; verify and cite the primary source, e.g. the 2024 work demonstrating degradation in models trained recursively on generated data, and characterize the result precisely.] When models are trained recursively on data generated by previous models, they progressively lose information — and they lose it in a characteristic, revealing way. The tails of the distribution go first: the rare events, the unusual cases, the minority patterns, the long tail of human expression that appears infrequently in the data. Each generation of model, trained on the slightly-flattened output of the last, captures a little less of the variety of the original, until the output converges toward a bland, repetitive, increasingly homogeneous core. The model forgets the edges of human expression and remembers only the center, more and more narrowly, generation by generation. [VERIFIED — the loss of distributional tails (rare/minority data) is the documented signature of model collapse; verify the precise mechanism and wording against the primary literature.]

There is something almost poignant about this failure mode, and it is worth sitting with, because it sharpens the book's central image rather than decorating it. A mirror reflecting a mirror does not produce infinite depth. It produces a corridor that dims and degrades with each reflection, the image growing greener and grainier until it fades. A model trained on model output is exactly this: a reflection of a reflection, losing fidelity at every step. The richness of the original — the strangeness, the outliers, the rare and the surprising, which is to say much of what is most human about human expression — is precisely what erodes first. What survives is the average, and then the average of the average.

This is why model collapse is not merely a technical curiosity but a fact with weight. It tells us that the human-ness of the training data is not incidental to these systems; it is the irreplaceable resource on which they depend. A model is only ever as rich as the distribution it learned from, and a distribution made of model output is a photocopy of a photocopy. The implication is direct and, I think, important: the genuine human record — written by people, with all its variety and its outliers intact — is a finite and now actively threatened input, and preserving access to it matters for the health of the very systems that are busy polluting it.

The finite well

That word — finite — deserves a section of its own, because it names a constraint that sharpens everything else in this chapter and that the scaling era spent years not needing to think about.

For the first decade of large-scale language modeling, the corpus felt effectively infinite. The web was vast, most of it had never been used for training, and the operative question was how much compute you could afford, not how much text existed. That era is ending. The frontier systems have now consumed a large fraction of the high-quality public text that exists — the books, the articles, the carefully written web — and published analyses project that, at current rates of consumption, the stock of high-quality human-written public text will be effectively exhausted as a source of new training data within roughly this decade. [SOURCED — published projections of training-data exhaustion estimate that the supply of high-quality public human text will be effectively consumed by frontier training within approximately the 2020s; verify the specific analyses, their assumptions, and their current status in the verification pass, as estimates vary and the field contests the details.] The precise year is contestable and contested; the direction is not. The corpus is a well, the well is finite, and the buckets have been getting exponentially larger.

Watch what this constraint does when it meets the other findings of this chapter, because the collision is where the real story is. A field running short of genuine human text faces an obvious temptation: generate more. Synthetic data — text produced by models to train other models — is cheap, unlimited, and exactly the recursive loop that model collapse warns about. The shortage, in other words, pushes the field directly toward the failure mode we just documented, and much current research is an attempt to thread that needle — to use synthetic data in careful, filtered, bounded ways that capture its volume without inheriting its degradation. [VERIFIED — the tension between data scarcity and collapse risk, and active research on safe synthetic-data use, are real and current; verify the state of the art in the verification pass.] Meanwhile, the same shortage is repricing the genuine article. Text that was scraped for free is becoming text that is licensed for money; publishers, archives, and platforms that hold large reserves of verified human writing are discovering that they are sitting on what this new economy considers a strategic resource. [SOURCED — licensing agreements between AI developers and publishers/platforms for training data are a documented and growing practice; verify representative examples in the verification pass.]

Step back and see the shape of it. The human record spent the first era of AI as free exhaust — a byproduct lying around to be scraped. It is ending that era as something closer to an aquifer: finite, depletable, contaminated in places by the very industry that draws on it, and rising in value precisely as its limits come into view. That reframing is not sentimental. It is the economics catching up with what the mechanism implied all along — that these systems have exactly one irreplaceable input, and it is us, writing. [INTERPRETATION — the "free exhaust to finite aquifer" framing is mine; the underlying scarcity, licensing, and collapse dynamics are documented above.]

The prescription, earned

From these three documented problems — inherited bias, corpus contamination, and recursive collapse — a single practical conclusion follows, and I want to state it carefully because it is the kind of conclusion that is easy to inflate into a slogan.

What the data problem teaches is that the inputs to these systems are not interchangeable, and that their quality, provenance, and diversity are load-bearing. Indiscriminate scraping produces biased, contaminated, and — as the corpus fills with synthetic text — collapsing models. The defensive response, which the field is adopting not on principle but in response to measured failure, is deliberate curation: training data that is checked for quality, traced for provenance, weighted for representativeness, and protected against the closing loop of model-on-model training. [INTERPRETATION — the synthesis of these three failure modes into a single "curation matters" prescription is my framing; the individual failure modes are documented, the unifying conclusion is argued.]

I have heard this idea dressed up in grander language — talk of "data sovereignty," of "heirloom data," of curating one's inputs as a quasi-spiritual discipline. I am going to resist that register here, because the plainer version is both truer and more persuasive. You do not need a metaphor to justify caring about what goes into these systems. You need only the three findings of this chapter: that models reproduce the bias of their corpus, that they absorb its contamination, and that they collapse when the corpus becomes their own reflection. Those are reasons enough. The case for treating training data as something precious and worth protecting rests on documented mechanism, not on analogy — and it is stronger for it.

Where this leaves us

As ever, let me separate the established from the argued.

It is established that language models reproduce the biases present in their training corpora, in documented and categorizable ways; that training corpora scraped from the open web contain large quantities of false, low-quality, and adversarial content, and that deliberate data poisoning is a real and studied vulnerability; and that model collapse — the progressive degradation of models trained recursively on generated data, with the rare tails of the distribution eroding first — is a genuine, recently demonstrated phenomenon.

It is argued, as the natural synthesis rather than a separate finding, that these three failures share a single remedy: that the quality, provenance, and human-ness of training data are not incidental but load-bearing, and that careful curation is the rational response to documented degradation rather than a matter of taste.

And it is worth marking, as we turn the page, what this chapter has quietly established about the book's larger argument. We have been treating the data problem as a problem about machines. But every one of its three failures traces back to us: the bias is our bias, the contamination is our pollution, and even the collapse is a consequence of our flooding the commons with cheap reflection. The mirror's troubles are, on inspection, our troubles, handed to a machine that cannot help but reproduce them. That pattern — the problem that looks technical and turns out to be human — is about to become the whole subject. For we now arrive at the hardest problem in the field, the one toward which this book has been building from its first page: not how to clean the data, but how to tell the machine what we actually want. And we will find that the difficulty there is not, in the end, a difficulty about machines at all.