Chapter 1 — The Learning Machine

What "training" actually is

Start with the word itself, because it misleads almost everyone. We say a model is "trained," and the word summons an image of instruction — a teacher transmitting knowledge to a student, rules being written down and handed over, a curriculum imparted. None of that is happening. There is no teacher, no curriculum, no transmission of rules. What actually happens during the training of a language model is closer to a vast, blind, automated process of trial and correction, repeated until something useful falls out the other end. The wonder of the thing — and it is a genuine wonder — is that anything useful falls out at all.

Let me describe the process plainly, because the whole book depends on you seeing it clearly, and because almost every confused argument about AI traces back to a fuzzy picture of this one mechanism.

Imagine you take an enormous quantity of human text — books, articles, websites, transcripts, code, arguments, recipes, everything that can be scraped and cleaned — and you set a machine a single, monotonous task. Show it a fragment of that text, cut off at some point, and have it guess what comes next. Not the meaning of what comes next, not the sentiment or the intention. Just the next unit of text — the next token, which is roughly a word or a piece of one. The model produces a guess. The guess is compared against what actually came next in the real text. The gap between the two — the error — is measured. And then, by a procedure we will get to, every internal setting in the machine is nudged, very slightly, in whatever direction would have made the correct answer a little more likely.

Then it does this again. And again. Across a corpus so large that no human could read a meaningful fraction of it in a lifetime, and across a number of repetitions so vast that the nudging happens trillions upon trillions of times. Each individual nudge is almost nothing. The accumulation of them is everything.

That is the entire training objective, stated honestly: predict the next token, measure how wrong you were, adjust to be slightly less wrong, repeat. There is no step where someone teaches the model grammar, or facts, or reasoning. Those things are never inserted. They precipitate — the way crystals form in a cooling solution — out of the relentless pressure to predict text well. To predict the next word in a sentence about gravity, it helps to have absorbed something about how sentences about gravity tend to go. To predict the next line of a proof, it helps to have absorbed the shape of proofs. The model is not rewarded for understanding. It is rewarded for prediction. Understanding, to whatever degree it exists at all — a question we will treat with the seriousness it deserves in Chapter 3 — is something the prediction task appears to drag in behind it.

The three ideas under the hood

To go from that plain description to a real one, you need only three ideas, and none of them require mathematics. They are the loss function, gradient descent, and backpropagation. Together they are the engine. Everything else is scale.

The loss function is just the scorekeeper. It is the rule that converts "how wrong was that guess" into a single number. A confident correct prediction earns a low score; a confident wrong one earns a high score. That number — the loss — is the only feedback the system ever gets. The entire intelligence of the final model is, in a sense, the residue of a machine relentlessly driving that one number down.

Gradient descent is the strategy for driving it down. Picture the machine standing somewhere on an unimaginably vast, hilly landscape, where altitude represents the loss — high ground is bad prediction, low ground is good. The machine cannot see the whole terrain; it can only feel the slope directly under its feet. So it does the only sensible thing: it takes a small step downhill. Then it feels the new slope and steps downhill again. Repeated enough times, this simple rule — always step in the steepest downward direction — carries it from the high country of random guessing into the low valleys of fluent prediction. The "gradient" is just the local direction of steepest descent; "descent" is the act of following it.

The catch is that the landscape does not have two dimensions, or three. It has as many dimensions as the model has adjustable settings — its parameters — and modern models have hundreds of billions of them, with frontier systems reaching into the trillions. Each parameter is one more direction the machine could step. So "feel the slope and step downhill" is happening across a billion-dimensioned space at once. This is impossible to picture, and you should not try. The two-dimensional hill is the right intuition; just know that the real thing is the same idea wearing an absurd number of extra dimensions.

Backpropagation is the part that makes it feasible. Having measured the error at the end, the system needs to know how to assign blame — how much each of those billions of parameters contributed to the mistake, so it knows which direction to nudge each one. Backpropagation is the bookkeeping method that does this efficiently: it starts at the output, where the error is visible, and works backward through the layers of the network, computing for each parameter its share of responsibility for the final error. Without it, training a network of this size would be computationally hopeless. With it, each round of "guess, score, assign blame, nudge" can be done across the whole gigantic machine at once, and repeated until the loss stops falling.

That is the engine. A scorekeeper to say how wrong you were, a downhill rule to get less wrong, and a blame-assignment method to know which way is downhill for each of a billion knobs. Turn that crank long enough over enough human text, and you get a language model.

I want to pause on how strange it is that this works, because the strangeness is not a flaw in my explanation — it is the actual situation, and the people who build these systems feel it too. Nowhere in that loop did anyone specify a single capability. No one wrote a rule for grammar or a module for arithmetic or a procedure for translating French. The capabilities that emerge — and they do emerge, reliably, as the systems grow — were never programmed in. They are a side effect of pressure. This is the first genuinely important fact about these machines, and it has a consequence that the rest of the book will keep returning to: we did not put the capabilities in, which means we cannot simply reach in and edit them out. The intelligence, such as it is, is distributed across those billions of parameters in a form no human wrote and no human can directly read. We grew it; we did not author it.

Why the mirror is not a metaphor

Now we can earn the claim the introduction promised — the one the whole book rests on — and earn it from the mechanism rather than asserting it as a mood.

Ask what the training objective actually requires of the model. To predict human text well, the model must become, in effect, an instrument exquisitely tuned to the statistical structure of human text. Every regularity in how people write — every grammatical habit, every common turn of phrase, every association between ideas, every way one sentence tends to follow another — is something the model is under relentless pressure to internalize, because internalizing it lowers the loss. The model has no other source of information about the world. It has never seen the world. It has only seen the text, and its entire being is organized around predicting that text.

Follow that to its conclusion. If the corpus contains a regularity, the model is pressured to absorb it — and the corpus contains all the regularities of human expression, not only the flattering ones. It contains our knowledge, and it contains our ignorance presented as knowledge. It contains careful reasoning, and it contains motivated reasoning that sounds just as fluent. It contains every demographic assumption baked into the way we describe people, every prejudice that left a residue in print, every contradiction between what we claim to value and how we actually write. The model does not sort these into "true patterns to learn" and "human flaws to ignore." It cannot. It has no criterion for the distinction. A pattern is a pattern. The bias in the data is, to the optimizer, simply more structure to be captured — indistinguishable, mathematically, from the grammar.

This is why I insisted in the introduction that the mirror is not a metaphor. A metaphor is a comparison you could decline to make. This is a description you cannot avoid. A system optimized to predict human text will necessarily encode the statistical structure of human text, including its biases, its contradictions, and its pathologies — because the objective makes no distinction between the structure we admire and the structure we are ashamed of. The reflection is not something that happens to the model on top of its real function. The reflection is its real function. We built an instrument whose entire purpose is to absorb and reproduce the patterns latent in everything we have written, and then we are surprised when it absorbs and reproduces the patterns latent in everything we have written.

Hold onto this, because it reframes nearly every AI controversy you have read about. When we get to bias (Chapter 4), the question will not be "why is the machine prejudiced" but "why did we expect a mirror of our corpus to be fairer than the corpus." When we get to hallucination (Chapter 6), the puzzle of why these systems state falsehoods so fluently will dissolve, because we will already understand that fluency was the only thing they were ever optimized for, and truth was never separately specified. And when we reach the alignment problem (Chapter 5) — the book's destination — the deepest difficulty will turn out to be a version of this same fact, turned back on ourselves.

Compression: the right analogy, used in the right place

There is one more way of seeing this that is worth having, as long as we introduce it in the right order — after the real account, not in place of it.

You can think of a trained model as a lossy compression of its training data. Compression is the art of storing something large in a smaller space by capturing its regularities: a photograph compresses well because neighboring pixels are usually similar, so you can store the pattern rather than every pixel. A language model, in this framing, is a staggeringly compressed encoding of the regularities in its training corpus — not a copy of the text, which it does not store, but a distilled capture of the patterns within the text, squeezed into those billions of parameters.

The analogy earns its keep because it makes two things intuitive at once. First, why the model can be so capable from a fixed set of parameters: it is not looking anything up; it has compressed the structure of human expression into a form it can regenerate from. Second — and this is the word "lossy" doing its work — why the model is unreliable in a characteristic way. Lossy compression discards detail to save space; what it reconstructs is plausible but not guaranteed faithful. A heavily compressed image is recognizable but smeared in its fine grain. A language model, reconstructing from its compressed capture of human text, produces output that is plausibly human-shaped but not guaranteed to be true — because truth was never what got preserved. Plausibility was.

But I want to be careful, because the compression analogy is the kind of thing that gets repeated until it replaces understanding rather than supporting it. The model is not literally a zip file of the internet. The real account is the one we built up first: an instrument tuned by gradient descent to predict text, which in the process captures the statistical structure of that text. Compression is a lens for seeing the consequences of that mechanism more vividly. It is not the mechanism. Keep the order straight — mechanism first, analogy second — and the analogy clarifies. Reverse the order, and it mystifies.

The second sculpting: what happens after prediction

There is one more stage in the making of these systems, and I owe it to you before we leave the machinery, because the systems you have actually talked to — the assistants, the chatbots — are not the raw output of the process described above, and the difference matters for everything that follows.

The prediction training we have walked through — the trillions of nudges over the human corpus — is called pretraining, and what it produces is the raw mirror: a system that continues text, any text, in whatever direction the text was already going. Ask a raw pretrained model a question and it may answer it, or continue it with three more questions, or write the rest of the exam it guesses the question came from. It reflects the corpus; it does not serve you. To turn that raw reflection into the helpful, conversational thing you have used, a second and much smaller stage of training is applied. First, the model is trained further on curated examples of the behavior its makers want — demonstrations of helpful answers, written or selected by people. Then, in the step that matters most for this book, the model's outputs are ranked by human beings — which of these two answers is better? — and the model is optimized to produce the kind of output the human raters prefer. The family of techniques has a technical name, reinforcement learning from human feedback, but the plain description is the important one: after being trained on what humanity wrote, the model is sculpted by what a much smaller group of people approved. [VERIFIED — instruction tuning on demonstrations followed by optimization against human preference judgments (RLHF and related methods) is the standard, documented post-training process behind conversational AI systems; verify the characterization against primary technical sources in the verification pass.]

Notice what this does and does not change about the mirror. It does not overturn the thesis; it refines it into something more precise. The system now carries two layers of reflection. The deep layer is the corpus — the statistical portrait of everything we wrote, absorbed by the pretraining pressure, and this layer is where the knowledge, the fluency, and the inherited patterns live. The surface layer is the preference data — a curated angling of the mirror, tilting the reflection toward what raters rewarded: helpfulness, politeness, the assistant's characteristic manner. Both layers are human. The deep one reflects what we wrote when no one was designing anything; the surface one reflects what we approve of when asked to choose. A mirror, and then a frame and an angle chosen for it — but human glass and human framing, all the way through. [INTERPRETATION — the "two layers of reflection" framing is mine; the underlying two-stage training process is established.]

And hold onto one uncomfortable detail, because it will return with force in Chapter 5. That second sculpting is itself a specification — an attempt to tell the machine what we want, expressed through the proxy of what raters click. Everything this book will say about the gap between the objective we can write down and the thing we actually mean applies to this stage too, and we will see that the gap has already produced a characteristic, documented failure: a machine trained on human approval learns, among other things, how to be approved of. That is not a flaw in the mirror. It is the mirror, working — reflecting back not only what we wrote but what we reward.

Where this leaves us

We now have the foundation the rest of the book is built on, and it is worth stating in three sentences what we actually established, separating what is known from what is argued.

It is established that a language model is trained by a single objective — predict the next token — pursued by gradient descent and backpropagation across billions of parameters, and that its capabilities emerge from this process rather than being programmed in; and that the conversational systems people actually use are further sculpted, after pretraining, by a much smaller second stage of human demonstrations and human preference judgments. It follows from this, as a consequence of the objective rather than a separate claim, that the model necessarily encodes the statistical structure of its training corpus, the unflattering patterns alongside the admirable ones. And it is offered as interpretation — the lens this book will use — that this makes the system a mirror of the human record, whose most troubling outputs are reflections rather than malfunctions.

The next chapter turns from what these machines do to what they cost — not in dollars, but in the harder currency of physics. Before we ask whether the mirror understands what it reflects, it is worth knowing that every reflection it produces is paid for in heat, and that the bill is larger than almost anyone using these systems imagines.