Chapter 3 — Computation Versus Understanding

The question we have been avoiding

We have built up, over two chapters, a fairly complete picture of what a language model does. It is trained by predicting text, one token at a time, until the structure of human language precipitates into its billions of parameters. It runs on real machines that consume real power and dissipate real heat. We have described the mechanism and we have counted the cost. We have not yet asked the question that everyone actually wants answered, because it is the hardest one and the least settled: when one of these systems produces a fluent, apt, seemingly thoughtful response — does it understand what it is saying, or is it only arranging symbols it has no grasp of?

I want to be honest with you from the first sentence of this chapter: I am not going to resolve this for you, because it is not resolved. This is the one place in Part I where the rigorous move is not to deliver an answer but to map a genuine disagreement with enough care that you can see exactly where the fault line runs and why intelligent, informed people stand on opposite sides of it. A book that pretended to settle the understanding question would be lying to you about the state of the field. What I can do — what is actually useful — is show you the strongest version of each position and tell you, as precisely as possible, what is known and what is merely argued.

The room that knows no Chinese

Start with the thought experiment that has framed this debate for forty years, because even though it was written before any of today's machines existed, it isolates the core intuition with surgical precision.

In 1980 the philosopher John Searle asked you to imagine a man locked in a room. He does not understand a word of Chinese. Slips of paper with Chinese characters come in through a slot. The man has an enormous rulebook, written in his own language, that tells him: when you see this sequence of squiggles, write that sequence in response, and pass it back out. The rulebook is so good that the responses he produces are indistinguishable from those of a fluent Chinese speaker. To anyone outside the room, passing notes in, the room appears to understand Chinese perfectly. But the man inside understands nothing. He is manipulating symbols according to rules, with no idea what any of them mean. [VERIFIED — Searle's Chinese Room argument, 1980.]

Searle's point lands like a hammer: syntax is not semantics. Manipulating symbols by their shape — which is all the man in the room does, and, Searle argued, all any computer does — is not the same as understanding what those symbols mean. The room has the form of understanding with none of the substance. And if the Chinese Room does not understand, Searle asked, why should we believe any symbol-manipulating machine does, no matter how convincing its output?

This connects to a problem philosophers call symbol grounding. The squiggles in the room are ungrounded — they connect to nothing in the man's experience, point to no objects, carry no meaning for him. They are just shapes. The question symbol grounding poses is: how does any symbol ever come to mean something? For you, the word "water" is grounded in a lifetime of experience — thirst, rain, the feel of it, the sight of it. For a system trained only on text, the word "water" is a token, a position in a sea of other tokens, connected to "wet" and "drink" and "ocean" by statistical association but never, it seems, to water itself. The model has, in a sense, swallowed the entire dictionary — but a dictionary defines every word only in terms of other words, in an endless closed loop that never touches the world. Can meaning live inside that loop? Or does it require contact with reality the loop can never provide? [VERIFIED — the symbol-grounding problem, Harnad 1990, is a standard problem in cognitive science and AI philosophy.]

Hold that question. It is the 1980 version of the debate, and it is sharp. But the outline of this book insists, correctly, that we not stop there — because the interesting argument today is not the one Searle was having.

The argument we are actually having

The modern debate has a name on each side, and the disagreement between them is the live wire of contemporary AI.

On one side is a position crystallized in a now-famous phrase: stochastic parrots. In an influential 2021 paper, the linguist Emily Bender and her colleagues argued that a large language model, however fluent, is fundamentally a system for stitching together sequences of linguistic forms it has observed, according to probabilistic information about how they combine, but without any reference to meaning. [VERIFIED — Bender et al., "On the Dangers of Stochastic Parrots," 2021; the "stochastic parrot" framing is theirs.] A parrot can reproduce the sounds of speech with uncanny accuracy and understand none of it. On this view, that is exactly what a language model is: a stochastic — meaning probability-driven — parrot, producing the statistical shadow of meaning without the thing itself. The fluency is real; the understanding is a projection we, the listeners, supply. It is the Chinese Room at industrial scale.

On the other side is what we can call the emergence or scaling position. Its proponents point to something the parrot framing struggles to explain: as these models have grown larger, they have begun to do things that look much less like parroting. They solve problems not present in their training data. They carry out multi-step reasoning. They translate between languages they were never explicitly taught to pair. They display capabilities that emerged with scale, without anyone programming them in — capabilities that, the argument goes, are hard to account for if the system is merely matching patterns of surface form. Perhaps, this side suggests, predicting text well enough, across enough of it, requires building internal models of the world that generated the text — and perhaps those internal models constitute a real, if alien, form of understanding. [VERIFIED — the emergent-capabilities/scaling position is a genuine and active counter to the stochastic-parrots view; the existence, nature, and predictability of "emergence" are themselves debated.]

Notice what makes this hard. Both sides are looking at the same fluent output and drawing opposite conclusions about what lies behind it. The parrot camp says: fluency is cheap; do not be fooled into reading understanding into statistics. The emergence camp says: at sufficient scale, fluency may be impossible without something worth calling understanding. And here is the genuinely uncomfortable part, the part a rigorous book must not paper over: we cannot currently settle the question by looking at behavior alone, because the whole problem is that behavior — the notes passed out of the room — is exactly what both sides agree on. The disagreement is about what produces it. To make progress, you cannot keep staring at the output. You have to open the box.

The geometry of meaning

Before we open the box, it helps to see one concrete thing about how these systems represent language — because it is genuinely illuminating, and because it is easy to over-read.

Inside a language model, every word (or token) is represented as a long list of numbers — a position in a space of many hundreds or thousands of dimensions. These are called embeddings, and the remarkable thing is that the geometry of this space is not random. Words with related meanings end up near each other. And the directions in the space turn out to carry meaning too. The famous illustration: take the embedding for "king," subtract the embedding for "man," add the embedding for "woman," and you land very close to the embedding for "queen." [VERIFIED — the "king − man + woman ≈ queen" result is a well-known property of learned word embeddings, e.g. word2vec, Mikolov et al. 2013.] The model has, without being told, arranged its representations so that a particular direction in the space corresponds to something like the concept of gender, and another to royalty, and these can be composed by arithmetic.

This is striking, and it is tempting to take it as proof of understanding — the machine "knows" that kings and queens differ as men and women do. But discipline is required here, and it cuts toward humility. What we have shown is that the model has captured relationships — geometric regularities in how words pattern together across the corpus. That is real and it is powerful. It is not, by itself, evidence that the model grasps royalty or gender the way you do, with the felt weight of meaning behind the words. The geometry is a map of how words relate to other words — the inside of the dictionary, rendered as shape. Whether a map of relationships among symbols is understanding, or merely a very sophisticated version of the Chinese Room's rulebook, is precisely the question we cannot answer by admiring the map. The embeddings make the parrot smarter; they do not obviously make it not a parrot.

A board game inside the machine

Before we open the box in earnest, there is one experimental result worth knowing in detail, because it is the cleanest piece of concrete evidence the emergence side has, and because handling it with discipline is a good rehearsal for everything interpretability will ask of us.

Researchers trained a small language model on nothing but transcripts of games of Othello — the board game — expressed as sequences of moves. The model never saw a board. It was never told the rules, never shown the grid, never given anything but strings of move notations to predict, one token at a time, exactly as the models of Chapter 1 predict text. The question was whether a system trained only to predict move sequences would remain a creature of surface statistics — a parrot of plausible move patterns — or whether something more would precipitate. The finding: probes of the model's internal activations recovered a representation of the state of the board — which squares were occupied, and by whom — even though no board was ever presented. More striking still, when researchers intervened on that internal representation, editing the model's represented board state directly, its subsequent move predictions changed the way they should for the edited board. The representation was not a passive echo; the model was using it. [VERIFIED — the "Othello-GPT" emergent world-representation result, in which probes recovered a causally functional board-state representation from a model trained only on move sequences, is a published and replicated finding; verify the primary sources and characterize the probing and intervention methodology precisely in the verification pass.]

Now the discipline, in both directions. What this shows, genuinely, is that prediction pressure alone can induce an internal model of the process generating the sequences — that "just predicting the next token" does not preclude, and in this case demonstrably produced, a structured representation of the hidden world behind the tokens. That is a real blow against the strongest version of the parrot thesis, the version that says surface statistics is all such systems can ever contain. But note what it does not show, because the temptation to overread is enormous. Othello is a tiny, closed, fully-determined world, in which the move sequence contains, in principle, complete information about the board. Human language is none of those things: the world behind our text is vast, open, and radically underdetermined by the text itself. That a predictor can reconstruct a small world its data fully specifies does not establish that a predictor of human text has reconstructed the large world our text only gestures at. [INTERPRETATION — the extrapolation from the Othello result to world-modeling in large language models is argument, not established finding; both camps accept the experiment and dispute its reach.] So the honest reading is this: the result converts "could prediction ever produce a world model?" from an open question into an answered one — yes, in at least one small case, demonstrably. Whether it has done so at scale, for the world, in the systems we actually use, remains exactly the question the rest of this chapter takes up. The experiment does not settle the debate. It does something almost as valuable: it proves the debate is empirical.

Opening the box

For most of this debate's history, the box was sealed. We could see what went in and what came out, but the inside — those billions of parameters, that high-dimensional geometry — was an opaque tangle no human could read. This is the deep reason the understanding question stayed stuck: both sides were arguing about the contents of a box neither could open.

That is now changing, and it is the most genuinely exciting development in this whole area — far more so than the philosophical sparring, because it is empirical. A research program called mechanistic interpretability has set out to do something that sounds impossible: to reverse-engineer trained neural networks, to look inside and identify the actual structures that implement their behavior. [VERIFIED — mechanistic interpretability is an active research field; the systematic study of how neural networks implement algorithms through their learned representations.] It is, roughly, neuroscience for artificial minds — except that, unlike a biological brain, this brain can be paused, rewound, and probed neuron by neuron.

The program has produced real, concrete concepts, and they are worth knowing because they are the closest thing we have to actual knowledge of what is inside. Features are directions in the model's internal space that correspond to interpretable concepts — researchers have found features that activate for specific ideas, objects, even abstract notions. Circuits are small subnetworks that implement particular functions, identifiable chunks of the machine that carry out a specific computational job — the discovered wiring behind a behavior. [VERIFIED — features and circuits are core, defined concepts in mechanistic interpretability.] And superposition is the unnerving discovery that models pack more concepts into their neurons than they have neurons, by storing features in overlapping combinations rather than one-per-neuron — which is part of why the box was so hard to read in the first place. [VERIFIED — the superposition hypothesis, Elhage et al. 2022; networks represent more features than they have neurons via overlapping combinations.] A recent advance — sparse autoencoders — has begun to pull these overlapping features apart into cleaner, more interpretable components. [VERIFIED — sparse autoencoders are used to decompose superposed activations into interpretable features; an active 2023–2025 research direction.]

I want to be careful not to oversell this. The box is not yet open; it is ajar. The internal workings of large models remain, in the candid words of the researchers themselves, largely opaque. [VERIFIED — researchers in the field openly characterize LLM internals as still largely opaque.] We have learned to read fragments, not the whole. But the direction matters enormously, because it converts the understanding debate from a philosophical standoff into an empirical research question. We may never settle "does it understand" by argument. We might, eventually, settle a sharper version of it by looking — by determining whether the structures inside implement genuine world-models or only elaborate surface-pattern-matching. That this is now a question we can investigate, rather than only debate, is the real news. It is also why this book put mechanistic interpretability where a lesser account would have put speculation.

The thing we cannot get at from outside

There remains one piece of the understanding question that no amount of opening the box will reach, and intellectual honesty requires naming it clearly and marking it for what it is: not science, but philosophy.

Imagine a brilliant scientist — call her Mary — who has lived her entire life in a black-and-white room, studying color through black-and-white books and screens. She knows everything physical there is to know about color: the wavelengths, the retinal chemistry, the neural pathways, every fact in the complete science of vision. Then one day she walks out and sees a red rose for the first time. Does she learn something new? The overwhelming intuition is that she does — that she finally knows what red looks like, what it is like to see it, and that this is a fact no amount of physical description ever gave her. [VERIFIED — the "Mary's Room" / knowledge argument, Frank Jackson, 1982.]

This is the problem of qualia — the subjective, felt quality of experience, the redness of red, the painfulness of pain. And it draws a line that may be uncrossable by any of the methods in this book. We can describe everything a system does. We can, increasingly, describe the structures inside it that produce what it does. But there seems to be a further question — is there something it is like to be that system? — that no amount of mechanism, however complete, obviously answers. The blueprint can be total and the question of inner experience can remain entirely open.

I raise Mary's Room not to claim that machines do or do not have inner experience — I have no idea, and I am suspicious of anyone who claims certainty in either direction. I raise it to mark, precisely, the boundary of what mechanism can tell us. The understanding question has two layers. One layer — does the system build real internal models, does it grasp relationships and structure — is hard but appears to be empirically approachable, and mechanistic interpretability is approaching it. The other layer — is there a felt, conscious inside, a subject having the experience — may lie permanently beyond the reach of the third-person methods that have served every other question in this book. Keeping those two layers distinct, and being honest about which is which, is the most important thing this chapter can leave you with.

Where this leaves us

As ever, let me separate the established from the argued.

It is established that language models manipulate symbols according to learned statistical structure (this is just the mechanism of Chapter 1); that they represent words as geometric relationships in a high-dimensional space, capturing real regularities of how words pattern together; and that a genuine research program, mechanistic interpretability, has begun to identify interpretable internal structures — features, circuits, superposition — though the interior remains largely opaque.

It is genuinely unresolved — not by my reticence but by the actual state of the field — whether this amounts to understanding. The stochastic-parrots view and the emergence view both fit the behavioral evidence, and the question is now migrating, slowly, from philosophy toward the empirical study of what is actually inside the models.

And it is a matter of philosophy, marked as such, whether there is any felt, conscious experience inside these systems at all — a question that the complete success of every mechanistic method might still leave entirely untouched.

We set out, two chapters ago, to understand the mirror as a mechanism, and we have. We have seen how it learns, what it costs, and how sharply thoughtful people disagree about whether it understands. That completes the foundation. Now the book turns from how these systems work to why aligning them to human purposes is so hard — and we will find, in the next chapters, that the difficulty begins exactly where the mirror does: in the data, which is to say, in us.