Chapter 6 — Inside the Black Box

The strange admission at the center of the field

Here is a sentence that ought to be more disturbing than it usually sounds: the people who build the most capable AI systems in the world cannot fully explain how those systems do what they do. This is not modesty, and it is not a marketing pose. It is a plain description of the current situation, stated openly by the researchers themselves. We built these machines — grew them, rather, by the process of Chapter 1 — and having grown them, we find we cannot read them. The intelligence is in there, distributed across billions of parameters, and it is written in a language no human designed and no human can yet fluently read. [VERIFIED — leading researchers openly characterize the internal workings of large models as not fully understood; this is a widely acknowledged state of the field.]

This chapter is about that fact and its most important consequence. We have spent two chapters on why these systems are hard to align — why we cannot easily tell them what to want. Now we confront the companion difficulty: even once a system is running, we cannot easily see what it is doing, or why it produced one output rather than another. And I want to connect that opacity to the single most misunderstood behavior these systems exhibit — the thing people call hallucination — because understanding what hallucination actually is turns out to dissolve a great deal of confusion about what these machines are and are not.

I will keep the same discipline as the rest of Part II. There is real interpretability science here, and I will lean on it. There is also a temptation to fill the gaps in our knowledge with metaphor dressed up as explanation, and I will resist it. The honest account of what we can and cannot see inside a trained network is more interesting than the mythology, and you deserve the honest one.

Why the box is hard to open

We met the interpretability research program in Chapter 3, where it entered as the empirical hope for the understanding debate. Here I want to be precise about why the interior is so hard to read, because the difficulty is not laziness or lack of effort — it is structural, and it follows directly from how these systems are made.

Recall that no one wrote the model's capabilities. They precipitated out of optimization. This means there is no source code for the behavior — no place a programmer wrote `if the user asks about gravity, do the following`. There is only the vast field of adjusted parameters, each a number, collectively encoding everything the model knows in a form that was never meant to be human-readable because it was never written by a human at all. Asking why a model produced a particular output is not like reading a program's logic. It is closer to asking why a specific pattern of connection strengths across billions of synapses produced a specific thought — a question we cannot fully answer about biological brains either. [VERIFIED — the opacity of trained networks stems from capabilities being learned via optimization rather than explicitly programmed; this is a standard characterization.]

The difficulty compounds because of something we encountered in Chapter 3 and should recall here: superposition. Models pack more concepts into their neurons than they have neurons, storing features in overlapping combinations rather than one concept per unit. [VERIFIED — the superposition hypothesis holds that networks represent more features than they have neurons, via overlapping combinations.] This means you cannot simply point at a neuron and read off what it does, the way you might hope to. A single neuron participates in many unrelated concepts; a single concept is smeared across many neurons. The tidy picture — one neuron, one idea — is not how these systems store meaning, and its absence is much of why reading them is so hard.

I want to mark honestly where this leaves us, because it would be easy to slide from "hard to read" into either of two false conclusions. The first false conclusion is that the interior is unknowable — a permanent black box we can only observe from outside. That is too pessimistic; the interpretability program is making real progress, as we will see. The second false conclusion is that we already understand these systems well enough — that the opacity is a solved or minor problem. That is too optimistic; the candid consensus is that we can currently read fragments, not the whole. The truth is the uncomfortable middle: the box is ajar, not open, and how far it can be opened is itself an open question. [INTERPRETATION — the "ajar, not open" framing is mine, consistent with the field's stated position.]

What we can actually see

It would be a disservice to leave you with only the opacity, because the genuinely exciting news is that the box is not sealed, and the tools for prying it open are real and improving. Let me give you the concrete state of what can be seen, because it is the closest thing we have to actual knowledge of the interior rather than speculation about it.

The interpretability program has established that models contain features — directions in their internal space that correspond to interpretable concepts. Researchers have identified features that activate for specific objects, specific ideas, even fairly abstract notions, and have shown that these are not projections read into the noise but real, manipulable structures: intervene on a feature and you change the model's behavior in the way the feature's meaning would predict. [VERIFIED — features as interpretable directions in activation space, and the ability to intervene on them to alter behavior, are established results in mechanistic interpretability.] Models also contain circuits — small subnetworks that implement particular functions, the discovered wiring behind a specific behavior. [VERIFIED — circuits are identifiable subnetworks implementing specific functions; a core concept in the field.] And a recent advance, sparse autoencoders, has begun to pull the overlapping features of superposition apart into cleaner, more separable components, making the interior meaningfully more legible than it was even a couple of years ago. [VERIFIED — sparse autoencoders are used to decompose superposed activations into more interpretable features; an active recent research direction.]

This matters for a reason beyond curiosity, and it connects back to Part II's central worry. If we can identify the internal structures that implement a model's behavior, we move — slowly, partially — toward being able to check what a system has actually learned to do, rather than only observing what it does. That is the thin end of a very important wedge: the difference between trusting a system because it behaves well on the cases we tested, and understanding a system well enough to predict how it will behave on cases we did not. We are nowhere near the second. But the first steps toward it are real, and they are being taken. [INTERPRETATION — the framing of interpretability as the path from behavioral trust to mechanistic understanding is standard in the field; presented here as argument.]

Still, the honest caveat has to travel alongside the good news. What has been read is fragments — particular features, particular circuits, in particular models — not a complete account of any large system's cognition. The interior remains, in the researchers' own candid framing, largely opaque. [VERIFIED — the field openly acknowledges that despite real progress, large-model internals remain largely opaque.] We have learned to read words here and there in a language whose grammar we do not yet possess. That is a genuine and thrilling start. It is not a finished translation.

Do not ask the machine why

There is an apparent shortcut around all of this difficulty, and it is so tempting that it deserves its own section to close off honestly: if the box is hard to read from outside, why not simply ask it? These systems produce language; they will, if prompted, explain their reasoning fluently, step by step, with every appearance of introspective access. Why grub through billions of parameters when the machine will narrate its own thinking on request?

Because the narration is not a readout. Here the mechanism of Chapter 1 must be applied without flinching: when a model explains why it gave an answer, that explanation is produced by exactly the same process as everything else it produces — next-token prediction, generating the most plausible continuation, in this case the most plausible-looking explanation. Nothing in the architecture wires the explanation to the actual internal computation that produced the answer. The model is not reporting its process; it is modeling what an explanation of such an answer would look like, drawn from a corpus full of humans explaining things. And the research bears the suspicion out: experiments have shown that models' stated reasoning can be systematically unfaithful — that factors demonstrably steering the model's answers, such as biases planted in the prompt, can be entirely absent from its confident, articulate account of why it answered as it did. The explanation reads as introspection and functions as confabulation. [VERIFIED — the unfaithfulness of model self-explanations and chain-of-thought reasoning, including cases where demonstrable determinants of the answer never appear in the stated rationale, is a documented finding in the interpretability and evaluation literature; verify representative studies in the verification pass.]

Two disciplined caveats, one in each direction. First, this does not mean step-by-step prompting is useless — having a model reason in visible steps measurably improves its performance on many tasks, and the visible steps are often genuinely useful to a human checking the work. The point is narrower and sharper: usefulness is not faithfulness, and an explanation can help you verify an answer without being a true account of how the answer was made. Second, this does not license despair about interpretability — it locates it. It tells us the box cannot be opened by interviewing the box, which is precisely why the field's serious effort goes through the parameters and activations rather than the prose. There is even a poignant symmetry here, and I will mark it as the aside it is: humans, too, confabulate — decades of psychology document our fluent, sincere, wrong explanations of our own behavior. A machine trained on our record was never going to learn introspective transparency from us. We do not model it. [INTERPRETATION — the parallel to human confabulation is offered as interpretation; the documented unreliability of human self-report in specific experimental settings is real, but the correspondence to model unfaithfulness is my framing, not an established equivalence.]

What hallucination actually is

Now to the behavior that this opacity makes most misunderstood — and where getting the mechanism right dissolves an enormous amount of confusion.

People speak of AI systems "hallucinating," and the word is doing quiet damage, because it implies a malfunction — a system that normally reports truth but occasionally glitches into fabrication, the way a healthy mind normally perceives reality but occasionally misfires into seeing things. That picture is wrong, and the way it is wrong is the single most clarifying thing you can understand about these machines. Here is the correct account, and it follows directly from Chapter 1.

The model is always doing exactly one thing: predicting the next token, producing the most plausible continuation of the text so far, based on the statistical structure it absorbed in training. That is the whole operation. When you ask it a factual question and it produces a true answer, it is generating a plausible continuation that happens to correspond to reality. When you ask it a factual question and it produces a false answer stated with equal confidence, it is doing exactly the same thing — generating a plausible continuation — that happens not to correspond to reality. The internal process is identical in both cases. There is no separate "now I am telling the truth" mode and "now I am hallucinating" mode. There is only continuation, and whether the world happens to agree. [VERIFIED — the account of hallucination as the same next-token prediction process that produces correct output, differing only in correspondence to fact, is a standard and accurate technical characterization.]

This is why I insisted, back in Chapter 1, that for these systems "fact" and "fabrication" are the same act distinguished only by whether the world agrees. The model has no internal notion of truth. It was never trained to track truth; it was trained to predict text. Truth and plausibility usually travel together in the training data — most fluent, confident text about the world is roughly accurate, so predicting plausible text usually yields true text as a byproduct. But when they part ways — when the most plausible-sounding continuation is not the true one — the model has no mechanism that notices, because it has no representation of truth to consult. It is not lying, which would require knowing the truth and choosing against it. It is not malfunctioning, which would require a normal mode it has departed from. It is doing the only thing it ever does, and the output happens to be false. [INTERPRETATION — the framing that the model "cannot lie because it has no truth to depart from" follows from the mechanism; marked as interpretation of the established mechanism.]

Once you see this, the word "hallucination" reveals itself as not just inaccurate but actively misleading, because it locates the strangeness in the failures. The deeper truth is that the successes are made of exactly the same material as the failures. Every correct answer a model gives you is a plausible continuation that happened to be true. The machine is not a knower that sometimes errs. It is a plausibility engine whose outputs we sort, after the fact, into "true" and "false" using a standard the machine itself never had access to. The mirror of Chapter 1 returns here in its sharpest form: the model reflects the statistical shape of what we have written, and a confident falsehood is as faithful a reflection of that shape as a confident truth, because our writing contains both in the same fluent register.

When the stakes are real

This is not an abstract point, and it is worth grounding in consequence, because the gap between fluent form and absent truth-tracking has already produced real harm in the world.

The documented cases follow a consistent and revealing pattern: a system produces output that is formally perfect and substantively fabricated. The most widely reported instances involve professionals who relied on model output without verification and were burned by inventions delivered with total fluency — legal filings citing cases that do not exist, complete with plausible-sounding names, plausible-sounding citations, and plausible-sounding summaries, none of which corresponded to any real case, because the model was generating what a citation looks like rather than retrieving one that exists. [VERIFIED — cases of AI systems generating fabricated legal citations, subsequently relied upon and exposed, are documented; verify and cite specific representative instances in the verification pass.] The form was immaculate. The semantics were void. This is the Chinese Room of Chapter 3 made costly: syntax without semantics, the shape of a citation with nothing real behind it.

The lesson is not "the machines are unreliable and should be avoided," which overcorrects into uselessness, nor "the failures are rare edge cases," which understates a structural feature. The accurate lesson is narrower and more useful: because these systems produce fluent form whether or not there is truth behind it, and because they have no internal signal distinguishing the two, the burden of verification falls entirely and unavoidably on the human. This is not a temporary limitation to be engineered away next year. It follows from what the machine fundamentally is — a plausibility engine, not a knower — and it sets up directly the division of labor that Part III will make central: the machine generates, the human verifies, and the verification cannot be delegated back to the thing that needs verifying. [INTERPRETATION — the framing of verification as a permanent and non-delegable human burden is argued from the mechanism; it anticipates the centaur argument of Chapter 7.]

One gesture toward the mirror, clearly marked

I have kept this chapter, like the rest of Part II, disciplined about mechanism. Let me close with a single interpretive gesture, marked plainly as interpretation and offered rather than asserted, because it connects this chapter's machinery to the book's larger claim and because the connection is, I think, genuinely illuminating.

We have said that the model has no notion of truth — only plausibility — and that its confident fabrications are faithful reflections of a corpus in which confident writing is not always true writing. Consider what that reflects back about us. The model produces fluent falsehood so readily because we produce fluent falsehood so readily — because the training data, which is to say the human record, is full of confident, well-formed, plausible-sounding text that happens to be wrong. The machine's inability to distinguish truth from plausibility is, in part, a reflection of how often, in our own writing, the two come apart while the fluency stays constant. When we are unsettled by a model that states falsehoods with the same confidence it brings to facts, we might ask how different that is from the corpus it learned from — from us, at our most fluent and least accurate. [INTERPRETATION — this closing reflection is explicitly offered as interpretation, not as a claim about the mechanism; consistent with the book's practice of marking such gestures.]

That is a lens, not a finding, and I mark it as such. But it is the lens the whole book is built around, and it will move from the margins to the center now, as we cross into Part III and turn the mirror, at last, fully toward ourselves.

Where this leaves us

As ever, let me separate the established from the argued.

It is established that the internal workings of large models are, by the candid acknowledgment of the field itself, not fully understood — that capabilities are grown rather than programmed and therefore have no human-readable source; that superposition makes the interior especially hard to read; and yet that the interpretability program has identified real, manipulable internal structures — features, circuits, and, more recently, cleaner decompositions via sparse autoencoders — so that the box is genuinely ajar even if far from open. It is established that hallucination is not a malfunction but the ordinary next-token-prediction process producing output that happens not to correspond to reality, by exactly the same mechanism that produces output that does — the model having no internal notion of truth, only plausibility. And it is established, in documented and costly real-world cases, that these systems produce formally perfect, substantively fabricated output, placing the burden of verification unavoidably on the human.

It is offered as interpretation, marked as such, that the successes and failures of these systems are made of the same material, that verification is therefore a permanent rather than temporary human responsibility, and — as the chapter's one gesture toward the book's lens — that a machine which cannot tell plausible from true is reflecting a corpus, and a species, for whom the two more often part ways than we like to admit.

That completes Part II. We have seen why these systems are hard to align and hard to read, and we have found, at every turn, that the difficulty runs back toward us. Part III now turns fully to that human side — beginning not with a problem but with a possibility: that the right relationship between human and machine is not competition but combination, and that understanding the machine's real nature is exactly what lets us combine with it well.