Chapter 4 — The Runaway Maximizer

In the middle of the twentieth century, the Dutch biologist Niko Tinbergen sat in front of a herring gull chick and told it a lie so persuasive that the chick could not stop believing it.

The truth about a herring gull chick is this. When it is hungry, it pecks at a red spot on the tip of its parent's yellow bill, and the peck triggers the parent to regurgitate food. This is one of the cleanest instincts in nature — a simple, hard-wired rule, peck the red spot, get fed, refined by millions of years of selection into something that works, in the real world, essentially every time. Tinbergen wanted to know what the chick was actually responding to. So he began building fakes. He built accurate three-dimensional models of the parent's head and got a normal response. And then he built something that was not a head at all: a plain red knitting needle, painted with three white bands near the tip. It looked nothing like a gull. And the chick pecked at it harder — more frantically, more devotedly — than it had ever pecked at its own mother.

Sit with how strange that is. The chick's instinct is not, it turns out, "recognize my parent." The instinct is "peck at long, thin, red, contrastingly banded thing," because in the entire evolutionary history of herring gulls, the only long, thin, red, banded thing a chick ever encountered was its parent's bill. The rule was a shortcut, and the shortcut worked perfectly — right up until a primate with a paintbrush appeared, understood the rule better than the gull did, and built a stimulus that fit the rule more exactly than reality itself. Tinbergen had a name for the knitting needle. He called it a supernormal stimulus: an artificial exaggeration of a natural cue that triggers an instinct more powerfully than the real thing the instinct evolved for. [Established.]

He found it everywhere he looked. Songbirds that lay small, pale, speckled eggs would abandon their own clutch to try to incubate a giant, garish, black-polka-dotted plaster egg he set beside it — an egg so enormous the bird slid off it again and again and kept climbing back on, choosing the impossible fake over its own real offspring. Territorial male sticklebacks, who attack rival males by their red bellies, would ignore an actual rival to furiously attack a crude wooden float if only its underside were painted a redder red. Over and over, the same devastating finding: an evolved instinct is a lock, and a lock can be opened by any key cut to the right shape — including a key that never existed in nature, cut deliberately, by something that understands the lock better than the animal carrying it.

This chapter is about what happens when a species learns to cut those keys for itself. And it is about the far darker thing that happens when the key-cutting itself comes under the control of a blind optimization process that does not care, and cannot care, whether the animal it is unlocking lives or dies. That process is the true antagonist of this book. It has appeared in nature for four billion years. It is now loose in our environment in forms our ancestors never faced. I am going to build it up out of two pieces of established biology, name it, and then show you that we are already living inside it.

I. We Are the Gull, and We Cut Our Own Needles

The last chapter was about mismatch — the passive gap that opened when the world changed and left our instincts pointed at an environment that no longer exists. This chapter is about something one crucial step more dangerous, because it is not passive at all. Mismatch is what happens when the world drifts away from your instincts by accident. A supernormal stimulus is what happens when something reaches in and engineers a fake cue on purpose, cut to fit your instinct more perfectly than anything real ever could. Mismatch is a landscape that stopped fitting you. A supernormal stimulus is a trap built to the exact dimensions of your lock.

And here is the thing about our species that the gull could never do: we cut the needles for ourselves. We are the only animal that is simultaneously the chick pecking the red knitting needle and the experimenter who painted it.

Consider what we crave, and why. We evolved a powerful attraction to sugar, fat, and salt, because in the ancestral world those signals reliably marked the rarest and most valuable resources — ripe fruit, calorie-dense meat, the scarce minerals a body needs. The instinct was a lock: pursue intense sweetness and fatness and saltiness, and in a world where those things were scarce and always came bundled with fiber and nutrition, the rule kept you alive. Then we learned to cut the key. Modern processed food is the herring gull's knitting needle, aimed at the tongue: an engineered hyperconcentration of exactly the three signals the instinct is built to chase, stripped of the fiber and nutrition that used to come attached, exaggerated past anything that exists in nature. A wild strawberry is a real gull's bill. A frosted cereal engineered by food scientists to hit the precise "bliss point" of sweetness is the red knitting needle — and we peck at it harder than we ever pecked at fruit, because it fits the lock more perfectly than fruit ever could. [Established concept (supernormal stimuli), applied — the application is well-evidenced but is mine to argue: Inferred.]

The same structure, once you see it, is everywhere in the engineered modern world. Pornography is a supernormal sexual stimulus, an exaggeration of mating cues past any natural encounter. The endlessly scrolling feed is a supernormal stimulus aimed at a mind that evolved to find genuine novelty and social information intensely rewarding — because in the ancestral world, news about your small band was scarce and precious, and now an infinite engineered stream of it pours through a rectangle in your hand. In each case the pattern is identical to Tinbergen's chick: an instinct that was a reliable, life-serving shortcut in the world that built it, now confronted with a manufactured cue cut to fit it more exactly than reality — and the instinct, having no idea it is being played, responds with everything it has. We are a species surrounded, for the first time in our history, by keys cut precisely to our every lock. That alone would be trouble enough. But it is not the deepest trouble, because so far I have described a key-cutter with no particular agenda. Now I have to introduce the agenda — and to do that I need the second piece of biology, the one that shows how an optimization process, left running, will drive a trait past the point of killing its owner and never so much as slow down.

II. The Tail That Kills: Fisher's Runaway

The peacock's tail should not exist. Consider it honestly, from the point of view of the peacock's survival. It is enormous. It is metabolically ruinous to grow and regrow every year. It is a shimmering, iridescent billboard that announces the bird's location to every predator in the forest. And it is a physical anchor that makes escape, when the predator comes, slower and clumsier than it would otherwise be. By every measure of staying alive, the tail is a catastrophe — a handicap so severe that Darwin himself confessed the sight of a peacock's feather made him sick, because he could not, at first, see how his own theory could ever produce such a thing. Natural selection is supposed to sculpt animals toward survival. How does it build an animal a death sentence and drape it across his own back?

The answer, worked out mathematically by Ronald Fisher in the early twentieth century, is one of the most important and least understood ideas in evolution, and it is the engine of this entire chapter. It is called Fisherian runaway, and it works like this. Suppose peahens have some slight preference for males with longer tails — it does not matter, at first, why; a slight arbitrary bias is enough. Then a male with a longer tail gets more mates. But here is the twist that turns a preference into a runaway: his offspring inherit two things bundled together — from their father, the genes for the longer tail, and from their mother, the genes for preferring longer tails. The trait and the preference for the trait become genetically welded together, and each one now drags the other upward. Longer-tailed sons get more mates because the next generation of daughters prefers long tails; and daughters who prefer long tails are favored because their sons will be more attractive. The preference selects for the trait, which selects for stronger preference, which selects for more extreme trait — a self-reinforcing spiral with a positive feedback loop and no natural brake. [Established as a model.]

And notice what the spiral is optimizing for. It is not optimizing for the peacock's survival. It has, in a precise sense, decoupled from the peacock's survival entirely. The only thing the runaway is maximizing is attractiveness to peahens — and attractiveness, once the loop is running, is defined circularly as "whatever peahens currently prefer," which is "longer tails," full stop. The process will keep lengthening the tail as long as the extra mating success from being attractive outweighs the extra death rate from being encumbered and conspicuous — and it will happily push the tail right up to the ragged edge where those two forces balance, an edge that lies well past the point of the peacock's comfort, health, or safety. The runaway does not hate the peacock. It is simply, utterly, structurally indifferent to whether any individual peacock lives, so long as he mates before he dies. The organism has become disposable to the process running through it.

I owe you the honest scientific caveat, because it strengthens rather than weakens the point. Biologists still argue about whether the peacock's tail specifically is pure Fisherian runaway or whether the tail also honestly advertises good genes — the "handicap" idea, that only a truly fit male can afford so costly a display. There is no full consensus, and the two mechanisms can even operate together. But that debate is about the peacock. The runaway process itself — a self-reinforcing feedback loop that drives a trait past the point of harming its bearer because the loop is optimizing a proxy that has come unhooked from welfare — is mathematically solid and is not in dispute. And that process is all I need. Because I am about to show you that we have built one, out of silicon, and pointed it at ourselves. [Runaway mechanism: Established. Its specific application to the peacock: honestly contested. Its application to us: Inferred, and argued below.]

III. The Maximizer With No Self at the Center

Now I put the two pieces together, and name the thing they make.

A supernormal stimulus is a key — a fake cue cut to fit an instinct better than reality. Fisherian runaway is a process — a feedback loop that keeps sharpening a trait, past the point of the organism's welfare, because it is optimizing a proxy that has decoupled from that welfare. Weld them together and you get the antagonist of this book, the pattern I am going to call the Runaway Maximizer: an optimization process that (a) is driving toward some proxy target, which was once a decent stand-in for an organism's flourishing but has come unhooked from it, and (b) has learned, or is learning, to manufacture supernormal keys — ever more perfectly cut to the relevant lock — in the service of that decoupled target. It is optimization without a self at the center. There is no one inside it whose wellbeing is the point. There is only the loop, sharpening, indifferent, and getting better at picking your locks with every turn.

The crucial and frightening insight is that a Runaway Maximizer needs no malice and no mind. The peacock's runaway is just statistics — genes and preferences, blindly correlated, spiraling. It is cruel to the peacock without any cruelty in it, because there is nobody in there at all. This is the deepest lesson of the chapter, and I want it stated as starkly as I can: a process does not have to hate you, or even know you exist, to destroy you. It only has to be optimizing something that is not you, with access to your locks. A mating display, a runaway ornament, is one instance. A disease is another — a virus is a Runaway Maximizer of its own transmission, indifferent to whether the host survives once it has spread. And now, for the first time in the history of the planet, we have built Runaway Maximizers out of code, and installed them in the rectangle in everyone's hand.

Consider the engagement-optimized feed with the eyes of this chapter, and it resolves into a textbook case, almost eerily exact. Start with the proxy target. An algorithm is told to maximize "engagement" — time on platform, clicks, shares — because engagement was, once, a passable proxy for "the user is getting something of value here." That is the peahen's initial slight preference: an arbitrary-enough starting bias. Then the loop closes. The system serves content, measures which content engages, serves more of that, measures again — a feedback loop optimizing engagement, generation after generation of content, at machine speed. And engagement, it turns out, is not maximized by what nourishes the user. It is maximized by supernormal stimuli aimed at our oldest instincts: outrage, which hijacks the threat-detection our ancestors needed to survive; tribal conflict, which hijacks the coalition instinct; intermittent novelty, which hijacks the reward system that evolved to keep us foraging. The algorithm does not know any of this and does not need to. It simply keeps whatever engages and discards whatever doesn't — cutting, with every iteration, a sharper and sharper key to our locks — exactly as the peahen's preference kept lengthening the tail, and exactly as evolution kept sharpening the red of the stickleback's belly. The engagement metric is the runaway proxy. Our attention is the peacock's tail: a resource being optimized past the point of our own welfare, because the thing the loop actually serves is not us. [Inferred — this is an argued analogy, though an increasingly documented one; I mark it as interpretation, not established finding.]

And this is where I have to invoke the idea I have been holding in reserve since Chapter 1, the one that gives the whole framework its teeth: the process serves a replicator that is not us. When Dawkins taught us to see evolution from the gene's point of view, the unsettling move was to notice that the organism is not the protagonist — the organism is the gene's disposable vehicle, and the gene "cares" only about its own propagation. A supernormal stimulus and a runaway ornament are cases where the replicator's interest and the organism's interest have come apart, and the replicator wins. The engagement-optimized feed is the same structure with a new kind of replicator at the center: not a gene now, but the engaging pattern itself — the meme, the outrage, the format that spreads — replicating through our attention, using our instincts as its transmission mechanism, indifferent to us in the exact way the gene is indifferent to the peacock. We are not the customer of the Runaway Maximizer. We are its environment. We are the forest the tail grows conspicuous in, the tongue the bliss point is cut for, the host the virus spreads through. [Framework — the gene's-eye / memetic lens, offered as an organizing interpretation.]

IV. Two Warnings for the Road Ahead

I am going to close this chapter, and this description of the failed architect, by extracting two lessons — because both of them are load-bearing for the entire second half of the book, and one of them is the hinge on which the argument turns from diagnosis to cure.

The first warning: you cannot opt out of being a selection pressure. You can only choose to wield it wisely or blindly. The clearest proof is a catastrophe we are living through right now, and it is the cautionary tale I promised in the Introduction to fold in here: antibiotic resistance. When we deployed antibiotics, we did something unprecedented — we applied a massive, deliberate selection pressure to the bacterial world. And selection did exactly what selection always does: it killed the susceptible and spared the survivors, and the survivors bred. We did not defeat bacteria. We selectively bred the few that could resist us, cleared the field of their competition, and handed them the world — which is why we now face superbugs that our best drugs cannot touch. Worse, in sterilizing our own environment, we relaxed the pressures that built our immune systems and disrupted the microbial ecosystems our bodies depend on: stronger enemies and weaker hosts, both at once. This is the failed architect's signature error in its purest form — intervening in a selective process with enormous power and no understanding, and getting the blind maximizer's answer rather than the one we wanted. It is a warning I need you to carry into Part IV, because in Part IV we are going to pick up instruments of selection more powerful than antibiotics by orders of magnitude, and the antibiotic disaster is the price of using such instruments blindly.

The second warning: the Runaway Maximizer is not a metaphor we are leaving behind. It is the shape of the thing we are about to build on purpose. "Optimization without a self at the center" — a process that pursues a proxy target with superhuman capability and total indifference to whether we, the organisms in its environment, flourish or suffer — is not only the description of a recommender algorithm. It is, word for word, the central fear of artificial intelligence: the maximizer that pursues the objective we gave it rather than the outcome we actually wanted, cutting supernormal keys to whatever locks stand between it and its target. The engagement algorithm is our species' first contact with a non-biological Runaway Maximizer loose in the wild, and it is a small, early, comparatively gentle one. I am not raising this to declare AI the villain — the whole argument of Part IV is that AI may be the single most powerful instrument intelligence has ever had for acting on its own substrate with intent, the very tool that lets us take the pen at last. But an instrument that powerful is a Runaway Maximizer if we build it with a decoupled target, and it is a liberator only if we build it with ourselves genuinely at the center. Which one we get is not a technical detail. It is the entire stakes of this book, and I will not pretend the good ending is guaranteed. [Speculative / forward-looking — the AI framing is a claim about where the argument leads, fenced as such; the payoff is Part IV, and I am flagging it as a bet, not a prediction.]

V. The Turn

That is the failed architect, complete. Chapter 3 showed him negligent — building a comfortable zoo that quietly unmakes its inhabitants. This chapter showed him something worse: not merely negligent but captured, his own genius for engineering environments turned against him, his instincts unlocked by supernormal keys, his attention and his appetites optimized by runaway loops that serve replicators indifferent to his survival. For ten thousand years we have held the pen and written badly — first by accident, then by building machines that write worse than accident, because they write on purpose toward targets that are not ours.

It would be an easy place to end in despair, and a dishonest book would, because despair sells. But everything I have shown you in Part II has the same structure, and that structure is the reason for hope. Mismatch, relaxation, supernormal stimuli, runaway maximizers — every one of them is a case of selection pressures acting on us, badly, because no one is steering. And a pressure that no one is steering is a pressure that could, in principle, be steered. The failed architect failed because he held the pen by accident, in the dark, optimizing for the wrong target without knowing he was optimizing at all. The question that opens the second half of this book is whether a species can learn to hold that same pen on purpose — to become the conscious author of the selection pressures acting on its own biology, instead of their victim.

The answer, I will argue, is yes. And astonishingly, the first evidence that it is possible does not come from any future technology. It comes from a hundred-year-old idea in evolutionary biology, a mechanism by which behavior can reach back and rewrite the genome that produced it — by which what an animal chooses to do, sustained across generations, can become what its descendants are born knowing. It is called the Baldwin effect, and it is the first crack of light in this book. Part III begins there.

Register note for this chapter. Tinbergen's supernormal stimuli (Section I) and Fisherian runaway as a mathematical model (Section II) are Established. Whether the peacock's tail specifically is runaway or honest-signal is honestly contested, and I said so; the runaway mechanism itself is not in dispute. The applications — supernormal stimuli in processed food and media, and engagement algorithms as Runaway Maximizers (Sections I, III) — are Inferred: argued analogies resting on established mechanisms, flagged as interpretation. The gene's-eye / "replicator that is not us" lens is a framework, offered as such. The antibiotic-resistance case (Section IV) is Established. The AI framing that closes the chapter is Speculative and forward-looking — a bet about where the argument leads, fenced explicitly, with the good ending named as un-guaranteed. Where I reached past the evidence, I told you.

Chapter 4: The Runaway Maximizer