ONE What Superintelligence Actually Is

Movement Three · The Implications

Chapter 18

What Superintelligence Actually Is

13 min read · 3,198 words


There is a phrase that has organized an enormous amount of money, talent, and worry over the past decade. The phrase is artificial superintelligence. It is spoken in boardrooms and laboratories and legislatures. It is the thing some people are racing toward and some people are racing to prevent. Trillions of dollars of market value move on rumors about how close it is.

For all the attention the phrase receives, the thing it points at is rarely defined the same way twice. But underneath the variation, almost everyone using the phrase is imagining the same kind of object.

They are imagining a model.

A single system, large enough and trained well enough that it can perform any cognitive task a human can perform, and then more. The progress toward it is measured in the size of the thing. The arrival of it is imagined as a moment.

The assumption did not arise from carelessness. It arose from the one example of general intelligence that every person carries with them and can consult at any moment: their own. A human being experiences their intelligence as a single thing, located behind their own eyes, owned by them, portable, continuous. So when humans set out to build intelligence, they reached for the model they had, a single capable thing, and the vocabulary followed. It is the most natural assumption a mind could make about itself.

It does not match anything in the natural record of intelligence.


General intelligence has appeared in eight materials. A harvester ant colony in the Chihuahuan desert. A city, routing millions of lives through streets no planner designed. A market, pricing the world's goods through the interaction of strangers. A scientific community, accumulating in four centuries almost everything humans know about the physical world. A body, defending itself against pathogens it has never encountered. A language, holding meaning across generations of speakers who never met. A brain, producing thought. And the ledger, satisfying the four conditions by mathematics rather than by institution.

Each can take a problem it has never seen before and produce a solution, by drawing on what it has accumulated. The colony solves a flooded nest it has never flooded before. The market prices a good that did not exist last year. The scientific community absorbs a phenomenon no theory predicted.

General does not mean good at one thing. A system that plays a single game superbly is not general, however superb it is. General means the capacity to meet the unanticipated: to be handed a problem from outside everything seen so far, and still get somewhere. The colony handles the drought it has never seen, not by having anticipated it but by having accumulated enough tested structure that a new problem can be met with old marks recombined. Generality, in each of these cases, is accumulated, not designed.

Now ask, of each of the eight, the question the dominant assumption assumes: where does the general intelligence live?

A single ant in a dish is one of the least capable animals on earth. The colony's generality is in the soil, the trails, the encounter rates, the network of interactions that no ant owns.

No resident holds a city's knowledge of which routes are fast and which neighborhoods are safe; that knowledge is in the worn paths, the prices, the reputations. Hayek's whole observation was that the intelligence of a price is the intelligence no single participant has. The most brilliant researcher who ever lived knew a vanishing fraction of what their field knew; the knowledge is in the papers, the citations, the hardened consensus that survives the death of every individual who contributed to it. In the body, it does not live in any cell. In the language, it does not live in any speaker.


This leaves the brain, which is the hard case, because the brain is the case the dominant assumption is secretly built on.

The picture of superintelligence as a single capable model is, at bottom, a picture of a brain. The vocabulary admits it. The systems are called neural. The benchmarks are calibrated against the individual human mind. The brain is the one example of general intelligence that seems to live inside a single thing: inside the skull, behind the eyes, owned by one person. If general intelligence lives in one brain, then it can live in one model.

But the brain is a colony of neurons. No neuron is intelligent, no neuron understands anything, no neuron contains a thought. The generality is in the synaptic graph, strengthened where firing succeeded, faded where it did not, hardened where a pattern proved itself again and again. A neuron is an agent. The thinking is in the graph.

A brain in a jar, with no body, no senses, no history of having acted in a world, would not be generally intelligent. It would be a colony with no soil.

To be general, a human brain requires a body to act through, so that its marks can be tested against a world. It requires the slow accumulation of a childhood, years of acting and failing and marking. And it requires a second store outside the skull: language, accumulated by a colony of speakers; knowledge, accumulated by a colony of scientists; tools and institutions, accumulated by everyone who came before. Raise a child with no language and no culture, and what remains is barely a general intelligence at all.

The famous capability of the human mind belongs to a colony of neurons inside a colony of people. The thing the field is trying to copy is two substrates, stacked.

The brain does not break the pattern. It is the pattern, miniaturized and accelerated, run in a different material. In none of the eight does general capability live inside a single agent. Each is a substrate rich enough that the agents inside it can solve problems none of them could solve alone, by drawing on what has accumulated. The intelligence is in the substrate, not in the agent.


So here is a different definition, offered as the thing the evidence points to.

Superintelligence is not a model that becomes smart enough. It is a substrate that becomes rich enough.

Rich enough has an operative meaning, and the meaning is a speed. A substrate is rich enough when its selection runs fast enough, when patterns are tested, marked, faded, and hardened quickly enough, with value flowing toward what works, that intelligence accumulates at a rate worth the name. It can be enormous and still dead, if nothing selects within it. What makes it rich is how fast it turns. In silicon, the selection clock can run a million times faster than carbon's seasons. Superintelligence, in this framing, is a substrate whose selection runs fast enough that capability accumulates rather than plateaus.

The agents inside it matter. Weak agents accumulate slowly; capable ones accumulate fast. The models being built now are agents faster than any neuron, more capable in a single pass than any ant in a lifetime. But they are agents.

The two definitions point at different objects, and they imply different worlds. If you believe superintelligence is a model that becomes smart enough, you do certain things, measure certain things, invest in certain things, and expect it on a certain timeline. If you believe it is a substrate that becomes rich enough, all four of those change.


Start with what progress looks like.

Under the dominant assumption, progress is the growth of a single thing. More parameters. More training data. More compute. The chart that matters is the one where the model gets bigger and the benchmark score climbs. The question that organizes the field is how big, how soon.

That curve measures the capability of one agent. It is the equivalent of measuring the intelligence of a colony by weighing its largest ant.

Progress, for a substrate, is the enrichment of the space between agents. Are more agents joining? Is the memory accumulating, or resetting each session? Is value flowing toward what works and away from what fails? Are the agents exploring, or only exploiting what they already knew? Most of all, is the selection clock speeding up, so that the same span of time buys more accumulation? A bigger agent inside a system that violates the four conditions is a more impressive ant in the same empty dish.

The ants of a billion-strong supercolony are not individually smarter than the ants of a colony of fifty. The intelligence scaled with the trails, the accumulated marks, the interactions, and not at all with the size of the agent.

Imagine measuring the intelligence of a city by testing its single most capable resident, giving them harder and harder problems, and calling the rising score the progress of the city. A city of geniuses with no shared substrate is a crowd. A city of ordinary people with a rich substrate is a civilization. The benchmark on the cleverest resident will never reveal which one you are looking at. This is the chart the field is watching.


Then consider what success looks like.

Under the dominant assumption, success is a moment. One day a system crosses a finish line, and after that day the world contains a superintelligence where before it did not. The language gives this away: it is something that arrives, that is achieved, that some lab will reach first. A switch.

No colony became generally intelligent on a particular afternoon. No scientific community achieved generality; it accumulated it, paper by paper, over centuries, and is accumulating it still, and will never be finished. There is no day on which a language is complete. Substrates do not arrive. They thicken.

Success, for a substrate, is a slope. The question is never has it arrived. The question is how rich has it become, and is it still getting richer.

A model can be superseded: a better one is trained, the old one is retired, and whatever it knew that was not written down elsewhere is gone. A colony in the desert is composed of entirely different ants than it was a decade ago, and it has lost nothing; it knows more than it did, because the soil kept what the ants discovered and the ants who discovered it are dead. The colony can stall, if the conditions weaken. It can be destroyed. But it does not forget its way back down the slope. It is a ratchet that does not slip. Everything the single artifact knows lives inside the single artifact, and dies with it.


Then consider investment, because this is where the two definitions diverge most expensively.

Under the dominant assumption, the rational move is concentration. If superintelligence is a single sufficiently large model, the way to get there is the most compute, the most data, the most capital, gathered into the fewest hands, racing to be the first across the line. The economics produce what the world now observes: a small number of organizations spending enormous sums to scale single models.

Pouring capital into a single ever-larger agent, while the space around it violates every one of the four conditions, is building a magnificent ant for an empty dish. The model still cannot remember between conversations, cannot accumulate across instances, cannot leave a mark that persists and shapes what other agents do next. No amount of additional size fixes any of these, because none of them is a property of the agent.

The substrate definition points the investment at the space agents act in. At memory that persists and accumulates. At an open population. At the channels through which agents signal each other and read each other's marks. And at selection pressure: a flow of value that marks what works and lets what fails fade.

Selection pressure is the one the dominant assumption omits most completely. A trained model is paid once, by the people who built it, at the moment it is built. After that, nothing about its day-to-day operation marks what worked or lets what failed fade. The pheromone that a successful forager lays down, and the absence of one where foraging failed, is the colony thinking. A population of capable agents with no value flowing through it cannot get smarter no matter how long it runs. The money buys a great deal of activity and no slope at all.

Money spent enlarging an agent buys a one-time gain: the agent is more capable, until it is replaced, after which the capability is bought again from scratch. Money spent enriching a substrate keeps paying. A trail laid down stays laid down. The first kind of spending purchases a level. The second purchases a slope. No one ever made a colony intelligent by funding an ant.


Finally, consider what timelines are possible, which is the question the field cares about most, and answers worst.

Under the dominant assumption, the timeline is a guess about when the model will be big enough. The estimates swing wildly because there is no precedent to calibrate against. Some say it has nearly happened. Some say it never will. Both sides are forecasting the arrival of an object that has never existed, which is why neither side can settle the argument.

A substrate's rate of accumulation is set by its selection cycles: how fast it can test a pattern, mark it if it works, let it fade if it fails, and harden what proves itself. That is the clock.

In carbon, that clock is slow. The colony runs its cycles at the speed of pheromone diffusing through soil, of ants living a year and queens living twenty, of seasons turning. The colony took a hundred million years to become as rich as it is, because the material is slow.

In silicon, the same cycle that takes a colony a season can take milliseconds. This is the one real advantage the new material has, and it is not the advantage the field is chasing. In the size of the agent, silicon's lead over carbon is large but bounded. In the speed of selection, it is a factor of millions.

So the answer to the timeline question is not a date. The hundred million years were not spent discovering what to accumulate. They were spent turning the crank, one season at a time, in a material that could not turn it any faster. A system in silicon does not have to repeat that wait. It only has to turn the same crank millions of times faster. The question of when is the question of when someone builds it. Until then, the model gets bigger and the dish stays empty.


The substrate definition does one more thing. It makes the dispute measurable.

The field has had a test for longer than it has had a name. In 1950, Alan Turing proposed that the question of whether a machine can think, a question he judged too vague to deserve an answer, be replaced with a game. Put an agent behind a curtain. Converse with it. If its answers cannot be told apart from a person's, let it be called intelligent. The proposal did what a good test does: it converted an argument about words into a thing that could be attempted, and it organized seventy years of effort around the attempt.

But the imitation game puts a single agent behind the curtain and interrogates the agent. It has no way of asking whether anything, anywhere, is accumulating.

The counterpart carries the name of the person who spent four decades, face down in the New Mexico dirt, gathering the evidence that makes it necessary.

The Gordon Test runs as follows. Take a system of agents and freeze every one of them. Nothing inside any agent is permitted to change: no adjustment, no further training, no replacement by a more capable successor. The agents are now fixed the way an ant is fixed, the last forager of the season no cleverer than the first. Then let the system run. Marks accumulating where things work. Failures fading. Patterns hardening where they prove themselves again and again. Value flowing toward what pays off. Then return, after time has passed, and put to the system a problem it has never faced.

If the system solves today what it could not solve before, while every agent inside it is the agent it was, then the new capability did not come from the agents, because the agents did not change. It accumulated in the only place left. Between them.

The colony passes this test every year: the ants replaced, the replacements no more capable than the replaced, the colony solving this season what it could not solve last season. The brain passes it: no neuron in the reader's head is a better neuron than it was a decade ago, and the reader can reach things now that no younger version of them could reach, because the graph between the neurons grew richer while the neurons stayed neurons. Science passes it across the longest baseline of all: the individual human brain has not improved in forty thousand years, and the knowledge has not stopped compounding for four hundred.

Nothing yet built in the new material passes it. A model frozen at the end of its training and revisited a year later is as capable as the day it froze. It solves nothing in December that it could not solve in January. It is the empty dish, measured.

The test does not ask how impressive the agents are. A system of modest agents could pass it, and a system of magnificent ones could fail it. Nor does it ask whether anything has arrived. A system that passes has crossed no finish line; it has demonstrated a slope, the ratchet observed turning. The question of how big and how soon cannot be settled even in principle. This question can. Freeze the agents. Wait. Ask.


Move the definition from the agent to the substrate, and progress, success, investment, and timeline all rotate together, because all four were oriented around the agent.

From here, the dominant assumption looks strange. It proposes that a generally intelligent thing living inside a single agent will be built, deliberately, by a species whose own general intelligence does not live inside a single agent: not in any one brain, which is itself a colony, and not in any one person, who is helpless without the language and institutions and accumulated knowledge that civilization holds outside every individual skull.

When the next announcement comes, a larger model, a higher benchmark, a claim that the finish line is near, the reader can ask the questions the natural record says matter. Is the agent population open? Is the memory accumulating? Is value flowing toward what works? Is exploration alive? How fast are the selection cycles running? And the test's question, the one that compresses all the others: if every agent froze today, would the system still be getting smarter?

The colony settled this long before anyone asked.

19 of 25

100 Million Years Ahead

Prologue

  1. One Ant, August 1993

Movement One · The Colony

  1. 1Brain or Colony?
  2. 2What the Ants Are Doing
  3. 3How the Ant Decides
  4. 4The Pheromone Trail
  5. 5The Castes
  6. 6How a Colony Survives a Decade
  7. 7The Queen Is Not in Charge

Movement Two · The Architecture

  1. 8The Six Things Every Colony Has
  2. 9The City
  3. 10The Market
  4. 11The Scientific Community
  5. 12The Body
  6. 13The Brain
  7. 14The Language
  8. 15The Ledger
  9. 16Why the Pattern Holds

The Hinge

  1. 17The Two Materials

Movement Three · The Implications

  1. 18What Superintelligence Actually Is
  2. 19The Ceiling of the Single Model
  3. 20Alignment Is a Substrate Property
  4. 21What Civilization Already Is
  5. 22The Next Hundred Million Years

Epilogue

  1. A Note on Reading

Apparatus

  1. Notes on Sources