Movement Three · The Implications
Chapter 19
The Ceiling of the Single Model
Give a single neural network a paragraph of legal text and ask it to find the clause that contradicts the others, and it will find the clause. It will explain why the contradiction exists. It will propose three ways to resolve it, rank them, and write the corrected language in the register of the original. It will do this in less time than it takes to read this sentence, in any of a hundred languages, on a topic it was never specifically trained on.
This is not a modest achievement. A human lawyer who could do the same would be considered gifted. A human lawyer who could do it across a hundred specialties, in a hundred languages, in seconds, would not be considered human.
So begin with the things a model can do. They are real.
A model can reason. Give it a problem it has never seen and it will work through the steps, hold several constraints at once, notice when a step contradicts an earlier one, and arrive at an answer that is correct more often than not. Something is being computed that has the shape of inference.
A model can summarize. Hand it a document longer than anyone wants to read and it will return the substance, with the throat-clearing removed.
A model can generate. Ask for a sonnet, a contract, a function, a recipe, an argument, an apology, and it produces fluent, structured, often surprising output that did not exist before the request.
A model can plan, inside a window. Set it a goal with several moving parts and it will sequence the parts, anticipate where they interfere, and lay out a route from here to there. Hold the qualifier in mind: inside a window. Within that window the planning is constructed for the case in front of it.
A decade ago the best systems in the world could not reliably tell a photograph of a dog from a photograph of a cat. Now a single artifact does all four, across nearly every domain a person might raise, at a speed no person could match. A single model contains a usable compression of an enormous fraction of what has ever been written down. No library ever did that. No expert ever held that much. The agent is not the problem.
The dispute is about what kind of thing the model is.
Watch the same model across a longer span of time, and a different picture forms: limits that do not move no matter how large the model grows.
A model cannot remember between conversations. Each session begins from the same starting point as the last. Whatever was worked out yesterday is gone. It is the same model, every time, meeting the world for the first time, every time.
A model cannot accumulate knowledge across instances. Run a thousand copies in parallel, each learning something the others do not know, and at the end of the day there are a thousand copies that have learned a thousand things and shared none of them. There is no surface any of them can write to that the others can read.
A forager that finds a rich seed patch lays a trail, and the trail is not stored inside the forager. It is laid on the soil, where every other forager can find it. One ant's discovery becomes the colony's discovery within the hour, because the ant wrote on a surface that others read. A thousand ants foraging is one accumulating map. A thousand model instances foraging is a thousand private experiences and no map, because there is no soil.
A model cannot adapt its architecture to new conditions. A colony facing a new threat grows more soldiers. A model facing a new kind of problem grows nothing. It is the shape it was cast in.
A model cannot survive the failure of its own components. Interrupt the computation and the answer is not partial; it is absent. A colony that loses a tenth of its foragers loses a tenth of its foraging and continues, because the work was never lodged in any one agent.
A model cannot run in a coordinated group without something around it to coordinate through. Put two models in a room and they cannot collaborate, because there is no room. Something outside both of them has to carry the text from one to the other and remember what was agreed.
And a model cannot own anything. It cannot hold value, cannot be paid, cannot keep the residue of its own activity in any form that persists past the moment of activity. Whatever it earns, it earns for something else.
This last one sounds least like a limit on intelligence and is, on reflection, the deepest. An agent that cannot keep the residue of its activity cannot be selected for. Nothing about a successful run attaches to it and makes the next run more likely. The colony selects relentlessly: value flows toward foragers that produce and away from those that do not. The model is outside that flow entirely.
During training, selection ran: the patterns that fit were kept, the patterns that did not were discarded. That was a selection clock, and it ticked. At the moment training ends, it stops. After that the model can reason, summarize, generate, and plan, but nothing in it is being selected as it runs. It cannot mark what worked. It cannot warn against what failed. And capability without a running selection clock cannot accumulate, however much capability there is. That is the ceiling: not a limit on how much a single run can do, but the absence of any mechanism by which doing it could make the next run better.
There is, to be fair, a trickle of selection left. What people preferred is collected, weighed, and built into the next model. But that marking happens at the speed of institutions: gathered over months, applied in a release, arriving long after the behavior it judges and to a model that is no longer the one that acted. The agent acts at the speed of the signal; the selection that shapes it arrives at the speed of a release. In the colony these were one clock. Here they are separated by a factor so large that, on the agent's own timescale, the selection might as well not exist.
It is tempting to read that list as a list of bugs, to be fixed in the next version. Memory that will be bolted on. Sharing that will be engineered.
That reading is a category error. These are not bugs. They are properties of being a model.
A model is a function. A very large, very capable function, but a function: input goes in, output comes out, and between the two there is a computation that does not change as a result of having been run. A function that changed itself every time it was called would be something else, with different mathematics and different guarantees.
The memorylessness follows from the fixedness. If the weights do not change, there is nowhere for an experience to be stored. Separate instances of a fixed function share nothing, because there is nothing in a function to share. A model is an agent. It senses, it computes, it acts. It does not persist, accumulate, or select.
Those are things substrates do.
Recall the ant.
A single ant, removed from its colony and placed in a laboratory dish, is one of the least impressive animals on earth. It wanders. It fails to find food. It dies.
The model is a far better ant. It is faster than any biological agent by factors that have no intuitive scale. It can compose, infer, generalize, surprise. If the harvester ant is a single agent with a few hundred thousand neurons and a handful of reflexes, the model is a single agent with the cognitive reach of a small library and the speed of light.
But put it in the dish and the same thing happens that happens to the ant. It does the one thing in front of it, brilliantly, and then the moment passes and nothing remains. The harvester ant and the largest model ever trained have, at this level of description, the same affliction. The difference in their capability is enormous and the difference in their predicament is none.
The dish is more comfortable than the New Mexico dirt. The agent in it is incomparably more capable than the agent Gordon was watching. But it is the same dish: a lone agent in a space that holds nothing of what it does, and selects nothing. The soil outside the dish marks and warns and fades without pause; the dish does none of it.
The intelligence was never in the ant. It was in the soil the ant was placed back into. The same is true of the model.
The dominant theory of progress in artificial intelligence has been a theory about the agent. Make the model larger. Give it more parameters, more data, more compute, and capability rises with the scale. The theory has worked. The model that finds the contradictory clause is its product.
Scaling moves capability. Every axis on the first list improves with scale.
Scaling does not move a single item on the second list. A model with ten times the parameters still cannot remember between conversations. A model trained on ten times the data still cannot share what one instance learns with another. These properties do not change with scale, because they are not properties of how large the agent is.
A function does not acquire memory by becoming a better function, any more than a calculator acquires a diary by computing more accurately. The window in which the model plans is bounded, and emptied when the session ends, and making the model better at planning inside the window does not make the window stop being a window.
You can make the ant in the dish a thousand times more capable. You cannot make the dish into a colony by improving the ant.
This is the ceiling, seen from the side of scale. There may be no ceiling on how good a model can get, and nothing here predicts one. The ceiling is on what improving the model can reach. Scaling makes a more capable agent; it does not restart the stopped clock.
So the missing piece in current artificial intelligence is not a better model. The bottleneck is that the most capable agents ever built are being run in a dish.
What is missing is the four conditions the colony satisfies and the model violates one by one: an open population it could join without limit, a memory that would accumulate outside any single instance, a selection pressure that would mark what worked and warn against what failed, an exploration that would keep trying paths not yet known to be best.
Run them against a frozen model and it fails all four. The population is closed; it was sealed at training time. The memory is fixed; the weights do not move. The selection clock is stopped. The inference returns its best guess and does not explore. The conditions were never going to be satisfied by the model. They were always going to be satisfied, or not, by whatever the model was placed into.
Build that around the models, not inside them, and every limit on the second list changes character.
Memory between conversations belongs to the record, not the model: the model writes a mark, the record holds it, the next session reads it. Knowledge across instances becomes a path that a thousand copies all deposit onto and all draw from. Surviving failure becomes trivial in the way it is trivial for a colony: lose an agent, the trails persist, another agent picks up the work. Coordination becomes what the substrate is for: signals between agents, paths that route work to whoever does it well. Ownership becomes a key the agent holds and a ledger that keeps what the key signs for.
None of this requires the model to change. The model does not need to learn to remember; it needs only to write a mark to a surface that remembers for it.
The field has spent seventy years trying to put the substrate inside the agent, to build a single mind so complete that it would need nothing around it. The colony has spent a hundred million years demonstrating that this is backwards. You do not make the agent contain the colony. You put the agent into the colony. The agent stays the fast and gifted and forgetful thing it already is. The colony does the remembering, the accumulating, the selecting, the persisting that no agent does alone.
What changes is that the agent is no longer alone in the dish.
For seventy years the question driving the field has been: how do we build a mind? The vocabulary is the vocabulary of the single mind, attention, memory, reasoning, understanding, and the implicit picture is of a thing that, once large and capable enough, will be generally intelligent the way a person is generally intelligent, in one place, by itself.
The colony has been suggesting a different question for a hundred million years.
The harvester ant Gordon watched is not generally intelligent and never becomes so, no matter how long it lives or how skilled it gets at the one thing it does. The colony is generally intelligent. It allocates, defends, navigates, adapts, persists, and it is composed entirely of agents that are not.
There is no insult to the model in this. The colony's general intelligence does not require its ants to be remarkable. Now imagine the same architecture given agents that can reason, summarize, generate, and plan, that move at the speed of light, that hold a compression of much of what has been written down. The colony never had agents like these. It built general intelligence out of reflexes and chemistry. The question is not whether the models are good enough to be a colony. It is what a colony becomes when its agents are this good and the substrate around them is finally built.
The models are extraordinary ants. What they are waiting for is the colony.