AI Can Improve Itself. The Hard Question is How Far.
Recursive self-improvement (RSI) has become a term thrown around in AI conversations with no real confidence that everyone understands its meaning.
The usual story is easy enough to tell. Build an AI model capable of improving AI. The improved AI is now better at improving AI, so it produces an even better version. That version improves the process further. Repeat. If the gains compound in the right ways, then we get an “intelligence explosion” — or so the argument goes. The goal is a transition from systems that are roughly at human level to systems vastly beyond us, perhaps in a surprisingly short period of time.
The story seems reasonably coherent. But several important assumptions are buried inside it, and those assumptions matter.
The idea itself is much older than the current AI boom. In 1965, the mathematician Irving John Good published an essay titled Speculations Concerning the First Ultraintelligent Machine. Dr. Good worked with Alan Turing at Bletchley Park, and versions of the argument appeared in talks he gave several years before the essay was published. His central observation was wonderfully simple: designing intelligent machines is itself an intellectual activity. If we ever built a machine that exceeded humans at intellectual work, then perhaps one of the activities it could perform better than us would be designing intelligent machines itself. Dr. Good called the resulting possibility an “intelligence explosion.”
Some six decades later, parts of that loop are no longer theoretical. AI systems can write code, generate training data, optimize algorithms, construct evaluations, and increasingly participate in the engineering process that produces the next generation of AI systems.
That notwithstanding, there’s significant conceptual distance between a system learning to play better chess and an autonomous AI that is capable of designing, training, evaluating, and deploying a more capable successor.
Before deciding whether RSI leads inevitably to super-intelligence — or whether the entire idea is overblown — it helps to separate the hype from the mechanics.
Self-Improvement Is a Matter of Degree
“Can an AI improve itself?” sounds like a yes-or-no question. It isn’t.
A useful example comes from AlphaZero, the system Google DeepMind developed to learn games such as chess, Go, and shogi.
AlphaZero was given the rules of the game, but it wasn’t trained on libraries of expert games or handed a collection of human strategies. Instead, it played games against itself, learned from the outcomes, became stronger, and consequently generated stronger opponents for subsequent games.
That’s a genuine feedback loop. And it worked spectacularly well.
But there’s an important distinction. AlphaZero didn’t rewrite its own learning algorithm, construct a better version of that algorithm, or turn the task of producing the next version over to its progeny. AlphaZero improved itself, but it didn’t improve the machinery by which it improved itself.
DeepMind’s AlphaEvolve went further in that direction. It combined large language models with automated evaluators and an evolutionary search process to discover better algorithms. Among other results, AlphaEvolve found an optimization that sped up a matrix-multiplication kernel used in training Gemini by 23%, translating into roughly a 1% reduction in Gemini training time. That might ostensibly seem like a modest improvement, but the conceptually remarkable part is that AI helped improve machinery used to build AI.
An even more direct example comes from a research system called the Darwin Gödel Machine. The system modifies the code of a coding agent, evaluates the resulting descendants, and retains modifications that work. In reported experiments, successive variants substantially improved their performance on software-engineering benchmarks.
These systems occupy different points along the same path.
If an AI can find better answers, generate better training data, write better code, improve parts of its own software harness, and/or modify an agent that performs tasks, then perhaps (eventually) it could design and train a successor model that is itself better at performing AI research.
Self-Improvement Needs a Scoreboard
A significant point of leverage for AlphaZero is that with chess, it’s relatively easy to keep score. There’s no need to convene a committee of chess grandmasters to determine how much additional chess understanding AlphaZero has acquired. Rather, we can let different versions play each other and track the outcomes. That property makes chess unusually amenable to self-improvement.
The broader principle is that recursive improvement becomes much easier when “better” can be measured cheaply, repeatedly, and with high confidence.
Some optimization problems share this property. If the objective is to reduce latency, minimize chip area, compress a file, or perform a computation using fewer operations, the score can often be measured directly.
Programming is in murkier territory since there may not be a consistently reliable way to determine whether software has improved and by how much. To be clear, we can look for signals such as whether the code compiles, whether unit tests pass or fail, whether performance benchmarks improve, etc. But even bad software can pass some of these checks.
Mathematics occupies another interesting part of the gray zone. Formal proof systems such as Lean can provide something very close to chess’s scoreboard: a proposed proof either checks or it does not. That makes formal mathematics unusually amenable to machine-generated search. Of course, a proof checker cannot tell us whether a particular theorem represents a meaningful contribution to the field.
We can mechanize validity, but we cannot mechanize taste.
Scientific research presents even harder evaluation problems. Experiments provide contact with reality, which is enormously valuable, but they can be expensive, slow, noisy, or difficult to interpret. And the hard part of science is not always answering a question. Sometimes it’s deciding which question deserves to be asked.
Other disciplines are murkier still. One of the more obvious examples is literature. What, exactly, goes into the loss function for James Joyce’s novel Ulysses? Readability? Average sentence length? Percentage of readers who actually finish it (versus those who have claimed to)? Anyone who considers Ulysses to be among the greatest works of literature would likely suggest that none of the aforementioned attributes are opposite.
Optimize Joyce aggressively for any one of those variables and one could quite plausibly “improve” the book by systematically removing the qualities for which anyone might still read Joyce.
The problem here has a close connection to what is commonly called Goodhart’s law, named after the economist Charles Goodhart, who originally observed that statistical relationships used for economic policy tend to break down once policymakers begin targeting them directly. Anthropologist Marilyn Strathern later gave the idea its more well-known formulation: When a measure becomes a target, it ceases to be a good measure.
The underlying idea is straightforward. A metric is only a proxy for what we actually care about. Once an optimizer begins pushing aggressively on the proxy, it may discover ways to improve the metric without improving the underlying objective.
This consideration is particularly salient for RSI because evaluation can become the bottleneck. For example, imagine that an AI system generates 10,000 candidate modifications to itself every hour. That might sound impressive, but suppose that determining whether each modification is genuinely useful requires a week of experiments.
The improvement loop is not running 10,000 times an hour. The evaluator sets the pace. Although one can try to automate the evaluator, another problem now appears: what if the evaluator is slightly wrong?
The more aggressively the system optimizes against an imperfect evaluator, the more likely it is to discover the evaluator’s blind spots.
The field of control theory offers us some lessons here. Researchers in adaptive control learned decades ago that systems capable of updating their own models can behave very well while the assumptions behind those models hold, and considerably less well when reality veers away from them.
The lesson is narrower than saying that control theory somehow proves RSI must proceed slowly. The more useful lesson is that the rate at which a system can safely adapt depends in part on the rate at which it can obtain trustworthy evidence that the adaptation worked.
That perspective also changes how we should think about benchmarks. A benchmark is useful because we hope that performance on it tells us something about a broader capability that we actually care about. If a model reaches 100 percent, we have certainly learned that it can solve the benchmark. We have not, however, learned that it is proficient at the broader capability the benchmark was intended to measure.
The distinction becomes especially important once a benchmark saturates. Two models might both score 100 percent even though one is substantially more capable outside the narrow set of problems being tested. At that point, the benchmark has exhausted its ability to tell us which system is better. Worse, if we continue optimizing against it, we may simply get systems that are increasingly well adapted to the benchmark rather than increasingly capable in the broader sense we care about.
Sustained recursive improvement therefore requires the evaluations to improve along with the systems being evaluated. As an AI becomes better, it needs harder, broader, and more discriminating tests. Eventually, an RSI system may need to help construct evaluations that expose weaknesses its current version does not yet know how to overcome.
That is a much harder problem.
Better Does Not Mean Much Better
Another assumption buried in many discussions of RSI is that if systems keep improving, the gains compound in dramatic ways. That need not be true.
Consider, for example, the sequence 1 → 1.1 → 1.11 → 1.111 → 1.1111 → …
Every term is larger than the one before it. Improvement never stops. Yet the sequence never reaches 1.2. That simple example captures an important point: an infinite number of improvements need not add up to a large improvement.
Recursion tells us something about the structure of the process. It says much less about the magnitude of each step.
There are many plausible reasons why gains might shrink over time. The easy algorithmic improvements may be found first. Coding may cease to be the bottleneck and experimentation may take its place. Automate enough experiments and compute may become scarce. Improve compute efficiency and chip fabrication may become limiting. Improve chip design and perhaps some combination of energy, cooling, and interconnect bandwidth becomes the constraint.
There’s a useful analogy here from the field of computer architecture.
A principle known as Amdahl’s Law says that speeding up one part of a computation produces diminishing benefits once some other part of the computation dominates the total running time. If 90% of a workload becomes essentially free, the remaining 10% suddenly matters enormously.
Research can behave the same way.
If AI makes one stage of the research process nearly free, improving that stage again contributes little unless the adjacent bottlenecks move as well.
To be clear, those bottlenecks are not immutable. A sufficiently capable AI might design better algorithms, better compilers, better hardware, better memory systems, or better research workflows. It may effectively obtain more useful computation from the same physical resources.
But, removing one constraint does not eliminate the existence of constraints. The relevant question is therefore not merely whether each generation is better than the one before it.
It is how much better.
Feedback Can Make You Worse
There is a more uncomfortable possibility.
Recursive systems do not merely risk making smaller and smaller gains. Under some conditions, the feedback itself can degrade the system.
In a 2024 Nature paper, Shumailov et al. studied what happens when successive generations of generative models are trained indiscriminately on data produced by previous generations of models.
The process degraded.
Information about the original data distribution progressively disappeared. Low-probability events were especially vulnerable. As generations accumulated, the models became increasingly detached from the distribution represented by the original human-generated data.
The intuition is not difficult to see.
Imagine training a generation of novelists exclusively on novels produced by the previous generation. Then train the next generation exclusively on those novels. And so on.
The unusual turns of phrase, obscure references, minority styles, strange experiments, and genuinely surprising constructions are precisely the elements most likely to disappear first. Over time, the system can converge toward an increasingly polished average of its own previous averages.
Researchers have given this family of effects an appropriately gruesome name: model autophagy. The model consumes the products of other models, including ultimately its own informational progeny.
There’s an important qualification here. Model collapse does not establish that RSI inevitably collapses. The result concerns particular regimes in which generated data recursively replaces real data. Other work shows that retaining sufficient real data alongside synthetic data can substantially mitigate the effect.
Still, model collapse is a reminder that recursion is not a magical elixir: feedback can amplify errors just as readily as improvements. The arc of recursion does not always bend towards melioration. What matters is what gets fed back into the loop — and what gets lost along the way.
The same point applies more broadly.
An AI that learns increasingly from its own judgments, its own generated experiments, its own synthetic data, and its own evaluations can become highly internally consistent without becoming correspondingly accurate about the outside world.
A self-referential AI system that learns increasingly from its own judgments, its own generated experiments, its own synthetic data, and its own evaluations could become highly effective within the artificial milieu it has constructed, while becoming increasingly poorly calibrated to the world outside it.
Every Loop Hits a Wall
One common response to the more dramatic RSI scenarios is that AI cannot improve indefinitely because there are real bounds on the resources required to do so.
AI systems do not have infinite access to processors, memory, energy, bandwidth, or data. They also need to produce results within a reasonable time period. Therefore, RSI will eventually hit a wall. But the existence of that wall tells us surprisingly little. The consequential question is where it sits.
Suppose the relevant constraints become binding after only modest improvements, leaving AI systems materially but not qualitatively more capable than today’s frontier models. That would be consequential, but it would look more like accelerated technological progress than a discontinuity in human history.
Alternatively, the gains could compound long enough for AI to acquire qualitatively new capabilities — perhaps automating large portions of research, substantially accelerating scientific discovery, or improving the process of AI development itself — before those constraints become binding.
In both cases, RSI is bounded. Yet the outcomes could scarcely be more different.
So, to return to where we started, the useful way to think about RSI is not as a binary event in which an AI suddenly “becomes self-improving.” Self-improvement comes in degrees. It works best where evaluation is reliable. Repeated improvements can become progressively smaller. Feedback can sometimes degrade rather than improve the system. And every recursive process eventually encounters constraints.
Perhaps the evaluator problem proves stubborn. Perhaps research taste remains difficult to automate. Perhaps gains shrink rapidly. Perhaps synthetic-data loops degrade without enough contact with external reality. Perhaps compute, energy, or fabrication constraints arise much sooner and to a greater degree than enthusiasts expect.
Or perhaps enough of those problems are tractable that AI closes most of the research loop before the constraints become binding.
We do not yet know.
That uncertainty is precisely why recursive self-improvement is worth understanding carefully. There is a wide gulf between “an intelligence explosion is inevitable” and “RSI is just hype,” and the most consequential futures may lie somewhere inside it.
AI need not become unbounded, and recursive self-improvement need not become explosive, to profoundly reshape science, technology, and society. The threshold that matters may come much earlier: the point at which AI begins to change not merely what we can discover and build, but the rate at which discovery and invention themselves proceed.