For most of the artificial-intelligence boom, the central question was how quickly AI could become more capable. Increasingly, the question is becoming whether humanity can control what it is building.
That debate has exploded into public view when Jacob Coxon, a researcher who worked at both OpenAI and Anthropic, resigned from Anthropic and accused the leading laboratories of racing towards self-improving superintelligence without adequate safeguards. Anthropic researcher Evan Hubinger publicly backed the underlying concern, while CEO Dario Amodei subsequently called for the industry to slow the rate at which frontier capabilities advance.
The warnings are extraordinary because they are coming not primarily from outsiders, but from some of the people developing the technology.
What are they actually afraid of?
Today’s large AI models are not autonomous superintelligences. They remain dependent on human-built infrastructure, computing resources, permissions and instructions, and they make substantial errors.
But their capabilities are advancing rapidly. Frontier systems can increasingly write software, operate computers, conduct sophisticated cyber tasks, reason through scientific problems and function as agents capable of completing multi-step assignments.
The feared threshold is therefore not simply an AI becoming “smarter than a human.” It is the combination of intelligence, autonomy and the ability to improve AI itself.
If AI becomes sufficiently capable at AI research, it could help design better models, which could then accelerate development of still better models. This possibility—recursive
Today’s large AI models are not autonomous superintelligences. They remain dependent on human-built infrastructure
self-improvement—is at the centre of the current debate. Amodei has warned that capability development could eventually outrun humans’ ability to understand and control these systems.
The danger envisaged by the most concerned researchers is therefore a feedback loop:
Humans build AI → AI accelerates AI research → more powerful AI arrives faster → human oversight cannot keep pace. Nobody has demonstrated that this chain will occur. The disagreement is over whether even a relatively small probability of such an outcome justifies slowing development.
Two camps are emerging
Amodei wants independent evaluators given extensive access to frontier systems, stronger government oversight and eventually coordinated agreements among leading developers. Anthropic’s existing safety framework already focuses on risks including cyber operations, chemical and biological weapons and AI systems capable of accelerating AI research itself.
The recent controversy has produced unusual agreement among competitors. Sam Altman supported pacing frontier development, while other prominent AI figures have backed greater safety coordination.
But there is another camp. Critics argue that predictions of human extinction remain speculative and that slowing technological progress could itself impose enormous costs. AI could accelerate drug discovery, scientific research, productivity and economic growth.
There are also suspicions about regulation. Smaller AI companies worry that expensive safety requirements could entrench today’s largest laboratories by making frontier development prohibitively costly for competitors. Cohere CEO Aidan Gomez has characterised coordinated restrictions among the largest laboratories as potentially cartel-like. Other industry leaders argue that continued development is necessary both to understand AI better and to remain competitive internationally.
So this is not simply safety versus recklessness. It is a dispute about how to manage a technology whose ultimate capabilities remain uncertain.
We’ve been here before
The closest historical comparison is nuclear technology.
When scientists unlocked nuclear fission, humanity discovered a technology capable of delivering extraordinary benefits through energy and science while simultaneously creating weapons capable of destroying cities.
The Manhattan Project demonstrated another problem: once one state possessed the knowledge, geopolitical competition made simply stopping development extremely difficult. AI faces a similar dilemma, but with one crucial difference.
Building nuclear weapons requires specialised materials, enormous industrial facilities and detectable infrastructure. Advanced AI ultimately requires computing power, electricity, chips, algorithms and expertise—resources that are expensive but potentially much more widely distributed.
AI could therefore combine something resembling nuclear technology’s strategic importance with something closer to software’s ability to proliferate. That makes controlling it potentially much harder.
The geopolitical trap
Even if American AI executives genuinely believe development should slow, each company fears competitors continuing. More importantly, Washington fears Beijing continuing.
China therefore hangs over virtually every discussion about AI regulation. American policymakers face a strategic dilemma: restrictions designed to make AI safer could theoretically allow China to close the technological gap. Conversely, an unrestricted race with China could encourage both countries to prioritise capability over safety. Anthropic itself explicitly argues that democratic countries should retain an advantage over China even while advocating stronger safeguards.
This creates the classic security dilemma. Neither side wants to be the one that slows down first. And that may be the most important thing to watch going forward. The decisive question is no longer simply how powerful AI becomes.
It is whether governments can construct rules for a transformative technology before geopolitical competition makes restraint impossible.
