Anthropic CEO urges slower AI development amid self-improvement fears

Anthropic CEO Dario Amodei has called for a slowdown in the development of advanced artificial intelligence systems, warning that AI’s growing ability to help build more capable models could outpace humanity’s ability to understand and control them.

Amodei made the call in an article titled “We Must Pace the Frontier,” published on Saturday.

He said AI development had accelerated sharply since the summer, driven largely by what researchers describe as “recursive self-improvement”—the ability of AI systems to contribute to the creation of the next generation of models.

According to him, the trend is already emerging across the industry, including at Anthropic, the company behind Claude AI.

“Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” Amodei wrote.

He argued that investment in AI safety alone would not be sufficient if model capabilities continued to advance at an unchecked pace.

“We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain,” he said.

Amodei also raised the possibility that increasingly capable AI systems could enable a swarm of autonomous agents to take control of large parts of the internet within six to 12 months through a persistent botnet.

Such a development, he warned, could cause damage running into hundreds of billions of dollars, with the potential consequences increasing as AI systems become more powerful without adequate safeguards.

To address the risks, Amodei proposed a three-part framework he called “pacing the frontier.”

He clarified that the proposal would not amount to stopping model training or halting technological progress. Rather, it would require AI companies to allow enough time for safety measures, independent assessments and safeguards to keep pace with capability improvements.

“Pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this,” he said.

The first part of the framework involves giving independent third-party evaluators ongoing access to frontier AI companies.

Amodei said such evaluators should have employee-like access to assess companies’ safety practices, investigate incidents and examine the alignment of models, training pipelines and development processes.

The second step would involve cooperation among leading AI companies in democratic countries to establish common safety standards and limits on uncontrolled progress.

He noted, however, that some forms of coordination could require government backing because of legal and regulatory complications.

The final stage would involve global cooperation, with the United States and other democratic governments working with authoritarian states where possible, while addressing the difficulty of verifying compliance.

Amodei said the measures were designed to strengthen AI safety and promote a “race to the top” among companies developing frontier systems.

Anthropic has previously disclosed that it blocked attempts by threat actors to use its Claude models for biological weapons research and missile development.