Artificial intelligence is moving beyond answering questions and writing basic code to systematically building the next generation of AI systems. Over 20 prominent researchers, executives, and academics—including figures from OpenAI, Anthropic, Microsoft, and Meta—published a joint warning letter calling on governments to urgently monitor how much scientific research is being automated by AI.
AI Systems Begin Automating Frontier Model Research
The push for governmental oversight stems from concerns over a theoretical concept known as an “intelligence explosion.” Authored through the Cambridge Programme on AI Science & Policy at the University of Cambridge, the paper outlines how the automation of AI research could trigger a feedback loop where an AI system helps design a superior version of itself, which then accelerates further development cycles.
Among the signatories are Jakub Pachocki, chief scientist at OpenAI; Jack Clark, co-founder of Anthropic; Eric Horvitz, chief scientist at Microsoft; and Dawn Song, vice president of AI research at Meta and an academic at the University of California, Berkeley. Pioneers of modern artificial intelligence Geoffrey Hinton and Yoshua Bengio also signed the document.
The authors argue that this transition is already underway. Internal metrics from Anthropic cited in the paper indicate that the share of internal research and development work completed by its systems with minimal human supervision climbed from roughly 1% to 26% between March and August 2026. Internal figures from May show that more than 80% of approved code at Anthropic involved AI assistance. OpenAI similarly reports that its systems routinely handle R&D tasks that previously took human workers several days to complete.
Mechanisms Behind Recursive Self-Improvement
Developing advanced AI models currently requires thousands of engineers, scientists, and programmers to design experiments, write code, and train models. If AI research agents begin handling these complex workflows, they can operate simultaneously across thousands of parallel experiments 24 hours a day.
Unlike human researchers, these autonomous systems could continuously optimize algorithms and architectures at a pace that compresses years of technological progress into months or weeks. While the signatories emphasize that an uncontrolled intelligence explosion remains a possibility rather than a confirmed outcome, they argue that the potential societal stakes require regulatory preparation before rapid automation outpaces institutional oversight.
Proposed Regulatory Frameworks and Safeguards
The Cambridge-published document outlines three primary recommendations for managing the risks associated with automated scientific research:
- Increased Transparency: Governments should require leading AI labs to disclose metrics regarding the percentage of automated research, model capability growth rates, and how AI systems drive key development decisions. The proposals also suggest embedding independent auditors within frontier labs.
- Safety Brakes and Safeguards: Labs should establish concrete safety prerequisites before developing more powerful models, implement temporary velocity caps on capability scaling, and isolate high-risk experiments on air-gapped networks.
- Emergency Preparedness: The report calls for coordinated international incident-reporting protocols and emergency response plans to address potential cybersecurity threats, biological risks, and workforce disruptions.
Worth a look