Why AI Researchers Want Regulators to Track AI Automating AI Research

The concern behind the latest call for oversight is not simply that AI models may become more capable. It is that AI could help build the next generation of AI, creating a feedback loop that speeds up research and development. Researchers from OpenAI, Anthropic, Meta and Microsoft want policymakers to examine how far that automation has progressed before the pace of change makes effective oversight harder. 23

That possibility—sometimes called an “intelligence explosion”—is a risk scenario, not evidence that AI is already improving itself without human direction. The central policy question is how to measure and govern AI’s growing role in AI research. 114

How AI-assisted research could speed development

AI can contribute to the work involved in developing future models. If those contributions help researchers produce more capable systems, and those systems in turn take on more research work, each cycle could potentially accelerate the next. The idea is recursive self-improvement: AI systems help improve the process used to build AI systems. 15

The researchers’ warning is about the possible effect on tempo. In one scenario described in reporting on their paper, advances that might otherwise take a year could happen in weeks. Such a timeline is a forecast, not a measured outcome; the sources do not establish that this pace has been reached. 314

Earlier research based on interviews with 25 researchers from frontier labs and academia found that many expected AI to automate AI research and create a positive feedback loop. That provides context for why the possibility is taken seriously in the field, but it does not show that an intelligence explosion is inevitable. 1

What researchers want policymakers to examine

The immediate request is for greater visibility into how extensively companies use AI in research and development. More than 20 AI leaders and researchers urged policymakers to scrutinize that progress and consider safeguards for the technology. 2

That distinction matters: the call is not proof that AI labs have already ceded control of research to AI. Better information about where and how AI is being used could help policymakers assess whether existing oversight is adequate as systems take on more of the development process. 23

Why some leaders want to pace development

A coordinated slowdown could create time for safety testing, independent evaluation and stronger oversight before capabilities change again. Researchers at OpenAI and Anthropic have publicly called for slower development, while other industry leaders have favored a more market-led approach. The disagreement shows that there is no settled industry consensus on how much to slow down—or what a workable pause would involve. 818

Pacing also raises practical questions: which systems or activities would be covered, who would monitor compliance, and how would a slowdown be coordinated across competing companies? Available reporting notes that even the meaning and enforcement of a pause remain unclear.

The case for an emergency shutdown capability

Microsoft president Brad Smith has argued that the people who create AI models, cloud-infrastructure providers and those managing AI agents should be able to turn systems off. That proposal treats shutdown capability as a possible emergency brake, not a substitute for prevention or ongoing supervision.

A shutdown mechanism also has limits. AI systems may run across distributed infrastructure, so no single switch can guarantee that every component will stop; experts have pointed to the need for monitoring and human intervention as well.

The commercial tension behind the debate

The proposed roles for developers and cloud providers create a governance question: companies may help oversee systems while also building or hosting them. Microsoft is part of AI model development and, as Smith’s remarks make clear, is also a cloud provider. That overlap makes independent checks and clear rules important; it does not, on its own, show that any company’s safety position is insincere. 2

There is also a broader coordination problem. Industry leaders are divided over the pace of development, and public discussion of slowing down takes place amid strong competition to build and deploy AI. A voluntary restraint by one company may be difficult to sustain if others continue moving quickly. The sources establish disagreement and competitive pressure, but they do not prove that investment interests determine any individual researcher’s views. 1218

The practical implication is that policymakers face two related questions: how to track AI’s role in developing AI, and how to make safeguards verifiable rather than dependent only on companies’ own assurances. The warnings justify examining those questions now; they do not resolve how likely an intelligence explosion is or which policy response would work best. 123