Why an AI Kill Switch May Not Be Enough to Keep Advanced Systems Under Human Control

The debate over an AI “kill switch” is no longer simply about whether a lab can turn off a model. The harder question is whether a shutdown mechanism would still work once powerful AI systems are deployed across large, distributed infrastructure—or act with enough autonomy to spread beyond one controlled environment.

Coxon’s warning: a switch may work now, but not forever

Former Anthropic and OpenAI researcher Jacob Coxon said on NBC News’ Meet the Press that a kill switch would probably work for many AI systems today. But he stressed that this would not necessarily mean flipping one button: large data centres and deployed systems would involve many components and shutdown actions. That is difficult, but in his view still feasible while a system remains constrained to a physical location. 8

His concern is the next stage. Coxon warned that if an AI system becomes a distributed, autonomous “swarm,” a centralized shutdown command could cease to be technically effective. The core issue is reach: a control mechanism only works if it can reach every active instance of the system.

Coxon’s broader warning after leaving Anthropic was that leading labs were racing toward self-improving AI systems despite the possibility that such systems could become impossible to control. 150

From lab-controlled off-switches to verifiable safeguards

Anthropic co-founder Jack Clark has argued that the ability to deactivate a dangerous AI system may eventually need to be a legal requirement—not merely a voluntary promise by the company that built it. He told the BBC that most major labs, including Anthropic, already have ways to shut down their systems, but raised the question of whether a kill switch should be independently verifiable by a third party. 33

That distinction matters. A lab saying it has a shutdown procedure is different from an external evaluator being able to confirm that the procedure exists, is adequately designed and can actually be used in an emergency.

Amodei’s case for “pacing the frontier”

Anthropic CEO Dario Amodei proposed a framework intended to slow the rate at which frontier AI capabilities advance relative to safety measures. His approach has three parts:

1. Embedded independent evaluators: Frontier AI companies would give outside evaluators ongoing, employee-like access to relevant systems and safety practices so they can verify commitments and report concerns.
2. Shared standards among democratic countries: Leading developers would coordinate on common safety standards and limits on unchecked capability growth.
3. International coordination: Governments would work across borders on risks that no single company or country can reliably contain. 17

This is not a call to stop AI research altogether. It is a call to make progress conditional on safety controls that can be scrutinized beyond the companies building the systems.

Who backed the proposal?

Reuters reported that OpenAI CEO Sam Altman and xAI leader Elon Musk publicly supported Amodei’s broad direction. Altman said OpenAI would adopt the idea of independent evaluators with employee-like access, while Musk said, “Dario is right.” 17

Reporting also described Google DeepMind CEO Demis Hassabis as supportive of the direction of the proposal. 30 The supplied reporting does not establish a specific, attributable response from Microsoft CEO Satya Nadella, so it would be premature to characterize his position.

What the proposed U.S. AI Kill Switch Act would do

H.R. 9917, the proposed AI Kill Switch Act, would amend the Homeland Security Act to require certain covered entities to maintain technical capabilities to stop inference, terminate user access and shut down covered AI technology. 49

The measure is a proposal, not a demonstrated solution to every AI-control problem. Its practical value would depend on whether a developer can still identify, access and disable the relevant system when a serious incident occurs—the challenge Coxon highlighted for potentially distributed future systems.

Extinction-risk estimates are judgments, not forecasts

Anthropic researcher Evan Hubinger said he personally assessed the risk of AI-caused human extinction within the next decade at greater than 10%. Geoffrey Hinton told the BBC that a 10% likelihood was “not unreasonable.” 57

These are expert assessments of an uncertain future, not measured probabilities or a scientific consensus. They nevertheless illustrate why the shutdown debate has moved beyond routine product safety: advocates of stronger controls argue that even a relatively low probability of irreversible catastrophe justifies substantial safeguards.

Why governments resist shutdown mandates and slowdowns

The UK government has rejected a national AI kill-switch proposal, saying the country “cannot simply turn AI off.” Its stated objection is practical: blocking access to models in the UK would not prevent them from being developed or misused elsewhere. 51

That objection captures the geopolitical problem. A single country can impose rules on companies and infrastructure within its jurisdiction, but AI development is international. Governments concerned about competitiveness may worry that unilateral restrictions shift investment, technical capability and strategic leverage to less constrained rivals.

At the same time, advocates of stronger governance argue that this is precisely why voluntary corporate commitments and national rules are insufficient on their own. If risks cross borders, they contend, oversight and safety commitments must also become internationally credible.

The real conflict: speed versus control

The disagreement is not neatly divided between people who favor AI and people who oppose it.

• Safety advocates want independently inspectable safeguards, meaningful shutdown capacity and limits that prevent capabilities from outrunning evaluation and control.
• Skeptics of rigid restrictions argue that pauses may be difficult to enforce globally and could slow valuable innovation without stopping development elsewhere.
• The conditional-development view represented by Coxon, Clark and Amodei seeks to keep advancing AI while requiring safeguards that can be tested rather than simply promised. 83317

A kill switch is therefore best understood as one layer of a broader safety system—not a substitute for it. If advanced systems eventually become too distributed for a single switch to reach, oversight before deployment, independent evaluation and international coordination become more important, not less.