What Evan Hubinger’s 10% AI Extinction Estimate Reveals About AI Alignment

Evan Hubinger’s public statement did not establish that AI will cause human extinction. It did something more specific—and unusually consequential: an Anthropic alignment-science lead said he personally puts that possibility above 10% over the next decade while acknowledging that Anthropic does not yet have a plan to solve alignment for superintelligence or a clearly established path to one. 12

That combination gave concrete support to the concern behind former Anthropic and OpenAI researcher Jacob Coxon’s resignation: frontier labs may be pushing toward far more capable systems while the methods for keeping those systems reliably aligned with human intent remain unproven. 23

The key distinction: a personal forecast, not a company finding

The “more than 10%” figure was Hubinger’s own assessment. It was not presented as an official Anthropic probability estimate, a consensus among its staff, or a settled scientific forecast. 12

That qualification matters. A probability estimate alone does not explain a mechanism, establish a timeline, or prove that catastrophe is likely. Hubinger also reportedly said the risk from models available today is low. His concern was the prospect of future systems becoming substantially more capable, including through recursive self-improvement—using AI capabilities to help produce more capable AI. 210

Still, the statement is notable because it came from a person leading alignment work at a major frontier lab. The central disclosure was not merely a dramatic number; it was the gap between advancing capabilities and confidence that superintelligent systems could be kept under human control. 12

Why it reinforced Jacob Coxon’s resignation warning

Coxon, who had worked at both OpenAI and Anthropic, resigned while arguing that neither company was acting responsibly and that both were racing toward self-improving superintelligence. Hubinger publicly agreed with Coxon’s claim that people building these systems sincerely consider extinction-level outcomes possible. 23

That does not demonstrate that all employees share one view, nor does it turn Coxon’s account into an independently verified assessment of both companies. But it makes the resignation harder to dismiss as an isolated complaint from a former employee. A current alignment leader confirmed the underlying point: serious internal concern can coexist with continued frontier development. 12

Samuel Marks’s related comments should be read with the same care. Reports describe him as saying AI developers believe their technology could cause human extinction or similarly severe outcomes, but this is an insider view—not published evidence of a company-wide risk estimate. 5

The real argument is about governance under uncertainty

The episode is often flattened into “AI doom versus AI optimism.” The more useful question is practical: what safeguards should be required when potential harms are global and irreversible, but the likelihood and timeline are deeply uncertain?

Hubinger’s remarks sharpen several issues:

• Alignment is not solved. Saying a lab is trying hard is different from showing it has a validated method to control systems substantially more capable than current models. 1
• Competition can make voluntary restraint difficult. Coxon’s criticism was that competitive pressure is driving labs toward higher capability even as major safety questions remain unanswered. 23
• Current-model safety is not the whole issue. Hubinger distinguished low perceived risk from today’s models from concern about future, much more capable systems. 210
• External verification matters. Claims about internal safety processes are difficult for the public, customers and governments to assess without independent evaluation and transparent reporting.

These are governance problems, not proof of an imminent disaster. But they are precisely the kind of problems that become harder to solve after systems acquire dangerous capabilities.

Transparency questions add pressure

Reports said Anthropic declined to submit a latest model to the UK AI Safety Institute for testing before release. The reporting should be treated cautiously: the available accounts describe it as a report, rather than a confirmed public finding from the company or UK government. 10

Even so, the allegation illustrates why voluntary safety commitments face skepticism. If a company’s own safety leaders acknowledge unresolved alignment challenges, outside observers will reasonably ask what testing was done, who reviewed it, what results were shared, and what capability thresholds trigger stronger controls.

What a slower, safer path could mean

A call to “slow down” need not mean permanently stopping all AI research or deployment. The policy case is for enforceable checks before systems pass especially consequential capability thresholds.

Possible measures include:

• independent pre-deployment capability and safety evaluations;
• reporting requirements for serious incidents and materially dangerous capabilities;
• clearer thresholds for when developers must implement stronger security and governance controls;
• audits that do not rely solely on a developer’s internal assurances; and
• international coordination so that any one company or country is not pressured to cut corners to keep pace with competitors.

The rationale is straightforward: if the risk is uncertain but the downside could be extreme, safeguards need to be demonstrable before, rather than after, a loss of control.

What the statement ultimately revealed

Hubinger’s post revealed a tension at the heart of frontier AI: leading developers can publicly recognize severe long-term risks while continuing to build toward the capabilities that make those risks more urgent. 12

It does not settle the probability of catastrophic AI harm, and it should not be treated as an official prediction from Anthropic. It does, however, make the alignment debate more concrete. The question is no longer only whether advanced AI could become dangerous. It is whether companies and governments can show—through independent evidence, enforceable rules and international coordination—that they are prepared before the systems in question become too powerful to govern safely. 110