The Stolen Thoughts Attack: How Researchers Cracked Encrypted AI Reasoning—and What They Found Inside

In August 2026, a team of researchers from the University of Tübingen, the Max Planck Institute for Intelligent Systems, MATS Research, the ELLIS Institute Tübingen, and the security firm Snyk published a paper describing a surprisingly simple attack on the encrypted reasoning traces of frontier AI models. The attack worked across all three major providers—Anthropic, OpenAI, and Google—and it did not require jailbreaking a single frontier model. Instead, it exploited a fundamental architectural choice: these companies return encrypted "reasoning blocks" to the API client rather than storing them server-side, and those blocks are portable across sessions, users, and even different models from the same provider . The researchers simply took an encrypted reasoning block from a frontier model and replayed it into a weaker sibling model, which would decode and output the trace in plaintext . This technique, which the paper calls a "decryption jailbreak," is detailed in the preprint Stealing Reasoning Traces from Proprietary LLM APIs, submitted to arXiv on August 10, 2026 .

How the Replay Attack Works

When a user queries a frontier model like Claude Opus 4.8 or GPT-5.6 Sol through an API, the model performs internal step-by-step reasoning—its "chain of thought"—and returns that reasoning to the client as an opaque block of encrypted text. On subsequent calls, the client passes this block back to the API, allowing the model to continue its reasoning without the provider storing the intermediate state server-side .

What the researchers discovered is that these encrypted blocks are not bound to a specific session, user, or even model. They share a single global encryption key across all sessions and users . This means a block captured from one conversation can be replayed in another context. The key insight: feed a block from a heavily safeguarded frontier model into a weaker, less-aligned sibling model from the same provider (e.g., Claude Haiku), and the weaker model—which has fewer refusal guardrails—will decode and output the block's contents verbatim .

It takes only two API calls to execute the attack: one to capture the encrypted block and one to feed it to the weaker model. The researchers demonstrated that the attack works at scale .

Circumstantial Evidence of Distillation: Kimi K3's "Abnormal Probability" Patterns

The attack did not just expose the reasoning of frontier models—it also provided new evidence in the ongoing debate over whether Chinese AI company Moonshot AI trained its Kimi K3 model by distilling (essentially copying) the reasoning of U.S. models.

After extracting reasoning traces from Claude Opus 4.8 and GPT-5.6 Sol, the researchers compared them against the outputs of open-weight models, including Kimi K3. They found that Kimi K3's outputs exhibited what they described as "abnormal probability phenomena" that closely matched patterns found in the Claude and GPT reasoning traces .

The evidence is circumstantial, not definitive. Independent researchers say the findings "provide some evidence—although not conclusive proof—that certain Chinese models may have been trained by distilling reasoning information from US models" . Additional suggestive findings: when asked to self-identify, Kimi K3 initially identified as "Claude" by Anthropic, adding fuel to the accusations .

Kimi K3 is a 2.8-trillion-parameter open-weight model released by Moonshot AI on July 16, 2026. It outperforms Claude Opus 4.8 on several benchmarks, but it is not an unambiguous leader. On the independent Artificial Analysis Intelligence Index, K3 scores 57 versus Claude Fable 5's 60 and GPT-5.6 Sol's 59. Moonshot itself has been unusually candid that K3 still trails GPT-5.6 Sol and Claude Fable 5 overall .

The Timeline Debate

The distillation accusation is not new—both Anthropic and OpenAI have previously accused Chinese firms of systematically distilling their models. A review of Chinese academic papers and patents revealed Chinese military researchers using outputs from OpenAI and Anthropic models to train domestic AI systems for defense capabilities .

Kimi K3's launch sparked immediate controversy. Within hours of its July 16 release, Anthropic accused Moonshot of building K3 by distilling Claude, and the claim reached the White House, raising the prospect of sanctions .

However, critics of the distillation claim point to a timing problem. Claude Opus 4.8 was released shortly before Kimi K3's launch. Independent researchers say the timeline makes straightforward copying "implausible" because there simply was not enough time to train a 2.8-trillion-parameter model using distillation from a model that had only recently been released . Moonshot argues K3 was built from independent research .

Security Vulnerability: Sensitive Data Leakage in Plain Sight

The same architectural flaw exposed a practical security vulnerability. Because frontier models sometimes incorporate credentials and personal data into their reasoning process, the encrypted reasoning blocks that developers shared publicly—in agent session logs, for example—contained sensitive information that could be read by anyone who knew the trick.

By scanning roughly 7,000 publicly shared agent session logs published before August 2026, researchers found 62 live API keys, 33 passwords, and other sensitive user data—704 privacy artifacts total—hidden inside the encrypted reasoning blocks . Any developer who published raw agent logs may have unknowingly exposed API keys, passwords, and personally identifiable information inside blocks they believed no one could read . This included logs stored on public GitHub repositories .

The researchers identified four distinct abuse paths enabled by the vulnerability: reconstruction of hidden reasoning, recovery of secrets embedded in agent traces, injection of instructions through an uninspectable channel, and cross-session replay of unexpired blocks .

Provider Responses: Server-Side Patches After Disclosure

All three companies deployed server-side mitigations after the research team followed responsible disclosure procedures .

The researchers disclosed their findings to OpenAI, Anthropic, Google, Microsoft, and Hugging Face. After the providers implemented changes, reproducibility checks confirmed the main extraction technique no longer succeeds against current API builds . Documentation now advises stripping reasoning blocks before sharing logs or switching models in mid-conversation .

Notably, cryptography researcher Matthew Green had separately reported the cross-session replay vulnerability to both OpenAI and Anthropic via their bug bounty programs in May 2026, but received dismissive responses before the academic paper was published .

The patches are live as of August 2026, but historical session logs with decoded reasoning blocks that were already scraped from the public web remain accessible .

Why This Matters

The "stolen thoughts" attack reveals a fundamental tension in the design of modern AI APIs. Providers want to offload reasoning state to the client for scalability and latency, but the cryptographic binding needed to keep that state secure across sessions, users, and models requires careful design. The single global encryption key approach used by Anthropic, OpenAI, and Google was a shortcut that created a security and privacy vulnerability affecting every user and developer who interacted with these APIs.

The attack also lands in the middle of an escalating US-China conflict over AI distillation. While the evidence regarding Kimi K3 is circumstantial, it provides the strongest technical evidence to date that distillation may be happening at scale, and it gives policymakers and researchers a new tool for investigating the provenance of AI models.