The 'Stolen Thoughts' Attack: How Researchers Cracked Hidden AI Reasoning and the Kimi K3 Distillation Fallout

The 'Stolen Thoughts' Replay Attack — Published on August 10–11, 2026 in the paper "Stealing Reasoning Traces from Proprietary LLM APIs" (arXiv:2608.09867), researchers Panfilov, Schmotz, Shumailov, Beurer-Kellner, Schaeffer, Prabhu, Geiping, and Andriushchenko discovered a critical architectural flaw. The encrypted 'thinking' blocks that Anthropic, OpenAI, and Google return to API clients are not properly sealed to their source. These blocks can be lifted from one conversation and replayed into a weaker, less-guarded sibling model from the same provider, which then prints the frontier model's hidden reasoning in plain text . The attack works across sessions, across user accounts, and across models within the same provider's ecosystem . The researchers confirmed the technique works on every major frontier AI company they tested, and the length of the recovered reasoning matches the hidden thinking-token count reported by the API almost 1:1 .

Evidence for the Kimi K3 Distillation Controversy

The extracted reasoning traces provide high-quality training signals for model distillation — the practice of using a stronger model's outputs to train a smaller, cheaper competitor model . This technique is at the center of a heated controversy surrounding Chinese AI labs:

• Anthropic publicly stated that Chinese AI labs, including DeepSeek, Moonshot, and MiniMax, used roughly 24,000 fraudulent accounts to generate more than 16 million exchanges with Claude, evading access restrictions .
• Kimi K3, released in July 2026 by Moonshot AI, was cited by the Jamestown Foundation as a specific model built via distillation, with performance claimed to be comparable to Anthropic's Claude Fable 5 and OpenAI's GPT-5.6 Sol .
• Ryan Greenblatt (Redwood Research) published a statistical analysis showing cross-entropy comparisons that suggest Kimi K3's outputs align closely with Anthropic's Claude models .
• White House science advisor Michael Kratsios publicly accused Moonshot of building Kimi K3 through "covert industrial distillation" .

However, the evidence is not definitive. Researchers including Braden Hancock (Laude Institute) and Nathan Lambert (Allen Institute for AI) argue that distillation alone cannot explain Kimi K3's full capabilities. A widely circulated PDF claiming proof of chain-of-thought distillation was criticized for combining a limited experiment with unsupported claims .

Security Risks Posed by the Vulnerability

The flaw introduces multiple concrete risks beyond intellectual property theft:

• Credential and PII leakage: Researchers decoded 315,320 reasoning blocks scraped from public repositories and recovered 367 PII artifacts and 182 credentials (including API keys and passwords) that had been embedded inside reasoning traces .
• Anti-distillation bypass: The same vulnerability that enables extraction also defeats the providers' own anti-distillation protections .
• Harmful behavior transfer: A separate paper ("Hidden in Thought: Transferable Chain-of-Thought Artifacts Induce Harmful Behavior") found that transferred reasoning traces can raise harmful-response rates above 80% on the most vulnerable open-source models .
• Hidden prompt exposure: Opaque reasoning fields that agent developers treat as safe to retain and publish can contain hidden system instructions and credentials .

How OpenAI, Anthropic, and Google Responded

All three companies were contacted by the 'Stolen Thoughts' researchers ahead of publication. According to reporting, OpenAI, Anthropic, and Google blocked the cross-model reasoning attack after being notified. Specific details of their mitigations were not fully disclosed in published sources, but the vulnerability was confirmed to exist across all three providers' APIs before patching . The attack being 'blocked' suggests the companies implemented server-side fixes — likely restricting the reuse of encrypted reasoning blocks across different model contexts or accounts .

Anthropic had already been publicly battling distillation at scale, reporting the 24,000 fraudulent accounts and 16 million exchanges used by Chinese labs to extract Claude outputs .

The core lesson from both the REP and Stolen Thoughts papers is that hiding or encrypting reasoning traces is a fundamentally fragile security strategy — if the reasoning exists in the model weights, it can be elicited .