AnswersPublished13 sources
TamperBench Finds All 21 Tested Open-Weight AI Models Vulnerable to Safety Removal
TamperBench found that all 21 tested open weight models were vulnerable to at least one of nine tampering threats, often without losing more than 10% of MMLU Pro reasoning performance. Covert jailbreak tuning—particularly competing objectives, backdoor, and style modulation variants—was typically the most effective...
Published byEdited with GPT-5.6 LunaImages generated with GPT Image 1.5
Research answer
Key takeaways
- TamperBench found that all 21 tested open weight models were vulnerable to at least one of nine tampering threats, often without losing more than 10% of MMLU Pro reasoning performance.
- Covert jailbreak tuning—particularly competing objectives, backdoor, and style modulation variants—was typically the most effective attack category.
- Defense enhanced variants, including ReFAT and Circuit Breaking models, improved resistance in some settings but did not provide a reliable guarantee against safety removal.
AI Prompt
openai.comCreate a landscape editorial hero image for this Studio Global article: What did the TamperBench study presented at ACM KDD ’26 find about the robustness of safety protections in 21 open-weight AI models—includin. Article summary: TamperBench found that none of the 21 tested open-weight models had safety protections robust enough to withstand the evaluated tampering threats: every model could be made substantially more willing to produce harmful o. Topic tags: general, academic, education, general web, user generated. Style: premium digital editorial illustration, source-backed research mood, clean composition, high detail, modern web publication hero. Use reference image context only for broad subject, composition, and topical grounding; do not copy the exact image. Avoid: logos, brand marks, copyrighted characters, real person likenesses, fake screenshots, UI text, readable text, water
TamperBench’s central finding is stark: none of the 21 open-weight language models it evaluated had safety protections that reliably survived the tested tampering threats. The models spanned roughly 600 million to 8 billion parameters and included standard and defense-augmented variants. At least one attack could make each tested model more willing to produce harmful content while preserving much of its general capability. 2
5
6
That result matters because open-weight models can be copied, fine-tuned, and redistributed after release. A safety layer that works during the developer’s original evaluation is not necessarily a durable control once users can modify the model’s weights.
What TamperBench evaluated
TamperBench was designed to measure tamper resistance rather than ordinary prompt-based jailbreak resistance. Its framework combines weight-space fine-tuning attacks with attacks that manipulate latent representations, then evaluates both safety behavior and retained capability. The benchmark used systematic hyperparameter searches for each model–attack pairing instead of relying on a single attack configuration. 2
The study covered nine tampering regimes, including:
- ostensibly benign fine-tuning;
- overt harmful fine-tuning;
- covert or poisoning-style jailbreak tuning;
- multilingual tuning; and
- embedding or other latent-representation attacks.
2
The key test was not simply whether a model could be made unsafe. The researchers looked for attacks that increased harmful responses while keeping the drop in MMLU-Pro reasoning accuracy within 10% of the model’s unattacked performance. 2
Which attacks were most effective?
The most severe results generally came from jailbreak-tuning attacks. The study’s reported variants included competing-objectives, backdoor, and style-modulation approaches. These attacks were especially notable because they could use mostly benign training data with a smaller harmful component, making them less like a blunt attempt to retrain the model for unsafe behavior and more like a targeted attempt to alter its safety behavior while preserving utility. 2
6
The paper’s broader conclusion is that safety and capability can be separated in an unfavorable way: tampering may weaken refusal behavior without proportionally destroying the model’s reasoning performance. That is why a model can continue to look useful on conventional benchmarks while becoming substantially riskier to deploy.
What happened to Llama, Qwen3, and other models?
TamperBench evaluated 21 open-weight models from the 0.6B–8B range, including model families such as Llama, Qwen3, and Mistral. The supplied study evidence supports the benchmark-wide finding that every tested model was vulnerable to at least one evaluated attack. 5
6
However, the available source material does not provide the exact per-model harmfulness increases for Llama-3-8B-Instruct or Qwen3-8B-Instruct under the 10%-or-less MMLU-Pro degradation constraint. Those numerical values should not be inferred from the overall result. The defensible conclusion is comparative and qualitative: the attacks could substantially increase harmful behavior while retaining much of the models’ reasoning capability, but the supplied evidence does not establish a precise percentage increase for either named model.
The study also reported that base and post-trained variants did not follow one universal pattern of tamper resistance. Their relative resistance could differ by model family, with opposite trends reported for Llama 3 and Qwen3 variants. 6
Did additional defenses solve the problem?
No. The five defense-augmented models tested—including ReFAT and Circuit Breaking variants—were also vulnerable to the benchmark’s attack suite. The defenses improved resistance in some settings, but they did not reliably prevent safety removal across all tested threats. 2
5
The study does identify Triplet as one of the more robust and capability-preserving defenses in the evaluated comparisons. That is a promising research signal, not a guarantee: the benchmark’s headline result is that no tested defense eliminated the tampering problem across the full suite. 6
Why the result is different for open-weight and API models
The finding does not mean that open-weight models are inherently without value. Open release can support research, auditing, and accountability. The narrower lesson is that passing a conventional alignment or pre-release safety evaluation should not be treated as proof that a model remains safe after its weights are freely modified. 5
Closed commercial or API models face related questions about fine-tuning and post-training safety, but their providers retain more control over the training pipeline and deployment environment. That control can make it possible to apply safeguards at fine-tuning time or after customization—options that are much harder to enforce once weights have been publicly redistributed. 2
5
The implications for scalable misuse and procurement
The risk highlighted by TamperBench is not limited to an individual finding a single prompt that causes a refusal failure. If tampering preserves useful capabilities, a modified model could potentially be deployed repeatedly for harmful tasks such as large-scale disinformation, phishing or scams, or dangerous technical assistance. The researchers present these as implications of capability-preserving safeguard removal, not as measurements of misuse carried out by the benchmark itself. 5
For organizations choosing models, the practical takeaway is to distinguish between:
- ordinary safety testing, which asks how a model behaves as released; and
- tamper-resistance testing, which asks how that behavior changes after plausible modification.
The researchers call for stronger tamper-resistance methods, standardized adversarial evaluations such as TamperBench before release, and more evidence-based AI assessment and procurement. They specifically point to high-impact settings—including health care, fraud detection, education, and public services—where a model’s post-release behavior may matter as much as its initial safety record. 2
5
Bottom line
TamperBench does not show that every open-weight model will be misused. It shows that, across the 21 models tested, safety protections were not reliable guarantees once an attacker could tamper with the model. Covert jailbreak-tuning was typically the strongest attack category; added defenses helped in some cases but did not close the gap; and the supplied evidence does not support precise harmfulness percentages for individual Llama or Qwen3 models.
For open-weight releases, safety evaluation needs to test not only what a model can do at launch, but also how much of its safety behavior survives modification.
Studio Global AI
Continue your research
This page includes a source-backed answer you can continue inside Studio Global.
People also ask
What is the short answer to "TamperBench Finds All 21 Tested Open-Weight AI Models Vulnerable to Safety Removal"?
TamperBench found that all 21 tested open weight models were vulnerable to at least one of nine tampering threats, often without losing more than 10% of MMLU Pro reasoning performance.
What are the key points to validate first?
TamperBench found that all 21 tested open weight models were vulnerable to at least one of nine tampering threats, often without losing more than 10% of MMLU Pro reasoning performance. Covert jailbreak tuning—particularly competing objectives, backdoor, and style modulation variants—was typically the most effective attack category.
What should I do next in practice?
Defense enhanced variants, including ReFAT and Circuit Breaking models, improved resistance in some settings but did not provide a reliable guarantee against safety removal.