DeepSeek V4 Pro Leads Aikido’s AI Vulnerability Benchmark—but the System Matters More Than the Model

Aikido’s August 2026 benchmark tested 10 AI models against 32 recently disclosed vulnerabilities, running each model three times. DeepSeek V4 Pro 0813 led pooled recall with 28 of 32 findings for about $295, but the broader lesson was less about choosing one winner than about designing a system that combines repeated investigation, validation and model specialization. 3

The headline result: DeepSeek Pro found the most vulnerabilities

DeepSeek V4 Pro 0813 reached 28 of 32 vulnerabilities, or 87.5% pooled recall, when the results from three runs were combined. The figure makes it the benchmark’s strongest discovery-stage model, but it should be read as a pass@3 result rather than as evidence that one invocation reliably finds 28 vulnerabilities.

Different runs can explore different files, paths and exploit hypotheses. Pooling them increases coverage because the system benefits from that search diversity. In practical terms, the result supports running several independent investigations instead of treating a model’s first answer as a complete security assessment. 3

How the models differed

DeepSeek V4 Pro 0813: strongest pooled discovery

DeepSeek Pro led total coverage at 28/32 for approximately $295 across three runs. Its advantage was breadth: it surfaced more of the benchmark’s vulnerabilities when its investigations were combined.

That makes it a strong candidate for the first stage of a security workflow. Its findings still need evidence-based validation, deduplication and severity review before they become production alerts.

DeepSeek V4 Flash 0731: low-cost parallel scouting

Three Flash runs found 24 of 32 vulnerabilities for roughly $108. That gives the smaller model a compelling coverage-per-dollar profile and makes it suitable for inexpensive parallel scans or additional reruns.

Flash is not automatically the best final reviewer. Its practical value is in increasing the number of investigative attempts within a fixed budget, then passing the combined findings to a more selective verification stage. 3

Grok 4.6: the consistency leader

Grok 4.6 stood out for repeatability: 21 vulnerabilities appeared in all three of its runs. That consistency matters when teams need predictable behavior, stable reports and fewer surprises between scans.

Its strength is different from DeepSeek Pro’s. A highly consistent model may be preferable for recurring monitoring or standardized workflows, even when a more exploratory model achieves higher pooled recall.

Claude Opus 5 and GPT-5.6 Sol: strong frontier alternatives

Claude Opus 5 reached 26/32 pooled recall, while GPT-5.6 Sol reached 25/32 in the comparison. Both remained high-performing options, but the results challenged the assumption that the most expensive closed model must deliver the best vulnerability coverage.

The right reason to select either model is therefore not simply brand or general benchmark reputation. Teams should evaluate whether its reasoning, reporting or review behavior adds value alongside the rest of the security pipeline. 3

Kimi K3: precision for verification and triage

Kimi K3’s standout result was 92.3% pooled precision. Precision measures how many reported findings were accepted, rather than how many vulnerabilities were found overall.

That distinction makes Kimi useful in a different role from a broad-search model. A high-recall agent can search widely and tolerate more noise; a high-precision model can help verify findings, prepare reports and reduce the number of alerts that reach engineers.

Qwen3.8-Max: repeated reasoning can waste or reveal

Aikido observed Qwen3.8-Max revisiting the same CVE or reasoning path. Repetition is inefficient when it consumes investigation turns without expanding coverage, but it can also be useful when a second examination exposes a missed condition, execution path or validation detail.

The practical answer is harness-level state tracking. An agent should know what has already been tested, repeated branches should have limits, and unresolved cases should be routed to a fresh investigation rather than automatically replayed.

GLM-5.3: a cheaper ensemble component

GLM-5.3 improved from 24/32 to 25/32 in the updated comparison while retaining a lower cost profile than the closed frontier alternatives discussed in the benchmark. That positions it as a credible high-volume worker or ensemble component.

Its value is greatest when organizations use cost and deployment flexibility to run more investigations, rather than relying on a single pass from any model.

Recall, precision and consistency answer different questions

A vulnerability-discovery system should not collapse all performance into one leaderboard number:

• Recall: How many of the known vulnerabilities did the system rediscover?
• Precision: How many submitted findings were accepted?
• Consistency: How often did separate runs find the same vulnerabilities?
• Cost efficiency: How much did it cost to produce validated findings?

These metrics describe different operational needs. Discovery favors high recall and diverse investigative behavior. Production alerting favors precision, reproducible evidence, deduplication and useful risk ranking. A model that is excellent at finding possibilities may therefore be a poor choice for sending unreviewed alerts directly to developers.

Why repeated runs changed the benchmark’s conclusion

The benchmark’s most important systems lesson is that model performance was stochastic. A single run samples only one investigation path; several independent runs increase the chance that the system explores different hypotheses.

That is why three relatively inexpensive DeepSeek runs could compete with or exceed the total coverage of a single pass from more expensive frontier models. The relevant unit is not just the model call. It is the complete investigation: model, tools, prompt, turn budget, memory, reruns and validation process. 3

What a practical security workflow could look like

A deployment built around these results would separate discovery from judgment:

1. Explore broadly. Run one or more cost-efficient agents across the codebase to maximize candidate coverage.
2. Track investigation state. Record files inspected, hypotheses tested and findings already reported so agents do not waste turns repeating the same path.
3. Deduplicate findings. Merge reports that describe the same underlying flaw before counting or routing them.
4. Validate with a precision-oriented model. Require evidence that the suspected flaw is reachable and meaningful.
5. Escalate difficult cases. Use a stronger reasoning model or a specialist investigation for ambiguous exploit chains.
6. Keep human review in the loop. Consequential findings should be confirmed before remediation, disclosure or automated enforcement.

This model-routing approach is more robust than treating open-weight or closed models as simple drop-in replacements for one another.

The benchmark’s limits

Aikido’s results are useful, but they are not a universal ranking of AI security models. The test covered 32 recently disclosed vulnerabilities, three attempts per model and a particular agent harness. Performance can change with programming language, repository structure, tool access, threat model, investigation budget and the organization’s tolerance for false positives. 3

The defensible conclusion is narrower and more useful: in this setting, open-weight models—especially DeepSeek V4 Pro—could rival or surpass closed frontier models when paired with repetition and orchestration. The winning security product is not necessarily the model with the highest headline score. It is the system that combines broad discovery, reliable validation and evidence that engineers can act on.