AnswersPublished15 sources
DeepSeek-V4-Flash Beat Claude Opus 4.8 on Five Agent Benchmarks—But the Harness Did the Heavy Lifting
Floatboat reported that the same DeepSeek V4 Flash model rose above Claude Opus 4.8 on all five tested agent benchmarks after only its execution harness changed; the claim is promising but has not been independently r... At the stated 3:1 input to output ratio, the reported blended API token cost was $0.175 per mill...
Published byEdited with GPT-5.6 LunaImages generated with GPT Image 1.5
Research answer
Key takeaways
- Floatboat reported that the same DeepSeek V4 Flash model rose above Claude Opus 4.8 on all five tested agent benchmarks after only its execution harness changed; the claim is promising but has not been independently r...
- At the stated 3:1 input to output ratio, the reported blended API token cost was $0.175 per million tokens for DeepSeek versus $10 for Opus—about 57.1 times higher for Opus.
- The reported harness gains increased from 1.9% on the shortest task to 23.6% on the longest, suggesting that state, tools, verification, and recovery become more important as agent tasks stretch across many steps.
AI Prompt
openai.comCreate a landscape editorial hero image for this Studio Global article: How did AOE Tech Labs’ Floatboat team’s August 7, 2026 single-variable Harness benchmark experiment show that DeepSeek-V4-Flash—the $0.14 mo. Article summary: Floatboat’s August 7 result is best understood as a vendor-reported ablation: it held the DeepSeek-V4-Flash model constant and replaced the execution harness. Its claim is not that model capability ceased to matter, but . Topic tags: general, general web, user generated, documentation. Style: premium digital editorial illustration, source-backed research mood, clean composition, high detail, modern web publication hero. Use reference image context only for broad subject, composition, and topical grounding; do not copy the exact image. Avoid: logos, brand marks, copyrighted characters, real person likenesses, fake screenshots, UI text, readable text, watermarks,
Floatboat’s August 7 benchmark result is not best read as proof that DeepSeek-V4-Flash is inherently more capable than Claude Opus 4.8. It is better understood as a vendor-reported ablation: the model was held constant while the execution harness was replaced. Floatboat says that change was enough to move the same low-cost model ahead of the published Opus 4.8 comparisons on five public agent benchmarks. 3
The more important conclusion is narrower and more useful: for long-running agent tasks, the system surrounding a model may contribute as much to success as the model itself. That conclusion is suggestive—not settled—because the experiment has not yet been independently replicated.
What Floatboat changed
Floatboat says it used the official DeepSeek-V4-Flash-0731 release with the same inputs and parameters, then compared DeepSeek’s minimalist official harness with its own evaluation harness. The latter used an isolated physical sandbox and provided a more persistent operating loop, including workspace and file access, tools, retained state, and opportunities to inspect results, correct errors, and continue working. 3
That distinction matters because an agent does not succeed by generating a good answer once. It must repeatedly act on an environment, observe what happened, update its plan, and avoid damaging the work it has already completed.
The five reported benchmark results
In Floatboat’s harness, DeepSeek-V4-Flash reportedly scored:
- BrowseComp: 87.80
- DeepSWE: 67.25
- Terminal Bench 2.1: 84.26
- AutomationBench: 27.50
- Toolathlon: 79.62
Floatboat says these results exceeded the published Claude Opus 4.8 comparisons on all five benchmarks. 3
The headline needs qualification. Public benchmark comparisons can combine scores produced with different harnesses, prompts, tool configurations, sampling settings, and evaluation procedures. The controlled part of Floatboat’s test—the same model with a different harness—is more informative than simply placing unlike systems on one leaderboard. Even so, it is still a company-reported result.
The cost gap was large—but only at the API-token level
The reported DeepSeek pricing was $0.14 per million input tokens and $0.28 per million output tokens. Using a 3:1 input-to-output ratio, the blended price is:
(3 × $0.14 + $0.28) ÷ 4 = $0.175 per million tokens
Floatboat used stated Claude Opus 4.8 prices of $5 per million input tokens and $25 per million output tokens. At the same ratio:
(3 × $5 + $25) ÷ 4 = $10 per million tokens
That makes the Opus comparison approximately 57.1 times more expensive on blended API-token pricing. 3
This is not the same as saying a completed task costs 57.1 times more. The arithmetic excludes tool charges, compute, caching, context length, retries, engineering, and the cost of operating the harness itself. It is a model-token price comparison, not a total-cost-of-ownership analysis.
Why the advantage grew on longer tasks
Floatboat reported relative gains of 1.9%, 9.6%, 12.6%, 19.9%, and 23.6% when the five benchmarks were ordered from shorter- to longer-horizon work. The company presents the sequence as non-decreasing: the harness mattered less on bounded tasks and more as the number of dependent steps increased. 3
A long-horizon agent can fail even when its individual decisions look reasonable. It may need to:
- locate and edit the correct files;
- call tools repeatedly and interpret their outputs;
- preserve plans, state, and intermediate artifacts;
- test changes and diagnose failures;
- recover from a mistaken action; and
- roll back an edit before it corrupts later work.
A weak execution loop can turn correct reasoning into lost context, invalid tool calls, premature termination, or unrecoverable changes. A stronger model may improve local decisions, but it cannot compensate for an environment that does not preserve state, expose useful feedback, or provide safe recovery.
That is why the result is relevant beyond one model comparison. It shifts attention from “Which model is smartest?” to “Which model-and-runtime combination can reliably finish the task?”
What the Harness Leverage Ratio measures
Floatboat introduced the Harness Leverage Ratio, or HLR, to compare the benefit of changing the harness with the benefit of upgrading the model:
HLR = gain from changing only the harness
─────────────────────────────────────
gain from upgrading to the stronger reference system
An HLR above 1 means that, on the selected benchmark and comparison, the harness change produced a larger score increase than the measured upgrade to the stronger reference system. The metric is not an industry standard, and its value depends on the selected baseline, reference model, benchmark version, and evaluation controls. 2
4
On DeepSWE, Floatboat reported that Flash rose from 54.4 in DeepSeek’s harness to 67.25 in Floatboat’s harness—a 12.85-point increase. The cited Opus 4.8 result was 58.0, a 3.6-point gap over the DeepSeek baseline. That produces an HLR of approximately 3.57. 3
Floatboat also reported that HLR increased from 0.78 on the shortest task to 3.57 on the longest. If replicated, that pattern would support the idea that runtime improvements become disproportionately valuable as tasks require more state, tool calls, and recovery. It should not be treated as a universal law: changing the reference system or evaluation setup could change the ratio.
What this says about Floatboat’s product thesis
The benchmark fits Floatboat’s broader product positioning. Public descriptions present it as an agent-native work environment for solo operators, combining files, browsing, AI interactions, scheduling, communications, and reusable skills or workflows. 1 A May 2026 launch announcement described the company’s focus on proactive, calendar-driven agent workflows and identified Sequoia and Welight Capital as backers.
18
That product direction treats the runtime as a core layer rather than a thin wrapper around a language model. The agent needs to maintain context across applications, act at the right time, respect permissions, and make progress without requiring the user to manually reconstruct every step.
The larger implication for everyday agents
If the benchmark result survives independent testing, the competitive advantage in daily-use agents may move toward the execution substrate: safe state management, observability, verification, recovery, permissions, and user control.
That would create a more flexible model strategy. Lower-cost models could handle routine, well-scaffolded work inside a reliable runtime, while premium models could be reserved for tasks that genuinely require deeper reasoning or judgment. The best product might therefore be neither the cheapest model nor the strongest model in isolation, but the system that uses each model where it is most effective.
The caveat remains central. Floatboat’s technical brief is the strongest evidence provided for the August 7 experiment, and the company acknowledges that broader cross-model comparisons use differing harnesses and evaluation configurations. Independent reruns, released evaluation artifacts, and tighter controls for prompts, budgets, tools, retries, and data versions would be needed before the “five wins” become an established industry result. 3
Studio Global AI
Continue your research
This page includes a source-backed answer you can continue inside Studio Global.
People also ask
What is the short answer to "DeepSeek-V4-Flash Beat Claude Opus 4.8 on Five Agent Benchmarks—But the Harness Did the Heavy Lifting"?
Floatboat reported that the same DeepSeek V4 Flash model rose above Claude Opus 4.8 on all five tested agent benchmarks after only its execution harness changed; the claim is promising but has not been independently r...
What are the key points to validate first?
Floatboat reported that the same DeepSeek V4 Flash model rose above Claude Opus 4.8 on all five tested agent benchmarks after only its execution harness changed; the claim is promising but has not been independently r... At the stated 3:1 input to output ratio, the reported blended API token cost was $0.175 per million tokens for DeepSeek versus $10 for Opus—about 57.1 times higher for Opus.
What should I do next in practice?
The reported harness gains increased from 1.9% on the shortest task to 23.6% on the longest, suggesting that state, tools, verification, and recovery become more important as agent tasks stretch across many steps.