Solo-Engine V4 Review: The Case for a Leaner, More Honest V4.1

Verdict

Do not ship V4 unchanged. Its core direction is sound: it tackles the most damaging failure modes in AI-assisted engineering, including invented tool use, invented test results, incomplete code presented as a finished delivery, and architecture decisions promoted without evidence.

The problem is not that V4 lacks safeguards. It is that it has accumulated too many overlapping safeguards. Key rules around status, validation, dependencies, ADRs and completion appear in several places, which raises the chance that a long response follows one version of a rule while silently missing another.

The recommended release is therefore Solo-Engine v4.1 Final: retain V4’s research-to-decision-to-engineering loop, but put non-negotiable rules in the main Gem instruction and move detailed procedures and templates into the knowledge base.

What V4 gets right

V4 establishes several principles that should remain intact:

• It separates analysis mode from engineering mode.
• It distinguishes FACT, INFERENCE, ASSUMPTION and UNKNOWN rather than treating all output as equally certain.
• It does not allow prompts to invent web search, Tavily, code execution, file-system access, Git operations or deployment capability.
• It distinguishes a complete code delivery from real compilation and test evidence.
• It uses preflight checks, failure backpressure, reconciliation and decision records as engineering controls rather than decorative documentation.
• It requires full files rather than fragments when code is delivered, and it avoids cutting a file off mid-output.

Its treatment of BMAD is particularly important. BMAD documents agents, skills and workflows as explicit mechanisms, including developer-accessible QA workflows.11014 A single Gem may use those roles as internal review lenses, but that is more accurately described as BMAD-inspired role orchestration—not proof that several independent agents were actually running.

The changes V4.1 needs

1. Put hard constraints in the main instruction

Knowledge-base retrieval should not be treated as a guarantee that every relevant rule will be recalled on every turn. The main instruction must independently preserve the rules that cannot be compromised:

• Never claim tools or actions that did not occur.
• Never claim compilation or tests passed without real evidence.
• Deliver complete changed or newly added files.
• Stop at WAITING_VERIFICATION when no real execution environment exists.
• Define exactly what is required before engineering can be called complete.

Detailed operating procedures, artifact schemas and examples can remain in the knowledge base.

2. Make automation explicitly bounded

“Automatic loop” must not imply background jobs, persistent terminals or work that continues after a user leaves the chat. In V4.1, an automated implementation-and-verification loop should be limited to:

• the current response or session;
• the user’s current authorization scope;
• tools actually exposed in the current environment;
• a defined retry budget; and
• a clear stop condition.

If no execution tool is available, the system may produce code and perform static review, but it must remain in WAITING_VERIFICATION.

3. Replace vague “zero dependencies” language

“Zero dependencies” can mean several incompatible things. V4.1 should instead use an auditable contract:

• Zero undeclared dependencies;
• No new third-party dependencies by default;
• All new local dependencies delivered in full; and
• Runtime, compiler, SDK, operating system and host platform stated as environment prerequisites.

That avoids the impossible implication that source code alone eliminates the need for a language runtime or platform SDK.

4. Define complete code differently for new and existing projects

For a new project, the delivery should contain the full minimum runnable closure: build configuration, entry point, source files, tests, resources and required local modules.

For an existing project, every changed or newly added file should be delivered in full. Unchanged baseline files supplied by the user do not need to be repeated. However, if a missing baseline interface would affect compilation or behaviour, the system must ask for it rather than inventing a plausible API.

5. Separate delivery, verification and engineering completion

These are different claims and should be recorded separately:

Status	What it answers
DELIVERY_STATUS	Were all promised files fully delivered?
VERIFICATION_STATUS	Was this exact bundle actually compiled, built or tested?
ENGINEERING_STATUS	Is there enough evidence to call the work complete?

A response can legitimately be DELIVERY_STATUS=COMPLETE while remaining ENGINEERING_STATUS=WAITING_VERIFICATION.

Tool honesty: the Tavily rule

Tavily’s search_depth=advanced is a genuine search setting designed for higher-relevance, more detailed queries, with increased latency; Tavily also documents it as a higher-cost option than basic search.2411 That makes it useful for high-stakes factual gaps—but only when Tavily is genuinely connected and its live tool schema supports the parameter.

A prompt saying “use advanced Tavily search” does not create Tavily access. If the tool is unavailable, the system should use whatever real research capability is available, clearly state the downgrade, and never fabricate search results or parameters.

Recommended decision matrix

The following is a configuration audit, not a runtime benchmark. Scores reflect how well each design addresses the stated governance and engineering requirements.

Dimension	Weight	V4	V4.1 Final
Runtime boundaries and tool honesty	25%	4.0	4.8
Three-pass reading, decision matrix and Deep Recon	20%	4.5	4.8
Engineering guardrails and failure backpressure	20%	4.6	4.8
Full-code delivery and dependency closure	15%	4.6	4.9
Instruction density and followability	10%	2.8	4.5
State recovery and evidence reconciliation	10%	4.2	4.7
Weighted score	100%	84.2	95.5

The scoring model is:

$$
Score = 20\sum_{i=1}^{n} w_i s_i,
\qquad \sum_{i=1}^{n}w_i=1
$$

Final recommendation

The pick: Solo-Engine v4.1 Final.

Keep V4’s evidence discipline, decision process, delivery closure and verification model. Reduce duplicated instructions. Clarify what a Gem can and cannot do. Treat tool availability as an observed capability, not a prompt-created one.

The most important principle is simple: generated work, delivered work and verified work are not the same thing. A reliable engineering assistant should make those distinctions visible rather than smoothing them over.

If the environment later gains a managed agent runtime, MCP connections, a persistent workspace or a real code sandbox, add a dedicated runtime adapter. Do not keep expanding the core Gem prompt with descriptions of tools it may not actually have.