AnswersPublished20 sources
How did Memento 3, developed by Huawei Noah’s Ark Lab, University College London and partners, clear all 25 games and 183 levels of the ARC AGI 3 public benchmark—which presents unfamiliar pixel style games without explaining their rules, goals or actions—in 7,518 actions versus
Memento 3’s reported gain came from changing the agent’s external memory and planning machinery—not the language model’s weights: it repeatedly wrote, tested and revised a rulebook and executable simulator, then used that simulator to plan Verification gap: Insufficient evidence in the retrieved excerpts to confirm...
Published byImages generated with GPT Image 2
Research answer
Key takeaways
- Memento 3’s reported gain came from changing the agent’s external memory and planning machinery—not the language model’s weights: it repeatedly wrote, tested and revised a rulebook and executable simulator, then used that simulator to plan
- Verification gap: Insufficient evidence in the retrieved excerpts to confirm Sergey Rodionov’s baseline1 figures of 99 RHAE and 8,347 actions, or the claimed highest reasoning setting.
- How “Code as Model” works Record hypotheses: A Markdown rulebook stores the agent’s current understanding of objects, action meanings and environment dynamics; unknown rules can remain explicitly unspecified rather than being treated as set
- Recover from bad edits: Reporting says Git records rulebook and code changes, allowing the agent to inspect history and restore earlier versions when an edit proves wrong.[14] Validation detail: Insufficient evidence in the retrieved excerp
AI Prompt
openai.comCreate a landscape editorial hero image for this Studio Global article: How did Memento 3, developed by Huawei Noah’s Ark Lab, University College London and partners, clear all 25 games and 183 levels of the ARC. Article summary: Memento 3’s reported gain came from changing the agent’s external memory and planning machinery—not the language model’s weights: it repeatedly wrote, tested and revised a rulebook and executable simulator, then used tha. Topic tags: general web, llm, ai, workflow, productivity. Style: premium digital editorial illustration, source-backed research mood, clean composition, high detail, modern web publication hero. Use reference image context only for broad subject, composition, and topical grounding; do not copy the exact image. Avoid: logos, brand marks, copyrighted characters, real person likenesses, fake screenshots, UI text, readable text, watermarks, charts
Memento 3’s reported gain came from changing the agent’s external memory and planning machinery—not the language model’s weights: it repeatedly wrote, tested and revised a rulebook and executable simulator, then used that simulator to plan actions.1
5
6 The available results support the headline public-benchmark achievement, but the retrieved excerpts are insufficient to independently verify several detailed numbers and implementation examples in your question.
7
27
Results and what explains the improvement
- Reported achievement: The Huawei Noah’s Ark Lab–UCL collaboration cleared all 25 public games and 183 levels in 7,518 actions, versus the reported human total of 17,135—approximately 44% as many actions, or 56% fewer—with RHAE 100.
8
27
- Frozen-model comparison: Reporting identifies Claude Opus 5 as the underlying model and gives approximately 40.7 RHAE for it in the official ARC Prize harness, while Memento 3 uses no weight updates or game-specific fine-tuning.
10
13
14
- Mechanism: Rather than relying only on conversational reasoning, the agent accumulates explicit, revisable knowledge of environment dynamics in external memory and turns that knowledge into executable predictions for planning.
1
5
6 This supports an architectural explanation for the improvement, but does not isolate how much each component contributed or establish a compute-matched comparison.
- Verification gap: Insufficient evidence in the retrieved excerpts to confirm Sergey Rodionov’s baseline1 figures of 99 RHAE and 8,347 actions, or the claimed highest reasoning setting.
How “Code as Model” works
- Record hypotheses: A Markdown rulebook stores the agent’s current understanding of objects, action meanings and environment dynamics; unknown rules can remain explicitly unspecified rather than being treated as settled facts.
7
- Build a simulator: The agent translates its rule hypotheses into executable Python code representing a simplified world model.
- Plan and revise: It uses the model to look ahead, then follows a continual cycle of observation, reflection, rule revision, compilation and verification.
- Correct transformations: A retrieved example describes learning that stepping on a pad rotates a carried badge 90° clockwise.
- Recover from bad edits: Reporting says Git records rulebook and code changes, allowing the agent to inspect history and restore earlier versions when an edit proves wrong.
- Validation detail: Insufficient evidence in the retrieved excerpts to confirm the specific procedure of replaying recorded moves and comparing predicted and actual 64-by-64 frames.
The important distinction is that “self-improvement” here means revising external knowledge and programs while the LLM remains frozen—not recursively retraining the base model.1
6
Ablations and Pong
- Ablations: Insufficient evidence in the supplied excerpts to independently verify the roughly 9% action reduction across three games or the two-model result of 899 versus 597 actions. Even taking those figures as stated, three games and one game respectively would provide narrow, suggestive evidence—not a benchmark-wide estimate of either component’s contribution.
- Pong outcome: The paper’s retrieved excerpt reports a learned feedback controller winning each of three evaluated episodes 21–0, with different openings and no further LLM calls.
- Pong learning process: Insufficient evidence to verify the approximately 9,500-frame exploration budget or the precise bounce-physics inference procedure. The supported conclusion is narrower: a learned executable controller could subsequently play those evaluated episodes without ongoing language-model inference.
Limitations and interpretation
- Base-model dependence: The reported system uses Claude Opus 5, so the result demonstrates the collaborators’ agent architecture around that model, not the performance of a Huawei-developed foundation model.
9
- Public-set scope: The reported clearance concerns public games; it does not establish equivalent performance on hidden evaluation games.
- Action efficiency is not total cost: Fewer environment actions alone do not establish lower token use, latency or monetary cost. Insufficient evidence here to quantify the claimed token-heavy exploration.
- Generalization remains open: Insufficient evidence for performance in noisy, partially observed or changing real-world environments. The available evidence supports a promising method for explicit world-model learning and executable planning, not a demonstrated general-purpose real-world agent.
Studio Global AI
Continue your research
This page includes a source-backed answer you can continue inside Studio Global.
People also ask
What is the short answer to "How did Memento 3, developed by Huawei Noah’s Ark Lab, University College London and partners, clear all 25 games and 183 levels of the ARC AGI 3 public benchmark—which presents unfamiliar pixel style games without explaining their rules, goals or actions—in 7,518 actions versus"?
Memento 3’s reported gain came from changing the agent’s external memory and planning machinery—not the language model’s weights: it repeatedly wrote, tested and revised a rulebook and executable simulator, then used that simulator to plan
What are the key points to validate first?
Memento 3’s reported gain came from changing the agent’s external memory and planning machinery—not the language model’s weights: it repeatedly wrote, tested and revised a rulebook and executable simulator, then used that simulator to plan Verification gap: Insufficient evidence in the retrieved excerpts to confirm Sergey Rodionov’s baseline1 figures of 99 RHAE and 8,347 actions, or the claimed highest reasoning setting.
What should I do next in practice?
How “Code as Model” works Record hypotheses: A Markdown rulebook stores the agent’s current understanding of objects, action meanings and environment dynamics; unknown rules can remain explicitly unspecified rather than being treated as set
Sources