We present a Test-time World-model Inference (Twin) system, in which a frontier coding agent writes an executable world model for completing continual learning tasks, such as ARC-AGI-3 games. Traditional approaches hand-engineer such models, one custom design per task. Each game hides its rules and goal, and our system constructs them from simulation and interaction alone. Its inductive prior over grid games is strong enough to recover the true transitions of the game and the goal on nearly all levels. Replay validation happens in a twin world model. The harness enforces that an action is not made until the program reproduces every previous observed game transition. Each mismatch between a world model prediction and the actual action result becomes a counterexample that is used to repair the world model. Twin clears 179 out of 183 levels (97.8%), and does so more efficiently than humans in 158 out of 179 levels (88.3%). The system infers the goal before any reward on 156 of the levels it clears (87.2%), and in the remaining levels automatically discovers the goal by search. The benchmark scores completion and action efficiency, between 0 and 100, against humans playing each game for the first time. Played directly, the base model scores only 7.8%; an off-the-shelf harness increases it to 61.1%, whereas our twin world model increases the same base model to 93.3%, clearing 23 out of 25 games. Building a usable world model is simpler than anticipated, whereas the harder problem is inferring the right goal.
Figures & tables
Figure 1: The Twin loop, illustrated with states from the level ft09. Left of the scored boundary, every step is unscored. The agent appends each real transition to the log D and rewrites the twin until Validate replays every logged transition without a mismatch. Explore then ranks repair targets, or proposes reachable goal candidates when no route to the goal is found. Planning runs only inside a validated world model. Plan searches the world model for the shortest route, and ExecuteChecked submits that route one move at a time against the real ARC-AGI-3 simulator. A halted move returns its transition as a counterexample.
System
Score
Won (/25)
Levels (/183)
Human reference
100.0
25
183
OPINE-World
78.4
20
160
Prime Agent
78.3
–
164
EWM
63.8
14
146
Codex
61.1
13
148
Twin (ours)
93.3
23
179
Table 1: Aggregate performance on the 25-game, 183-level public set. Score averages the benchmark action-efficiency values over games. Won and Levels count full clears and cleared levels, and all systems use the same published human baselines. The human row is the score’s normalization reference, not a measured system. Twin , as shown in the last row, scores 93.3 of 100 using GPT-5.6 Sol, ahead of OPINE-World 78.4, Prime Agent 78.3 using GPT-5.6 Sol, EWM 63.8 using GPT-5.5, and Codex using GPT-5.6 Sol 61.1.
Figure 2: Per-game ARC-AGI-3 action-efficiency scores for Twin , OPINE-World, and EWM, ordered by Twin score. For each cleared level ℓ , the raw efficiency factor is eℓ=min{1.15,(hℓ/aℓ)2} , where hℓ and aℓ are the human baseline and agent action counts; uncleared levels have eℓ=0 . The level-index-weighted game score is capped by the weighted completion fraction, at most 1.0, then scaled to 100. Each game carries three dots joined by a gray range bar, Twin blue, OPINE-World orange, EWM violet. The blue dot leads or ties on 22 of the 25, with OPINE-World ahead on g50t, sp80, and sc25.
Figure 3: Action efficiency relative to the human reference on the 23 games Twin wins. Each dot places one game by its human action count against Twin ’s, both axes logarithmic. Green dots lie at or under the dashed parity line, no more actions than the first-time human, 21 of the 23 at a mean ratio of 0.61× . The red dots tn36 ( 1.5× ) and ka59 ( 1.6× ) lie above.
Measurement
Tests
Exact frames
Cell match
Replay of recorded transitions
11,314
99.4%
0.9996
Engine match, observed pairs
2,354
98.9%
0.9985
Engine match, unobserved pairs
20,790
70.1%
0.9956
Cross-level or cross-game probes
98
16.3%
0.9014
Table 2: Five next frame comparisons pooled over all 25 games. Each game’s twin is the world model the agent learned while playing that game. Every test gives the twin one state–action pair and grades the twin’s predicted next frame against a reference. The rows differ in which pairs they use and which reference grades them. Tests counts the comparisons. Replay uses the pairs from the original run, graded against the next frames recorded in that run. Observed-pair engine match grades those same pairs against a fresh rerun of the ARC-AGI-3 engine. Unobserved-pair engine match , the primary generalization test, grades previously unobserved pairs against the rerun engine. A previously unobserved pair is a visited state with an action never taken from that state. sp80’s probes are graded under the counter state a log-order replay produces, since its twin stores a call counter. Boundary probes grade 98 cross-level or cross-game pairs with no recorded next frames against the rerun engine. Exact frames counts a prediction as exact only when it agrees with the reference on every one of the 64×64=4,096 cells. One differing cell fails the whole frame. Cell match is the mean fraction of agreeing cells. The twins exactly predict 70.1% of previously unobserved pairs. Figure 4 reports results by game.
Figure 4: Per-game world-model accuracy against the ARC-AGI-3 engine. Every point compares next frames predicted by a game’s twin with next frames the engine returned. A prediction counts only when all 4,096 cells match. Left: open circles show how often the twin’s predictions were exact during the run, one prediction before every action. Filled circles show how often the final twin is exact on pairs it never saw, a visited state with an action never taken from it. A filled circle to the right of its open circle means the twin improved during the run. The open circle averages the whole run, early mistakes included. The filled circle tests only the final, fully repaired twin. Games are ordered by the filled-circle rate. Right: the same measurements pooled, with the number of pairs under each bar. The values correspond to Table 2 . Filled-circle accuracy ranges from 9.9% on wa30 to 98.8% on tu93. wa30 replays its recorded run almost perfectly yet fails on pairs it never took: exact replay does not guarantee generalization.
Figure 5: Spatial distribution of final-twin errors on previously unobserved pairs. Each panel shows, in grey, one visited state from that game. Red shows, per cell on a shared scale, how often the twin’s prediction is wrong there across the game’s previously unobserved pairs. Panels are ordered by the exact-frame accuracy shown in their titles. Most errors are spatially concentrated: although cd82 misses 43.3% of its next frames on unobserved pairs, only 11 of 4,096 cells account for 90% of its total disagreement, all within the one-cell progress bar along the bottom edge. The hottest 5% of cells contain a median of 60% of each twin’s disagreement mass. wa30 is the diffuse exception, with errors spread across the playfield and no row above 2.5% of the mass. tu93, ar25, and ft09 remain nearly error-free. Tests begin only from visited states, and rates include only tests where the twin returns a frame.
Figure 6: We test the same bounded BFS in two worlds, once inside the twin and once inside the ARC-AGI-3 engine, from the same restored start state, over all 179 completed levels. The race compares which model’s forward dynamics has a better search to the finishing goal state, the twin’s or the ARC-AGI-3 game engine’s own. One row per game, one cell per level, in play order. The left block, BFS: 6 before the win , branches six recorded actions before each level’s win; the right block, BFS: from the start , branches at the level’s entry state. Blue: both searches find a finish. Orange: only the engine’s does. Light blue: only the twin’s search returns a plan. Dark gray: neither search succeeds within budget. Budgets: 20,000 states per search in the left block, 200,000 in the right, wall clocks as safety caps only. Six actions out the searches broadly agree, both finishing on 102 levels; from the start the engine finds 61 and the twin 47, agreeing on 33.
Figure 7: The endgame gallery, one board per golden run (25), at the moment a level is about to be won. Numbered dark circles: the run’s recorded finishing moves, never a plan. Gray circles and a gray twin row mean the twin returned no plan; the row then repeats the recorded route the circles trace. Stacked clicks share one badge; bp35’s route is suppressed, its camera re-anchoring every action. The title is the twin’s verdict for that finish. optimal : certified shortest. +3 : three extra actions. fails : failed in the engine. no plan : none returned. claims solved : the twin returns the empty plan, its goal firing early; never executed, so outside the 82. A plus sign and a filled square mark plans that stop one interact press short and win once that press is appended. Rows below each board: twin plan (blue) over engine plan (orange), red where they differ; an ellipsis ends a row too wide for its panel. The gray line counts the whole level’s actions, the first-time human baseline against the agent’s run. tn36’s engine row is the recorded finish, its search defeated by the frame merging the text describes. Across the 174 analyzed levels: 107 plans, 82 wins, all 82 certified, 80 exactly optimal.
Table 4: Aggregate evaluation outcomes and compute. Twin and no-twin ablation values are recomputed from their released run artifacts. EWM values come from its published run. Tokens : total processed tokens. Actions : scored environment actions. tok/act : processed tokens per scored action. Wall-clock time is omitted because EWM does not publish it, and OPINE-World is absent because it reports no token counts. Twin scores 93.3 on 2.60B total tokens against EWM’s 63.8 on 20.0B. The no-twin ablation runs cheaper, 48k tokens per scored action and 1.06B in total; the twin’s action efficiency is bought with compute.
Figure 9: End-to-end control flow for Algorithm 1 . Node borders identify the responsible component: blue for the harness, violet for the coding agent, amber for walls and decision points, red for scored actions, and green for goal-consistency checks. The spine runs Perceive , Validate , Plan , ExecuteChecked . The two explore arms are the cases of Eq. 4 : arm 1 answers the dynamics wall with ranked repairs, and arm 2 answers the goal wall with champion states, a proposed predicate, and the consistency check before any plan is built on it. Every node before the scored one is free, so a scored action is taken only when nothing unscored remains. Three outcomes re-enter the loop, each carrying evidence: a transition mismatch returns to Validate with a counterexample, a level boundary returns to Perceive with the first reward, and a predicted goal reached without a boundary retires the candidate and restarts goal discovery.
Figure 10: Goal discovery and consistency checking, with node borders as in Figure 9 . At the goal wall the harness searches the twin without a reward model R^ , at depth 14 over 30,000 nodes, and keeps the strongest state per progress signal, shorter paths winning ties. The coding agent turns the top champion’s salient change into the tentative predicate R^ . The green check requires R^ to reject every recorded frame, each one a certified negative, before any plan is built on it. A survivor is tested along its cheapest route, one scored action at a time under the halt-on-mismatch guard, and each outcome settles a different question. A level boundary confirms the candidate and becomes the first reward example. The predicted goal reached without a boundary retires it: the physics were right and the objective was wrong. A transition mismatch is the reverse: the physics get repaired and the goal candidate stays tentative. The released harness enforces the green check as a hard constraint, refusing any predicate that fails it; during the reported runs the UpdateGoal prompt asked the agent to apply the same check.
Figure 11: Every scored action in the 25 runs, one row per game, with games sorted by score. The horizontal axis is run time, measured in scored actions and normalized so that every run spans the same width. Blue marks a validated plan step whose prediction was correct, amber a deliberate probe, and red a plan step whose failed prediction produced a counterexample. The arrow marks the action at which ar25 learned its mirror rule. Probes account for 7.1% of the 11,562 actions the agent chose, and the remaining 92.9% execute plans tested in the twin.
Configuration (same base model)
Score
Direct play, leaderboard
7.8
Direct play, official evaluation, OpenAI’s run
13.3
+ retained reasoning + compaction
38.3
Off-the-shelf Codex harness
61.1
Prime Agent harness
78.3
Twin
93.3
Table 6: Direct play and world-model methods on the 25-game public set, on the benchmark’s action-efficiency score. Row one is the leaderboard’s direct-play score. Rows two and three are OpenAI’s own runs as published ( OpenAI 2026 ) : row two the official evaluation, row three the same run plus the two memory settings, retained reasoning and history compaction. The last three rows are from Table 1 and hold the base model fixed while changing only the harness. The three direct-play scores are as published, not rescored here. The strongest of them remains 55.0 percentage points below Twin ’s 93.3.
Figure 12: Each panel displays a single ARC-AGI-3 game, where the twin predicts the next frame and compares that prediction against the ARC-AGI-3 engine’s frame given the same action. Each data point is a rolling average, the ratio of exactly matched frames over the ten most recent predictions, so one miss moves the curve by ten percentage points. The first data point on the line is produced after ten actions have been submitted to the simulator, when the first ratio is recorded. The x axis is the action index for that game, so the value at 100 shows the frame prediction accuracy at the 100th action. Every prediction is hash-stamped before its action is sent, committed ahead of the answer. Vertical dashed lines are level boundaries. The background is a heat map of each level’s benchmark score, red at 0 to green at 100 on the printed scale; an unfinished level scores 0, so the two unsolved runs end in deep red. The dotted line pools all 25 runs: 8,532 of 10,744 predictions with a recorded outcome were exact, 79.4%. Each title gives the run’s exact rate over its whole history, the share of all its predictions, first to last, that matched exactly, plus levels cleared, red for the two unfinished games. The title counts every prediction while the curve counts the last ten, so ls20 reads 75% even as its final window falls. Read each curve as that game’s training signal. Flat at the top on green, lf52 and ar25, means the rules were learned once and the game came easy. Dips at a boundary that recover, sb26 and g50t, are new mechanics becoming repairs. Red under a flat curve, tn36, is accuracy spent hunting the goal rather than the rules. A curve that keeps dropping and rising, ka59’s long middle or sc25’s opening, means the rules keep breaking and the game is hard; sc25 and sp80 stay unsolved.
Figure 13: bp35’s answer to partial observability, pay once per new place. Left to right: the twin predicts everything except the strip about to appear and concedes it. One scored probe looks. The revealed strip is filed in model.py under a hash of the frame it was seen from, 56 entries by the run’s end. The planner then searches through that camera move at no cost. Rebuilt this way, the twin reproduces all 407 unique recorded transitions exactly.
Figure 15: The observation, as the agent receives it. Each step the harness writes the current board to frames/latest.json , 64 rows of 64 integers 0 to 15, with a rendered frames/latest.png beside it and a state/turn.json carrying step, level, and the legal actions. Shown is ft09’s first frame from the golden run, rows and columns truncated for width. The agent’s observation is the integers, read as text. The rendered image is written beside them for the record; the agent has no vision channel, though a few runs parsed the image’s pixels with code.
the action then completed…
the twin claimed…
a level
nothing
total
a win
138
508
646
nothing
41
10,870
10,911
total
179
11,378
11,557
Table 11: Every goal claim across the 25 runs, graded by what the action did. A goal claim is mechanical, not conversational. At each submission the harness runs the twin’s step on the current frame and the action about to be sent, giving the twin’s predicted next frame, the same prediction Figure 12 grades. It then runs the agent-written goal_reached on that predicted frame. If goal_reached returns true, the twin is claiming that this action ends the level. The claim is logged before the outcome arrives, so it cannot be revised. Every action then lands in one of four cells. If goal_reached claimed a win and the level completed, the claim was right, 138 times. If it claimed a win and nothing completed, the claim was wrong, 508 times. If it claimed nothing and the level completed, it missed a win, 41 times. If it claimed nothing and nothing completed, it was correctly quiet, 10,870 times. The 11,557 graded actions are the runs’ 11,614 scored actions less 52 automatic resets, which write no log entry, and 5 entries that share a step index. Recall is 138/179=0.771 , precision 138/646=0.214 , and the false-alarm rate over non-winning actions 508/11,378=0.045 . Most wins are claimed, and most claims are wrong.
Figure 16: Where a false claim lands inside its level. One row per game, precision printed at the left. Each level’s actions are cut into ten equal parts, and orange marks the parts holding that game’s false claims, darker for more. The blue column counts correct claims; those land at the finish by construction. The strip beneath pools all games against the dashed uniform 10%. The map places 334 of the 508 false claims; the remaining 174 came after a run’s last completed level, where position cannot be measured. The question is whether false claims crowd the finish, as a predicate firing slightly early would leave them. They do not: the median lands 0.533 through a level, and the final tenth holds 12.6% against the uniform 10%.
ARC-AGI-3 turns abstraction into an interactive problem of skill acquisition. A player must infer an unfamiliar game's rules, hidden state, and goal while maintaining action efficiency because every move counts. We formalize these environments as parameterized rendered deterministic Moore machines and introduce Tycho, a coding-agent system that constructs and uses game-specific models during interaction. Tycho separates actionable observations from intermediate animation, level-completion, and game-over frames. From this structured history, an agent can model, test, plan with, repair, or bypass a free-form executable hypothesis. In one matched public-set run per policy, we compare four orchestration policies on all 25 public games using Claude Opus 4.8 under matched inference budgets. Actor-requested delegation to a model builder obtains the highest observed mean Relative Human Action Efficiency (RHAE), 88.49. With this selected policy, GPT-5.6 Sol and Opus 5 both reach 100.00 RHAE and complete all 183 levels. Their game-balanced first-run human-replay midranks are 98.5 and 100.0. Opus 5 uses 61% fewer scored actions than the aggregate official human baselines. Automatic repair after verification failures produces models that reproduce observed transitions much more accurately, yet reaches only 83.07 RHAE. Transition match indicates whether a simulator reproduces observed dynamics, not whether it has identified the objective or improves the next action. Strong play also requires deciding when to construct, repair, use, or bypass a model. We call this joint problem active abstraction: generating a testable model from costly interaction and deciding when acquiring or using it is worth its cost.
Jens Lehmann, Andrei Aioanei, Sahar Vahdati
Dresden University of Technology · Amazon · TIB – Leibniz Information Centre for Science and Technology +1
World modeling is an unsettled field: architectures, training objectives, and state representations interact in complex ways, and no single recipe dominates across environments. This makes it an ideal testbed for AI coding agents acting as autonomous researchers--a setting in which the improvement direction is not specified in advance, unlike the engineering-to-spec tasks that dominate current agent benchmarks. We introduce AutoWorldModel-Bench, a closed-loop benchmark in which frontier coding agents autonomously improve a provided world-model starter under a fixed compute budget. The benchmark spans eight game environments under a unified structured-state representation--ground-truth entity state extracted from each game and consumed through a shared tensor format--which isolates dynamics modeling from perception and enables minutes-per-run iteration. Across 64 sessions, Codex-5.4 and Claude Opus 4.6 improve their starter on 63; in 91% of sessions the winning edit is a non-trivial research-style modification--a new objective, representation, rollout procedure, or architectural change--rather than a hyperparameter tweak. Our benchmark offers a setting in which frontier coding agents can be evaluated on open-ended research rather than engineering-to-spec problems.
Marjan Moodi, Xuankang Zhu, Fernando De Mesentier Silva +2
World-model synthesis aims to turn interaction experience into an internal model of environment dynamics. Existing symbolic approaches often fit observed transitions or mixtures of local rules, but they do not produce a complete executable program that can run independently of the real environment. We present Mind-Studio, a framework that synthesizes executable pygame-style world models from state-action-next-state trajectories using large language models. Mind-Studio combines entropy-selected traces with a lightweight game skill file containing object, action, and static scene information extracted from screenshots. We evaluate synthesis quality with a K-step lookahead fidelity protocol that compares generated world-model rollouts against Real-ALE rollouts from the same state. On Montezuma's Revenge, Mind-Studio improves chosen-action next-state prediction from 0.3% for PoE-World to 48.7% while verifying 5 of 8 subgoals; across Alien, Assault, and Skiing, it achieves stronger branch-level fidelity than prior learned lookahead sources.
Yifei Dong, Mingen Zheng, Linquan Wu +2
Hong Kong University of Science and Technology · City University of Hong Kong · University of Edinburgh +1