Learning through exploration and using newly acquired knowledge to achieve goals are central capabilities of general intelligence. Understanding these capabilities requires tracing how learners build, use, and revise world models during interaction. We construct the first dataset that records how humans discover and use the mechanisms of novel environments, using ARC-AGI-3 as a testbed: 144 trajectories from 64 participants across 25 public games, each pairing time-synchronized actions with think-aloud audio, plus 250 trajectories from large language model (LLM) agents with and without harnesses on the same games. We annotate mechanism triggers and recognition, action purposes, and errors and recovery, linking changes in understanding to the observations and actions that support them. We analyze characteristics of human cognition and compare them with those of LLM agents. Both groups anticipate mechanisms and use familiar analogies. Qualitative representations are more common in successful human trajectories than in successful LLM trajectories (50.0% versus 14.3%). Humans more often explore unfamiliar situations without a concrete hypothesis, while hypothesis testing is common in both groups. We also trace how learners recover through belief revision, replanning, and execution correction. As LLMs with well-designed harnesses reach saturated performance on public ARC-AGI-3 games, our cognitive analyses motivate next-generation benchmarks that challenge familiar priors, require accumulated evidence and abstraction, and test planning through subgoals, experimentation, and recovery. Our dataset is available at https://arc3-human-think-aloud.github.io/.
Figures & tables
Figure 1: Human trajectories and concurrent think-aloud data. Key frames and aligned think-aloud excerpts trace participants’ attempts to solve the first level of (a) LS20, (b) BP35, and (c) FT09. Each row includes the key mechanics needed to solve that level.
Figure 2: Data annotation. Annotations cover three components: world-model changes , tracking mechanism triggers and when and how learners recognize mechanisms; action generation , distinguishing goal pursuit, information gathering, and actions serving both purposes; and errors and recovery , identifying mechanism, planning, and execution errors and linking them to any subsequent corrections.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Inferring a goal before reaching it (TU93). The participant identifies the green location as a candidate goal before first reaching it.
Figure 4: Revising an analogy-based hypothesis (BP35). The participant retrospectively describes a Mario-style teleport-pipe interpretation, rejects it, and tries another route.
Figure 5: Pursuing a subgoal (TU93). The participant plans an approach and executes a sequence that removes the enemies.
Figure 6: Combining goal pursuit and information gathering (CN04). Testing candidate shape connections both advances the solution and reveals which configurations work.
Figure 7: Exploring unfamiliar objects (LF52). The participant tests clicks and movement before proposing an explanation for the purple objects.
Figure 8: Testing a hypothesis (SC25). The participant predicts teleportation, tests the yellow pattern, and confirms the predicted effect.
Figure 9: Investigating a prediction error (SC25). An unexpected teleport destination prompts further tests and the hypothesis that returning requires a larger character.
Figure 10: Revising a mechanism condition (CN04). Resizing in place leads the participant to retract the belief that entering another shape is necessary.
Figure 11: Discovering the controls (LF52). After unsuccessful clicks, the participant identifies the arrow-key controls and completes the level.
Figure 12: Clarifying the connection goal (CN04). The participant tests how the shapes should fit together and reports understanding after completing the level.
Figure 13: Replanning under coupled movement (LP85). Moving the yellow object disrupts the orange object’s placement, prompting a revised sequence that places both at their targets.
Figure 14: Correcting an input (LP85). A click just outside the button has no effect; a small adjustment activates it.
Condition
Triggered/obs.
Coverage
Dexp≤5
Claude direct
218/238
91.6%
42.2%
GPT-max direct
177/195
90.8%
46.9%
GPT-high direct
159/182
87.4%
39.6%
Astra Standard
266/285
93.3%
47.7%
Vista –Claude
294/320
91.9%
54.1%
Vista –GPT
305/320
95.3%
49.5%
Appendix
Table 2: Trigger coverage and supplementary exploration statistics. Observable instances are main mechanisms introduced at reached levels.
Condition
Recognition rate
Paired/obs.
Dind<0
Claude direct
84.0%
193/238
11.4%
GPT-max direct
79.0%
150/195
12.7%
GPT-high direct
63.7%
114/182
13.2%
Astra Standard
77.5%
214/285
9.3%
Vista –Claude
92.8%
284/320
11.3%
Vista –GPT
93.1%
291/320
9.6%
Appendix
Table 3: Recognition coverage, paired instances, and recognition preceding a trigger.
Condition
Eligible/completed
Missing recognition
Claude direct
45/95
50
GPT-max direct
21/48
27
GPT-high direct
10/30
20
Astra Standard
14/137
123
Vista –Claude
70/183
113
Vista –GPT
79/183
104
Appendix
Table 4: Eligible planning levels and missing recognition evidence.
Learning to complete tasks in unfamiliar environments with unknown rules remains a key challenge for LLM agents. Current LLM agents often record their discoveries in prose, which may not provide a compact, explicit account of how the environment works. Inspired by how scientists organize observations into testable, predictive theories, we introduce Schema, an agent harness that organizes learning and action through interactive program induction. The LLM agent decides what to investigate and how to act, expressing its evolving understanding of the environment as executable programs. The harness consists of a persistent program workspace and a small set of interfaces for checking these programs against the interaction history, planning within them, and executing plans under step-by-step verification. Schema raises ARC-AGI-3 RHAE from 58.7% to 99.2% with the same base model, solves 100% of the public DiG-bench games, and reaches the median performance of the top-50 human players on MazeBench. Extensive analysis shows the effectiveness of Schema in unknown mechanism discovery, and ablations confirm the contribution of each component.
Guanning Zeng, Jiani Wang, Wenjie Ma +8
Carnegie Mellon University · Impossible Research · UC Berkeley
Learning from experience is essential for LLM agents to adapt to unfamiliar and dynmaic environments. Evaluating this ability is therefore important for understanding how effectively agents acquire and use new knowledge. Existing benchmarks have sought to evaluate this ability, but they primarily evaluate tasks whose rules are provided in the instructions or already familiar to pretrained models, making it difficult to distinguish learning from interactions from reasoning with existing knowledge. To address this, we introduce Learn2Play Bench, a benchmark of newly designed text-based games, whose rules are novel or counterintuitive, requiring agents to acquire knowledge through interaction rather than rely solely on pretrained knowledge. These games provide reproducible feedback and automatic scoring, enabling controlled evaluation of learning across repeated attempts. We also vary game instances to test whether agents can apply what they have learned to new situations. Therefore, we evaluate how backbone models, self-evolving methods, and agent harnesses affect agents' learning ability, revealing three findings: (1) Experience retention: Retaining complete records of actions and feedback can support more effective learning than summarizing these experiences into rules or strategies. (2) Human agent gap: Top-performing human players achieve higher peak scores than the evaluated agents. Human explore more varied strategies, and repeat actions less. (3) Harness matters: With the backbone fixed, changing the harness can improve performance while reducing estimated inference cost. Together, these findings provide insights into how LLM agents learn from experience and suggest directions for future work to improve their learning ability. Project website: https://liushiliushi.github.io/learn2play-bench-website/
We introduce ARC-AGI-3, an interactive benchmark for studying agentic intelligence through novel, abstract, turn-based environments in which agents must explore, infer goals, build internal models of environment dynamics, and plan effective action sequences without explicit instructions. Like its predecessors ARC-AGI-1 and 2, ARC-AGI-3 focuses entirely on evaluating fluid adaptive efficiency on novel tasks, while avoiding language and external knowledge. ARC-AGI-3 environments only leverage Core Knowledge priors and are difficulty-calibrated via extensive testing with human test-takers. Our testing shows humans can solve 100% of the environments, in contrast to frontier AI systems which, as of March 2026, score below 1%. In this paper, we present the benchmark design, its efficiency-based scoring framework grounded in human action baselines, and the methodology used to construct, validate, and calibrate the environments.