Mine Odyssey: Benchmarking Spatial Agentic Intelligence in the Wild

Yuxuan Cao; Junlong Li; Hao Li; Junxian He

cs.AIOct 8, 2026

Mine Odyssey: Benchmarking Spatial Agentic Intelligence in the Wild

Authors: Yuxuan Cao, Junlong Li, Hao Li, Junxian He

Organizations: The Hong Kong University of Science and Technology

Abstract

Advances in foundation models are driving efforts to introduce agents to assist people in the physical world. Such agents require agentic spatial intelligence: exploring unfamiliar environments, updating spatial understanding through interaction, and adapting actions based on feedback to sustain progress toward a sequence of goals. Existing benchmarks cover only a limited range of spatial layouts, scales, and traversal requirements. We introduce Mine Odyssey, a benchmark for evaluating agentic spatial intelligence using Minecraft reconstructions of real-world locations. It comprises 180 tasks covering 30 such locations across 20 countries and regions on five continents, including 20 outdoor and 10 indoor settings. These settings span diverse spatial scales, layouts, terrains, and connectivity patterns, from Midtown Manhattan and rural Entrup to Santa Lucía Hill and Buckingham Palace. We select meaningful waypoints, such as landmarks, buildings, and rooms, and manually verify their accessibility. Each task provides a natural-language instruction specifying which waypoints to visit and in what order. Completing these tasks requires agents to find accessible routes and entrances, open doors, and move between levels using stairs and ladders, while monitoring their progress and recovering from navigation errors. Across eight evaluated state-of-the-art models, GPT-6 Astra achieves the highest success rate of 85.6%. However, the second-best model, Claude Opus 5.5, completes 73.9% of tasks, while the strongest evaluated open-weight model, DeepSeek-V4.1-Flash, reaches 23.9%, highlighting substantial room for improvement in the agentic spatial intelligence of current models. Comprehensive analyses and ablation studies on Mine Odyssey reveal current models' limitations and provide insights for advancing agentic spatial intelligence.

Figures & tables

Figure 1: Overview of Mine Odyssey environments and task construction. (a) 30 Minecraft reconstructions and geographic coverage. (b) Task construction with White House and Ueno Park examples: reference locations are mapped to waypoints and ordered into navigation tasks. S denotes the start; numbered markers and dashed lines indicate visit order.

Figure 2: Representative terrain, spatial constraints, and interactions across the diverse collection of maps in MINE ODYSSEY.

Figure 3: Agent interaction and independent verification in MINE ODYSSEY.

Proprietary models
Model	SR (All) $\uparrow$	SR (Outdoor) $\uparrow$	SR (Indoor) $\uparrow$	CC $\uparrow$	SPL $\uparrow$	Turns	PL
GPT-6 Astra (medium)	85.56	87.20	81.82	89.04	53.70	105.8	1033.3
Claude Opus 5.5 (max)	73.89	78.40	63.64	86.11	50.35	250.6	1051.7
Claude Opus 5 (max)	50.56	58.40	32.73	67.53	24.29	242.8	2020.2
Claude Opus 5 (high)	44.44	51.20	29.09	59.99	17.09	231.4	2921.5
Claude Opus 5 (medium)	39.44	47.20	21.82	57.48	12.95	233.8	3311.3

Table 1: Main results on 180 tasks, with success rates broken down by 125 outdoor and 55 indoor tasks. SR, CC, and SPL are percentages; PL is the mean sampled 3D distance over all tasks.

DeepSeek-V4.1-Flash
Setting	SR	CC	SPL
HUD	$30.0\;(+6.1)$	$52.1\;(+7.8)$	$12.8\;(+1.2)$
Multi-view (6)	$26.1\;(+2.2)$	$48.7\;(+4.3)$	$11.3\;(-0.3)$
Third-person view	$28.5\;(+4.5)$	$50.0\;(+5.3)$	$13.5\;(+1.8)$
Route guidance	$21.1\;(-2.8)$	$46.3\;(+1.9)$	$9.7\;(-1.9)$
Midnight	$22.2\;(-1.7)$	$46.3\;(+2.0)$	$10.1\;(-1.5)$

Table 2: Ablation across six conditions: SR, CC, SPL.

Figure 4: The default view and six ablation conditions are shown outside Hofburg Palace.

Figure 5: Local stays over at least ten turns within one block. Blue: episodes per task (left axis). Orange: task-mean share of aligned turns (right axis).

Figure 6: (a) White House: Astra takes a detour; DeepSeek revisits rooms. (b) Plaza Hotel: Astra returns downstairs; Grok remains upstairs. S/E: start/end; M: shared upper landing; stars: targets. Times cover the S–E segments.

Figure 7: Discovering an upstairs route in Hagia Sophia 003. Astra explores the side branch and ascends, while Opus 5 approaches the end door and leaves without exploring the branch.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Map	Tasks	Map	Tasks
Buckingham Palace	5	Cape Town / Bo-Kaap	9
Chain Bridge	1	Copacabana Waterfront	3
Dublin Departments District	7	Entrup	6
Hagia Sophia	3	Hofburg	7
Innopolis	9	Memorial Hall Park	1
Miljacka Riverside	4	Harbor City Shipyard	11

Appendix

Table 3: Formal task counts in the selected benchmark.

Figure 8: In-game world-map views of all 30 benchmark environments.

Factor	Default
Time / weather	Noon (frozen) / clear (frozen)
Perspective	First person
Observation	Event-based, single view
Player scale	1.0
Optional HUD / minimap panel	Disabled
Full-screen world map	Available

Appendix

Table 4: Default settings for the main comparison. Controlled variants override the relevant entries.

Figure 9: SR versus estimated cost per task, with cost on a log scale. Connected points show Claude Opus 5 effort settings. Costs average over all 180 tasks, including failures, using the accounting in Appendix D . The horizontal interval indicates the estimated cost range for Opus 5.5.

Figure 10: SR versus output-token usage. Left: the eight main model configurations. Right: Claude Opus 5 at low, medium, high, and max reasoning effort; connected points show increasing effort. Values use all 180 tasks, including failures; K denotes thousands. Both panels use linear axes, with different SR ranges.

Figure 11: Outcomes over 180 tasks per model. First stop means the first intermediate stop, or the destination if none. Deaths count as failures even after all stops. All shares use the full 180-task denominator. Model configurations match the main comparison.

Figure 12: Information access and command structure across eight models. Values are percentages of execution requests, computed per task and averaged equally across tasks; categories can overlap. Map key counts map toggles, Look counts camera-orientation commands, State query includes queries combined with controls, and Image proc. counts explicit image-processing operations. Shell sequence counts command sequences or pipelines within a request, and Loops counts requests containing shell or Python loop syntax.

Figure 13: From map pixels to a movement target. Step 176 captures the map; Step 177 computes the estimated U20 coordinates; Steps 179 and 181 use the estimate for orientation and movement. S marks the player position and G the estimated goal; dashed lines indicate pixel offsets, not an executed route. The displayed map provides context because the measured PNG is unavailable. The adopted scale is uncalibrated, and execution does not establish arrival.

Figure 14: From composition to reuse in Plaza Hotel, Steps 35–36. Step 35 executes one circuit from Y40 to Y48; Step 36 executes all three circuits from Y48 through Y56 and Y64 to Y72. The cutaway uses the retained world’s native textures, with selected walls and room volumes removed. Command-window position samples and matched state endpoints are projected at their original heights over the scene for visibility. Matching colors link each step in the enlarged scene to its height transition and complete shell command. The original observation screenshots are omitted. Only whitespace is reformatted in the code. Later steps are outside the displayed window; the cutaway is a rendering, not an agent screenshot.

Explore similar work

Jun 8, 2026cs.AI

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world. However, existing benchmarks predominantly rely on passive evaluation (e.g., static VQA) or simulator-specific pipelines, failing to assess general interactive spatial understanding. We introduce SpatialWorld, a unified benchmark designed specifically for evaluating the interactive spatial understanding of multimodal agents in complex real-world tasks. Integrating eight heterogeneous simulation backends under a shared, simulator-agnostic protocol, SpatialWorld features 760 human-annotated tasks across diverse domains (e.g., household routines, travel, social collaboration). Agents must solve tasks under vision-only partial observability, actively gathering egocentric visual evidence and expressing decisions via a unified, text-based action interface native to MLLMs. For reliable evaluation, each task includes a human-validated initial state, a reference trajectory, and a terminal-state verifier. Evaluating 15 advanced agents reveals that robust spatial task solving remains challenging: the strongest model, GPT-5, achieves an average task success rate (TSR) of only 17.4%, while the leading open-source model, Qwen-3.5, reaches 14.1%. Further analysis exposes a clear mismatch between task success and execution efficiency, alongside substantial domain-specific performance variations. These bottlenecks in active exploration and long-horizon planning position SpatialWorld as a rigorous testbed for future spatial agents.

Hongcheng Gao, Hailong Qu, Jingyi Tang +18

Tsinghua University · Chongqing University · Peking University +6

Spatial Reasoning Benchmarks Visual Spatial Reasoning Long-Horizon Agent Evaluation AI Agent Benchmarks Multimodal Agents Spatial Reasoning

Ask PDF

May 29, 2026cs.CL

MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft

Multimodal large language models (MLLMs) have shown strong capabilities in perception, reasoning, and action generation. However, their ability to sustain exploration in dynamic open worlds remains unclear. Existing embodied and game-based benchmarks often compress interaction into short-horizon tasks or entangle success with domain-specific game mechanics. In this paper, we introduce MineExplorer benchmark for evaluating open-world exploration capabilities of MLLM agents in Minecraft. We first filter atomic tasks whose solutions rely heavily on Minecraft-specific knowledge to better reflect general open-world reasoning. Then we organize the benchmark around a ReAct-style capability formulation and compose atomic tasks into implicit multi-hop tasks. To further construct reliable instances, MineExplorer uses a multi-agent synthesis workflow that jointly designs task graphs, sandbox scenes, and rule-based milestone evaluators. Human evaluation shows that the multi-agent synthesis workflow produces significantly more reliable instances than a single-agent baseline. Experiments with advanced MLLM agents show that open-world exploration remains challenging, as strong models can handle many single-hop tasks but degrade sharply when hidden prerequisites must be coordinated over longer trajectories. Further analysis finds that task difficulty tracks agent completion, and larger models or thinking modes do not consistently translate into better performance. Code and dataset are available at https://github.com/Jometeorie/MineExplorer.

Tianjie Ju, Yueqing Sun, Zheng Wu +7

School of Computer Science, Shanghai Jiao Tong University · 2Meituan

Long-Horizon Agent Evaluation AI Agent Benchmarks Multimodal Agents

Ask PDF

Jul 1, 2026cs.AI

AGI Maze as a Benchmark Framework for World-Modeling Agents

Large language models (LLMs) are powerful pattern-completion systems, but their default operating mode - predicting the next token from a static context - does not reliably produce persistent, manipulable representations of an external world. Many tasks that look like "reasoning" in text become substantially harder once the environment is partially observable, stateful, and requires memory and structured hypotheses about hidden state. AGI Maze is a lightweight framework for building such environments without requiring high-dimensional sensory inputs. It provides a family of grid-based maze tasks with a clean API and multiple difficulty regimes. The goal is to create benchmarks where agents must learn and use world state representations, not just infer a local rule over readily provided observations. We provide an initial evaluation of several vanilla LLMs on simple mazes showing that they fail to represent mazes internally at LLM inference time. We also introduce a baseline agent, which is allowed to use its message history as a working memory to construct descriptions of observations at agentic runtime. Although this can improve performance, it is still insufficient for an LLM agent to reliably solve even small mazes within a step budget that is more than enough for humans.

Alexey Potapov

SingularityNET Foundation

Agent Memory AI Agent Benchmarks LLM Agents

Ask PDF