Robotic manipulation has advanced rapidly in mapping visual observations and language instructions to actions. However, reliable manipulation in open-world environments requires robust spatial understanding: the ability to identify targets in complex layouts (spatial reasoning) and adapt to scenes that continuously evolve through interaction (spatial memory). Yet existing embodied benchmarks fall short in evaluating spatial understanding: they often introduce strong visual cues that bypass spatial reasoning, and focus on static scenes that offer little insight into an agent's capacity for spatial memory. To address this gap, we introduce GeoMind, a systematic benchmark for structured spatial understanding through tabletop manipulation. GeoMind comprises 54 evaluated task variants organized around spatial reasoning and spatial memory. It also incorporates a scalable pipeline that automates the generation and annotation of task instances with verified targets and executable interactions. Extensive evaluations across diverse embodied agents reveal their consistent weaknesses in spatial understanding. Our findings highlight that these agents struggle to track identities after spatial changes and convert relational or geometric constraints into executable actions. We release the GeoMind benchmark at https://github.com/Notfound-JH/GEOMIND-benchmark.git.
Figures & tables
Figure 1: Overview of GeoMind . GeoMind organizes structured spatial understanding along two diagnostic dimensions: spatial memory and spatial reasoning . The six task families cover different combinations of these dimensions. Surrounding examples illustrate how GeoMind couples spatial instructions with structured object layouts and scene changes. The instructions are illustrative; evaluated actions and success criteria are specified in Appendix D .
Benchmark
Design
Spatial Memory
Spatial Reasoning
Evaluation
Structure Generation
Scene Updating
Identity Tracking
Target Identification
Relational Grounding
Constraint Composition
Geometric Inference
Diagnostic Metrics
VIMA [ 23 ]
✓
✓
✓
✓
✓
✓
✗
✗
GENMANIP [ 30 ]
✓
✗
✗
✗
✓
✗
✗
✗
RoboMME [ 32 ]
✓
✓
✓
✗
✗
✓
✗
✗
MEMOBench [ 33 ]
✓
✓
✗
✗
✓
✗
✗
✓
RoboSPA [ 34 ]
✓
✗
✓
✓
✓
✓
✗
✗
Table 1: Comparison of spatial capabilities and benchmark design. ✓ : explicitly evaluated; ✗ : no explicit evaluation identified in the reviewed protocol.
Figure 2: Visualization of the six fundamental structural layouts and their derived variants.
Figure 3: Overview of the benchmark construction pipeline. The pipeline first proposes task descriptions and structure templates, then resolves spatial queries over instantiated scenes to derive ground-truth annotations, and finally validates candidate episodes through interaction-graph-based simulation verification.
Figure 4: Detailed information about GeoMind task suites. (a) shows how GeoMind-Aug is constructed from GeoMind-Base . (b) shows the distribution of evaluated variants across task families and augmentation types (20 Base, 7 Semantic, 7 Visual, and 20 Execution).
Embodied Agents
Spatial Reasoning
Spatial Memory
AVG
Target Iden.
Relational Grounding
Constraint Comp.
Geometric Inference
Scene Updating
Identity Tracking
Oracle
100.0
98.0
100.0
100.0
98.0
100.0
99.3
General Policies
OpenVLA
37.5
0.0
6.2
5.0
41.7
37.5
20.7
π0
4.2
0.0
0.0
25.0
6.3
20.9
11.3
π0.5
4.2
0.0
0.0
27.5
6.3
25.0
12.5
Spatially Enhanced Policies
SpatialVLA
12.5
0.0
0.0
17.5
45.8
8.3
18.3
Table 2: Success rates (%) of different paradigms of embodied agents on GeoMind-Base . Excluding the oracle, best and second-best overall results are highlighted with green shades, while best and second-best family-level results are highlighted with purple shades.
Policies
Corresponding Base
Semantic
Visual
Execution
OpenVLA
25.0
16.6 ( −8.4 )
12.5 ( −12.5 )
12.5 ( −12.5 )
π0
16.7
12.5 ( −4.2 )
16.7 ( +0.0 )
10.2 ( −6.5 )
π0.5
18.5
12.5 ( −6.0 )
16.7 ( −1.8 )
10.2 ( −8.3 )
SpatialVLA
18.8
8.3 ( −10.5 )
12.5 ( −6.3 )
12.5 ( −6.3 )
InternVLA-M1
37.5
27.1 ( −10.4 )
37.5 ( +0.0 )
35.4 ( −2.1 )
Spatial-Forcing
18.8
8.3 ( −10.5 )
12.5 ( −6.3 )
16.7 ( −2.1 )
Table 3: Success rates (%) on GeoMind-Aug and the reported base references. Parentheses show changes in percentage points.
Capability Dimensions
Task Family
Failure Rate ↓
SGE (m) ↓
TTE (m) ↓
Spatial Memory
Scene Updating
68.89%
0.189
0.330
Identity Tracking
85.00%
0.192
0.486
Spatial Reasoning
Target Identification
72.08%
0.116
0.811
Relational Grounding
98.33%
0.180
0.672
Constraint Composition
87.50%
0.142
0.627
Geometric Inference
86.67%
0.158
0.727
Table 4: Family-level failure rates and auxiliary spatial errors. Lower values are better.
Figure 5: Visualization of typical failure cases in GeoMind .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Setting
Simulator
Isaac Sim / IsaacLab
Benchmark package
GeoMind Lab
Robot
Franka Research 3 (FR3)
Gripper
FR3 parallel-jaw gripper
Low-level controller
Differential inverse kinematics
Physics frequency
480 Hz
Appendix
Table 5: Simulation and control settings used for all evaluations.
Type
Field
Description
Policy input
Instruction
Natural-language task instruction
Table-view RGB
Static external camera observation, 224×224
Wrist RGB
Wrist-mounted camera observation, 224×224
Joint state
Robot joint positions and velocities
End-effector / gripper state
Current end-effector pose and gripper state
Policy action
Translation
Relative Cartesian displacement
Appendix
Table 6: Observation and action interface. Privileged simulator states are excluded from VLA policy inputs.
Item
Setting
GeoMind-Base
20 variants, 48 episodes per variant, 960 episodes per method
GeoMind-Aug
34 augmented variants, 48 episodes per variant, 1632 episodes per method
Task instances
Shared across methods
Random seeds
Shared across methods
Episode horizon
1000 control steps
Policy control frequency
24 Hz
Appendix
Table 7: Evaluation protocol for GeoMind-Base and GeoMind-Aug .
Model
Fine-tuning
Checkpoint
sm
Rotation / gripper
OpenVLA
No
Fixed
2.0
Rotation enabled; binary gripper
π0
No
Fixed
0.5
Rotation disabled; fixed gripper
π0.5
No
Fixed
0.5
Rotation disabled; fixed gripper
SpatialVLA
No
Fixed
5.0
Rotation enabled; binary gripper
InternVLA-M1
No
Fixed
0.5
Rotation enabled; binary gripper
Spatial-Forcing
No
Fixed
0.2
Rotation enabled; binary gripper
Appendix
Table 8: Deployment settings of VLA baselines. sm denotes the normalized action gain used before controller-level command limiting. All models are evaluated with fixed checkpoints and without GeoMind -specific fine-tuning.
Information
VLA policies
Modular baselines
Language instruction
Yes
Yes
RGB observations
Yes
Yes
Robot proprioception
Yes
Yes
Object poses
No
No
Object identities
No
No
Segmentation masks
No
No
Appendix
Table 9: Information access for end-to-end VLA policies and modular baselines. Both method families are evaluated without privileged simulator information during inference. Modular baselines use explicit VLM-based grounding and scripted execution, so their results are interpreted as cross-paradigm comparisons with additional diagnostic value.
Figure 6: Example layouts generated by the spatial structure modeling and construction pipeline.
Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world. However, existing benchmarks predominantly rely on passive evaluation (e.g., static VQA) or simulator-specific pipelines, failing to assess general interactive spatial understanding. We introduce SpatialWorld, a unified benchmark designed specifically for evaluating the interactive spatial understanding of multimodal agents in complex real-world tasks. Integrating eight heterogeneous simulation backends under a shared, simulator-agnostic protocol, SpatialWorld features 760 human-annotated tasks across diverse domains (e.g., household routines, travel, social collaboration). Agents must solve tasks under vision-only partial observability, actively gathering egocentric visual evidence and expressing decisions via a unified, text-based action interface native to MLLMs. For reliable evaluation, each task includes a human-validated initial state, a reference trajectory, and a terminal-state verifier. Evaluating 15 advanced agents reveals that robust spatial task solving remains challenging: the strongest model, GPT-5, achieves an average task success rate (TSR) of only 17.4%, while the leading open-source model, Qwen-3.5, reaches 14.1%. Further analysis exposes a clear mismatch between task success and execution efficiency, alongside substantial domain-specific performance variations. These bottlenecks in active exploration and long-horizon planning position SpatialWorld as a rigorous testbed for future spatial agents.
Hongcheng Gao, Hailong Qu, Jingyi Tang +18
Tsinghua University · Chongqing University · Peking University +6
Robot tasks require grounding spatial language in physical scenes, but task success alone cannot establish whether spatial instructions are followed. We present BEHAVIOR-Spatial, a benchmark built on BEHAVIOR-1K that measures spatial question answering, point-based target grounding, and robot action to examine how VLM spatial understanding relates to VLA behavior. It comprises 3,256 spatial questions across 49 activities, 1,240 grounding queries across 14 manipulation tasks, and action evaluations covering those 14 tasks and 10 navigation tasks. Our manipulation protocol evaluates held-out configurations and instruction following when the requested target changes within a fixed training layout. The baseline achieves 41.6% on held-out configurations but only 16.1% under changed instructions. VLM adaptation and robot-data pretraining provide limited improvements, whereas oracle visual target cues raise these rates to 60.3% and 38.0%, respectively. Perception evaluations reveal policy-dependent correlations between VLM question-answering and grounding accuracy and VLA manipulation performance. Together, these findings motivate evaluating spatial instruction following alongside task completion and examining when spatial perception performance translates into robot behavior. Project website: https://arielchen07.github.io/behavior-spatial/.
When should a robot look at the cup, and when at the portafilter? A view that is informative for one task can be useless for another, even at the same workstation. We introduce Seeing What Execution Needs, a benchmark for task-conditioned actionable spatial belief in simulated tabletop manipulation. Its aligned three-domain track contains 1,080 frozen cases (360 each for coffee workstation, drawer/cabinet, and microwave/appliance). The primary coffee protocols -- closed-loop actionability and same-object functional out-of-distribution generalization -- produce over 115,000 evaluator episodes across evaluated methods, budgets, and inference repeats. Each case pairs candidate RGB views with a manipulation command, shared scripted skills, and hidden postconditions, directly measuring whether selected observations support execution. A metadata-assisted diagnostic reference pipeline maintains spatial-functional beliefs and ranks views by missing evidence. At primary budget settings, it reaches 98.8% closed-loop and 98.9% functional-OOD success; removing functional fields or command binding reduces functional-OOD success to 0.0% and 19.7%, respectively. Under a separate full-preview contract, seven vision-language models reach 26-43%. Replacing coverage metadata with frozen RGB-derived scores retains 98.5%/98.9% success; shuffling these scores reduces success to 15%/11%, supporting dependence on image content. Separately, the aligned three-domain evaluation at H=3 supports portability of the benchmark interface under frozen domain-specific schemas. These results suggest that spatial belief for manipulation should be evaluated by whether acquired evidence meets the current command's execution requirements.
Yina Jian, Mingkun Xu
Columbia University · Stanford University · Guangdong Institute of Intelligence Science and Technology, Zhuhai, China