GeoMind: A Benchmark for Spatial Understanding in Robotic Manipulation
Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University · Southern University of Science and Technology
Abstract
Robotic manipulation has advanced rapidly in mapping visual observations and language instructions to actions. However, reliable manipulation in open-world environments requires robust spatial understanding: the ability to identify targets in complex layouts (spatial reasoning) and adapt to scenes that continuously evolve through interaction (spatial memory). Yet existing embodied benchmarks fall short in evaluating spatial understanding: they often introduce strong visual cues that bypass spatial reasoning, and focus on static scenes that offer little insight into an agent's capacity for spatial memory. To address this gap, we introduce GeoMind, a systematic benchmark for structured spatial understanding through tabletop manipulation. GeoMind comprises 54 evaluated task variants organized around spatial reasoning and spatial memory. It also incorporates a scalable pipeline that automates the generation and annotation of task instances with verified targets and executable interactions. Extensive evaluations across diverse embodied agents reveal their consistent weaknesses in spatial understanding. Our findings highlight that these agents struggle to track identities after spatial changes and convert relational or geometric constraints into executable actions. We release the GeoMind benchmark at https://github.com/Notfound-JH/GEOMIND-benchmark.git.
Figures & tables
| Benchmark | Design | Spatial Memory | Spatial Reasoning | Evaluation | ||||
| Structure Generation | Scene Updating | Identity Tracking | Target Identification | Relational Grounding | Constraint Composition | Geometric Inference | Diagnostic Metrics | |
| VIMA [ 23 ] | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| GENMANIP [ 30 ] | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| RoboMME [ 32 ] | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ |
| MEMOBench [ 33 ] | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ |
| RoboSPA [ 34 ] | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| Embodied Agents | Spatial Reasoning | Spatial Memory | AVG | ||||||
| Target Iden. | Relational Grounding | Constraint Comp. | Geometric Inference | Scene Updating | Identity Tracking | ||||
| Oracle | 100.0 | 98.0 | 100.0 | 100.0 | 98.0 | 100.0 | 99.3 | ||
| General Policies | OpenVLA | 37.5 | 0.0 | 6.2 | 5.0 | 41.7 | 37.5 | 20.7 | |
| 4.2 | 0.0 | 0.0 | 25.0 | 6.3 | 20.9 | 11.3 | |||
| 4.2 | 0.0 | 0.0 | 27.5 | 6.3 | 25.0 | 12.5 | |||
| Spatially Enhanced Policies | SpatialVLA | 12.5 | 0.0 | 0.0 | 17.5 | 45.8 | 8.3 | 18.3 | |
| Policies | Corresponding Base | Semantic | Visual | Execution |
| OpenVLA | 25.0 | 16.6 ( ) | 12.5 ( ) | 12.5 ( ) |
| 16.7 | 12.5 ( ) | 16.7 ( ) | 10.2 ( ) | |
| 18.5 | 12.5 ( ) | 16.7 ( ) | 10.2 ( ) | |
| SpatialVLA | 18.8 | 8.3 ( ) | 12.5 ( ) | 12.5 ( ) |
| InternVLA-M1 | 37.5 | 27.1 ( ) | 37.5 ( ) | 35.4 ( ) |
| Spatial-Forcing | 18.8 | 8.3 ( ) | 12.5 ( ) | 16.7 ( ) |
| Capability Dimensions | Task Family | Failure Rate | SGE (m) | TTE (m) |
| Spatial Memory | Scene Updating | 68.89% | 0.189 | 0.330 |
| Identity Tracking | 85.00% | 0.192 | 0.486 | |
| Spatial Reasoning | Target Identification | 72.08% | 0.116 | 0.811 |
| Relational Grounding | 98.33% | 0.180 | 0.672 | |
| Constraint Composition | 87.50% | 0.142 | 0.627 | |
| Geometric Inference | 86.67% | 0.158 | 0.727 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Setting |
| Simulator | Isaac Sim / IsaacLab |
| Benchmark package | GeoMind Lab |
| Robot | Franka Research 3 (FR3) |
| Gripper | FR3 parallel-jaw gripper |
| Low-level controller | Differential inverse kinematics |
| Physics frequency | Hz |
| Type | Field | Description |
| Policy input | Instruction | Natural-language task instruction |
| Table-view RGB | Static external camera observation, | |
| Wrist RGB | Wrist-mounted camera observation, | |
| Joint state | Robot joint positions and velocities | |
| End-effector / gripper state | Current end-effector pose and gripper state | |
| Policy action | Translation | Relative Cartesian displacement |
| Item | Setting |
| GeoMind-Base | variants, episodes per variant, episodes per method |
| GeoMind-Aug | augmented variants, episodes per variant, episodes per method |
| Task instances | Shared across methods |
| Random seeds | Shared across methods |
| Episode horizon | control steps |
| Policy control frequency | Hz |
| Model | Fine-tuning | Checkpoint | Rotation / gripper | |
| OpenVLA | No | Fixed | Rotation enabled; binary gripper | |
| No | Fixed | Rotation disabled; fixed gripper | ||
| No | Fixed | Rotation disabled; fixed gripper | ||
| SpatialVLA | No | Fixed | Rotation enabled; binary gripper | |
| InternVLA-M1 | No | Fixed | Rotation enabled; binary gripper | |
| Spatial-Forcing | No | Fixed | Rotation enabled; binary gripper |
| Information | VLA policies | Modular baselines |
| Language instruction | Yes | Yes |
| RGB observations | Yes | Yes |
| Robot proprioception | Yes | Yes |
| Object poses | No | No |
| Object identities | No | No |
| Segmentation masks | No | No |
Spatial Reasoning Benchmarks
Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.
See all 176 papers