SpatialScout-V2: Can VLM Agents Actively Explore Real-World 3D Scenes and Localize Objects?
Organizations: Peking University · Tianjin University
Abstract
Spatial intelligence requires agents to acquire visual evidence and integrate partial observations into coherent three-dimensional representations. We introduce SpatialScout-V2, which extends SpatialScout's active mapping task from simulation to real residential scenes through three advances: real visual observations, preservation of original scene content, and interactive exploration of recorded views. Built from 100 homes in Realsee3D, the benchmark covers 3,171 target objects across 29 categories. Given a fixed target inventory, agents autonomously select recorded viewpoints and viewing directions, integrate calibrated RGB observations, and iteratively estimate and revise object centers within a fixed interaction budget. We evaluate the localization accuracy and completeness of object-center maps using mean IoU, IoU-threshold hit rates, and target coverage. Evaluating seven frontier vision-language models (VLMs), we find that six achieve an all-target mean IoU below 20%. Even the best-performing model, Claude Opus 5.5, reaches only 22.52%, with just 18.36% of targets exceeding an IoU of 0.5. Further analysis shows that broad submission coverage can coexist with limited localization accuracy, while target omissions further reduce overall scores. These results highlight substantial challenges in constructing accurate and complete object-center maps through active exploration of real scenes.
Figures & tables
| Model | Effort | All targets | Submitted targets | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Mean IoU | Mean IoU | ||||||||
| Claude Opus 5.5 | xhigh | ||||||||
| Claude Sonnet 5.5 | xhigh | ||||||||
| GPT-6 Astra | xhigh | ||||||||
| GPT-6.1 Sol | xhigh | ||||||||
| MiMo-V2.6-Pro | xhigh | ||||||||
| Outcome | Factor | Total |
|---|---|---|
| Success | S ufficient multi-view grounding | 6 |
| I ncremental submission with continued coverage expansion | 0 | |
| C ontinued refinement | 4 | |
| Failure | I ncomplete object coverage | 6 |
| S uspected correspondence errors despite low triangulation residuals | 14 | |
| U nresolved inconsistencies in geometric measurements | 21 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Active Visual Perception
Active visual perception studies how agents choose where, when, and how to look to gather task-relevant evidence efficiently.
See all 54 papers