cs.CVNov 19, 2026

SpatialScout-V2: Can VLM Agents Actively Explore Real-World 3D Scenes and Localize Objects?

Authors: Zhaoxi Zhang, Hao Liang, Linxuan Wu, Yiming Xu, Yifu Yuan, Kun Liang, Chenming Tang, Meiyi Qiang, 

Organizations: Peking University · Tianjin University

Abstract

Spatial intelligence requires agents to acquire visual evidence and integrate partial observations into coherent three-dimensional representations. We introduce SpatialScout-V2, which extends SpatialScout's active mapping task from simulation to real residential scenes through three advances: real visual observations, preservation of original scene content, and interactive exploration of recorded views. Built from 100 homes in Realsee3D, the benchmark covers 3,171 target objects across 29 categories. Given a fixed target inventory, agents autonomously select recorded viewpoints and viewing directions, integrate calibrated RGB observations, and iteratively estimate and revise object centers within a fixed interaction budget. We evaluate the localization accuracy and completeness of object-center maps using mean IoU, IoU-threshold hit rates, and target coverage. Evaluating seven frontier vision-language models (VLMs), we find that six achieve an all-target mean IoU below 20%. Even the best-performing model, Claude Opus 5.5, reaches only 22.52%, with just 18.36% of targets exceeding an IoU of 0.5. Further analysis shows that broad submission coverage can coexist with limited localization accuracy, while target omissions further reduce overall scores. These results highlight substantial challenges in constructing accurate and complete object-center maps through active exploration of real scenes.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SpatialScout: Can VLM Agents Actively Explore and Map 3D Scenes?

    Oct 19, 2026Zhaoxi Zhang, Hao Liang, Linxuan Wu +73D MappingActive Perception

  2. SceneActBench: Can Agents Act on the 3D Scenes They See?

    Jul 24, 2026Yifei Zhao, Xiangxin Zhou, Wenhao Yang +11AI Agent BenchmarksTool-Using Vision-Language Agents

  3. Humans in Space: Can MLLMs Construct Human-Centric Spatial Understanding via Active Exocentric Exploration?

    Oct 25, 2026Tianchen Guo, Tengyue Zhang, Chen Liu +2Spatial Reasoning BenchmarksMultimodal Spatial Reasoning