Spatial intelligence requires agents to actively acquire visual evidence and integrate observations into a coherent representation of the three-dimensional world. Yet spatial question answering offers limited evidence of whether models can construct accurate metric maps through autonomous exploration. We introduce SpatialScout, a benchmark for evaluating vision-language model (VLM) agents on active object-level 3D mapping. SpatialScout comprises 80 diverse indoor scenes containing 3,140 target objects. Given an object inventory, RGB observations, camera calibration, and camera poses, agents control a camera to explore each scene and estimate object centers in a shared coordinate frame. The task requires coordinating viewpoint selection, cross-view association, geometric estimation, and map revision under bounded interaction budgets. Evaluation jointly measures localization accuracy and target coverage through object-scale overlap scores that penalize missing predictions. Experiments with frontier VLMs reveal substantial limitations: the strongest evaluated model, Claude Opus 5.5, achieves a mean overlap score of 41.71%, with 47.23% of targets exceeding an intersection-over-union (IoU) threshold of 0.5 and only 3.25% exceeding 0.9. These results reveal a substantial gap between recovering coarse object layouts and constructing complete, precise metric maps. SpatialScout provides a quantitative testbed for developing agents that build reliable 3D spatial representations through active perception. Code is available at https://github.com/JesseZZZZZ/SpatialScout.
Figures & tables
Figure 1: Overview of SpatialScout. A VLM agent observes, explores, and localizes the position of objects in a 3D environment.
Benchmark
Output format
Primary evaluation target
Active
Multi-round
Explicit spatial
Direct 3D
interaction
env. feedback
construction
coordinate scoring
VSR [ 31 ]
T/F
Answer correctness
×
×
×
×
BLINK [ 13 ]
MC
Answer correctness
×
×
×
×
VSI-Bench [ 55 ]
MC, NA
Answer correctness
×
×
×
×
OpenEQA [ 34 ]
OE
Answer correctness
×
×
×
×
E3VS-Bench [ 43 ]
OE
Answer correctness
✓
✓
×
×
Table 1: Comparison with spatial intelligence and scene understanding benchmarks. T/F: true/false; MC: multiple choice; NA: numerical answer; OE: open-ended.
Tool
Effect or output
list_entities
List targets and metadata.
get_camera_pose
Return the camera pose.
get_camera_intrinsics
Return camera calibration.
set_camera_position
Set the camera position.
move_camera
Translate the camera.
rotate_camera
Rotate the camera.
Table 2: Shared agent interface. No tool reveals ground-truth target positions or evaluation scores.
Figure 2: Wordcloud of the diverse objects in SpatialScout.
Figure 3: Evaluation of VLM agents on SpatialScout. A VLM agent navigates through the 3D space by controlling a camera, and applies observation and mapping tools to finally submit the positions of identified objects.
Model
Effort
All targets
Submitted targets
Mean IoU
>0%
>50%
>90%
Mean IoU
>0%
>50%
>90%
Claude Opus 5.5
xhigh
41.71
77.36
47.23
3.25
46.69
86.60
52.87
3.64
Claude Sonnet 5.5
xhigh
32.26
67.13
33.60
1.40
38.99
81.14
40.61
1.69
GPT-6 Astra
xhigh
33.16
67.79
35.15
1.62
41.16
83.86
42.69
2.09
GPT-6 Astra
medium
31.18
67.73
31.55
1.27
37.16
80.72
37.60
1.52
GPT-6 Sol
xhigh
20.83
51.85
18.95
0.48
28.65
71.31
26.06
0.66
Table 3: Object-micro localization results on 80 scenes. All values are percentages, and threshold columns use strict IoU comparisons. The >k% columns measure the percentage of box overlaps that are greater than k%.
Figure 4: Examples of accurate, incomplete, and mislocalized object-center maps. The red arrows indicate a missing or mislocalized object in the reconstructed scene.
Figure 5: Tool-use patterns across interaction steps for four models. Claude Opus 5.5 makes the most tool calls, whereas GPT-6 Astra achieves competitive mapping performance with fewer calls.
Figure 6: Illustration of tool-call frequency of multiple models.
Outcome
Factor
Total
Success
S ufficient multi-view grounding
6
I ncremental submission with continued coverage expansion
6
C ontinued refinement
6
Failure
I ncomplete object coverage
26
S uspected correspondence errors despite low triangulation residuals
21
U nresolved inconsistencies in geometric measurements
13
Table 4: Success factors and failure patterns across 40 trajectories. Each trajectory may exhibit multiple factors (non-exclusive).
Metric
Astra
DS-V4.1
S ubmitted objects with prior successful triangulation
99.33%
63.48%
T riangulation with geometric warnings
2.69%
38.65%
M ean image captures per task
32.41
39.63
M edian object center error (cm)
7.08
68.17
Table 5: Geometric grounding on 80 shared tasks. Both models use the xhigh setting. Triangulation support is measured over submitted objects, warnings over successful triangulation calls, and median center error over submitted objects.
Figure 7: All-target mean IoU vs. average inference cost per task.
Figure 8: Visualization of scene distribution in SpatialScout
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Visualization of object distribution in SpatialScout
Figure 10: The user prompt provided to the visual mapping agent for the default evaluation with B=V=100 .
Figure 11: Tool descriptions and summarized parameter schemas provided to the visual mapping agent.