Spatial agents have access to reconstruction and segmentation tools, yet inconsistent predictions and fragmented object identities can prevent their outputs from becoming reliable 3D scene information. A simple oracle experiment exposes this perception bottleneck: replacing tool-predicted perception with ground-truth 3D bounding boxes raises an agent to nearly full score across five object-centric VSI-Bench tasks, while retaining the same reasoning tools. Motivated by this finding, we believe the devil of spatial reasoning lies in 3D itself, and introduce DeSpatial(Devil of Spatial), which explicitly decomposes spatial understanding into perception and reasoning. DeSpatial has a shared 3D perception tool to reconcile independent perception outputs into a unified, metric, gravity-aligned 3D scene. This perception tool has two core components: a) a joint segmentation--depth co-refinement module that consolidates masks of the same object and removes projection artifacts caused by inconsistent segmentation and depth within a single frame; b) a cross-view instance fusion module that consolidate voxel-overlapping instances but not allowed to merge same-frame instances. This fusion module additionally fuses spatially disjoint fragments caused by insufficient multi-view observations, by checking their 3D structure and source mask appearance. After receiving the complete and unified 3D scene information, the agent chooses reasoning or computational tools to solve the spatial problem. DeSpatial achieves leading performance on VSI-Bench, MindCube-Tiny, and ViewSpatial, with additional long-video results on VSI-SUPER, revealing the bottleneck of spatial understanding lies in a precise 3D perception itself.
Figures & tables
Figure 1: Overview of DeSpatial. The VLM receives the video and question and calls our shared perception tool. Joint segmentation–depth co-refinement cleans per-frame observations; cross-view instance fusion associates them into distinguished individual objects. Colors identify individual masks before fusion and persistent instances afterward. Gravity estimation establishes a common vertical direction. The VLM then reasons and answers using reasoning and computational tools.
3D Tool
Obj. Count
Abs. Dist.
Obj. Size
Rel. Dist.
Rel. Dir.
Avg.
SAM3 + Pi3 (raw)
42.70
32.54
49.68
47.62
79.37
50.38
GT 3D Boxes
98.76
96.73
99.37
98.59
97.70
98.23
Table 1: Oracle and raw-perception results on the same 4,030 questions across five VSI-Bench categories.
Figure 2: Joint segmentation–depth co-refinement. (a) Same-class mask consolidation unifies highly overlapping masks and visually verified parts of an occluded object. Negative or uncertain identity judgments keep masks separate. (b) Confidence, depth-edge, and 3D outlier filters remove unreliable geometry. Component selection then removes residual detached points that distort the bounding box, retaining the top- k components for k confirmed object parts, usually one.
Figure 3: Cross-view instance fusion. (a) Voxel overlap links partial observations, while same-frame exclusions prevent distinct objects from merging, including through indirect connections. (b) Visual fragment association handles observations without shared voxels. Multiple source views and a common 3D view provide complementary appearance and structural evidence to an identity judge. A positive decision combines the observed fragments and their provenance.
Numerical (MRA)
Multiple choice (accuracy)
Method
Backbone
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
Avg.
Trained models
Cambrian-S
Qwen2.5-7B
73.2
50.5
74.9
72.2
71.1
76.2
41.8
80.1
67.5
VLM-3R
LLaVA-NV-7B
70.2
49.4
69.2
67.1
65.4
80.5
45.4
40.1
60.9
CaMo-3B
Qwen2.5-VL-3B
64.8
18.6
59.2
43.2
45.2
45.3
25.8
49.9
44.0
Training-free inference pipelines ( Li et al., 2025c ; Ma et al., 2026 )
Table 2: VSI-Bench results (%): MRA for numerical questions and accuracy for multiple choice. Avg. averages eight categories (four multiple-choice categories for Think3D). Unmarked methods use the full benchmark. ∗ Subset evaluations from the original papers.
Method
Among
Around
Rotation
Avg.
Qwen2.5-VL-3B
36.00
45.20
34.00
37.81
GPT-4o
43.00
46.40
35.00
42.29
Gemini-2.5-Pro
43.80
67.20
88.50
57.90
Gemini-3-Pro
68.20
77.20
93.00
75.10
VADAR GPT-4o
40.67
46.80
33.50
40.76
pySpatial GPT-4o
66.33
71.20
41.00
62.67
Table 3: Results on MindCube-Tiny and ViewSpatial. Scores are accuracy percentages; Avg. denotes overall accuracy. Model references are listed in Appendix C.1 .
Recall (VSR): accuracy
Count (VSC): MRA
Method
10
30
60
120
240
10
30
60
120
MovieChat
18.3
21.7
16.7
26.7
25.6
0.0
0.0
0.0
0.0
Flash-VStream
28.3
33.3
23.3
28.3
31.7
0.0
0.0
0.0
0.0
Cambrian-S-7B
38.3
35.0
6.0
0.0
0.0
0.0
0.0
0.0
0.0
Cambrian-S-7B-LFP
45.0
41.7
40.0
40.0
40.0
40.6
42.0
35.0
34.0
S-Agent
75.0
55.0
63.3
66.1
77.2
10.6
4.2
0.0
0.0
Table 4: Results on VSI-SUPER. Recall (VSR) uses accuracy and Count (VSC) uses mean relative accuracy (MRA), both in percent; durations are in minutes. Model references and result sources are listed in Appendix C.2 .
Configuration
Avg (%)
Raw mask lifting
57.56
+ Geometry filtering
62.90
+ Largest component ( k=1 )
65.22
+ Same-class mask consolidation
66.45
+ Part-aware top- k selection
68.83
Table 5: Ablations on a fixed stratified VSI-Bench subset (504 questions, 63 per category); Avg. averages eight categories. (a) Full cross-view fusion is fixed. (b) Full co-refinement is fixed; Exclusion: same-frame exclusion; FragAssoc.: visual fragment association. All runs share the agent settings. IoU/voxel thresholds: 0.3/0.1; voxel size: 0.1 m.
Gravity estimator
Correct / Total
Correctness (%)
RANSAC plane-based
107 / 156
68.59
GeoCalib joint calibration
156 / 156
100.00
Table 6: Human assessment of projected up directions on the same 156 scenes.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Human annotation interface for gravity estimation. Both panels show the same scene and eight source views. Green arrows visualize the estimated up direction, and the highlighted button indicates the saved scene-level judgment.
Development run
Easy
Medium
Hard
Rel. Dir.
Correct / Total
RANSAC-based (Jul. 11)
63.70
48.72
56.10
56.17
158 / 267
GeoCalib-based (Jul. 20)
81.51
66.67
67.07
71.75
200 / 267
Appendix
Table 7: Historical relative-direction scores (%) on identical questions. Easy, Medium, and Hard contain 146, 39, and 82 questions, respectively; Rel. Dir. is their unweighted mean. This development comparison does not isolate gravity as the sole intervention.
3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.