Spatial understanding is a necessary step in moving multimodal large language models (MLLMs) toward reasoning about the 3D physical world. Their visual pretraining data, however, is almost entirely 2D, so the visual tokens processed by MLLMs often lack inherent 3D structural information. A growing number of methods inject such structure by transferring representations from external vision or geometry models, and collectively establish that such transfer helps. Behind these methods lies a largely underexplored question: what makes a visual representation good for spatial understanding? To answer it, we first separate what is transferred from how it is transferred, and then trace the resulting differences back to properties of the representations themselves. What we transfer comes from 3D geometry models, self-supervised image and video models, vision-language models, and generative models, in the form of latent features or explicit depth and point maps. We transfer it either by distillation, using it as a supervision target, or by fusion, feeding it as an additional input. We find that a representation's 3D structure correlates strongly with how much it helps, measured by cross-view consistency for latent features and by depth and reconstruction accuracy for explicit geometry. We further show that the form of a representation dictates the mechanism it needs. Latent features gain as much from fusion as from distillation; explicit geometry gains little from fusion, and most of its distillation gain depends on a geometry decoding head. Guided by these findings, we distill both latent features and explicit geometry into a single MLLM. Taking only RGB video as input, it consistently improves over its base model on both static 3D and dynamic 4D spatial reasoning benchmarks.
Figures & tables
Fig. 2: Overview of the study. We transfer a representation from a pretrained model—one of five families—into an MLLM whose base model is fixed across all variants. The representation takes one of two forms, backbone features or explicit geometric outputs, and enters the MLLM by one of two mechanisms, distillation or fusion. The MLLM itself takes only RGB video and is evaluated on spatial understanding.
Fig. 3: Measuring 3D-aware cross-view consistency. Per-pixel features from different views are binned into a shared 3D voxel grid. The score Sˉ is how far genuine correspondences stand above mismatched pairs in mean cosine similarity, so a representation that fails to distinguish locations scores near zero.
Fig. 4: Backbone features: 3D-aware consistency tracks the gain. Across models, the cross-view consistency score correlates strongly with the gain from distilling the feature (Pearson r≈0.83 ), and the trend holds across voxel resolutions.
Fig. 5: 2D feature-structure metrics do not track the gain. Unlike 3D-aware cross-view consistency (Fig. 4 ), the three 2D metrics (LDS/CDS/RMSC) are at most weakly correlated with the gain ( r≈0.26,0.22,−0.11 ).
Fig. 6: Explicit geometry: depth and reconstruction quality track the gain. Each panel scores the three geometry models’ own predictions on a standard depth or reconstruction benchmark against the gain from distilling them. The three fall in the same order on all four.
Fig. 7: Backbone features and geometric outputs specialize by task. Using distillation, we compare backbone features against geometric outputs across tasks, averaged over three geometry models (VGGT/DA3/ π3 ). (a) Mean relative improvement over the base model by VSI-Bench task category. (b) The two forms’ per-task gains are strongly correlated (Pearson r=0.88 ): single-object tasks favor geometric outputs (above the diagonal), relational/temporal tasks favor backbone features (below).
Fig. 8: Backbone-feature transfer: distillation vs. fusion. Bars are relative improvement over the base model, per model. The two mechanisms yield broadly similar gains for backbone features, with no consistent winner.
Fig. 9: Explicit-geometry transfer and its quality. (a) Distillation far outperforms fusion for geometric outputs ( ≈ +4.7% vs. +0.6%). (b) Without the pretrained DPT-style decoding head, the gain falls to ≈ +1.4%. (c) Sorting the 288 VSI-Bench evaluation scenes ( 150 from ARKitScenes, 88 from ScanNet, 50 from ScanNet++) by depth+point reconstruction error into quartiles, QA accuracy falls from Q1 (64.0%) to Q4 (53.9%).
Table 9
Method
Avg.
Count
Abs. Dist
Obj. Size
Room Size
Rel. Dist
Rel. Dir
Route
Appr. Order
Proprietary
GPT-4o [ 47 ]
34.0
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
Gemini-1.5 Flash
42.1
49.8
30.8
53.5
54.4
37.7
41.0
31.5
37.8
Gemini-1.5 Pro
45.4
56.2
30.9
64.1
43.6
51.3
46.3
36.0
34.6
Gemini-2.5 Pro †
53.6
46.0
37.4
68.7
54.4
62.0
43.9
47.4
68.8
GPT-5 †
55.0
53.3
34.5
73.3
47.5
63.7
48.7
50.3
68.9
TABLE III: VSI-Bench per-category accuracy (%). All methods use 2D video input. Categories are object counting (Count), absolute distance (Abs.Dist), object size (Obj.Size), room size (Room), relative distance (Rel.Dist), relative direction (Rel.Dir), route planning (Route), and appearance order (Appr.Order). Best among reported entries in bold . † Numbers are reported by GUIDE [ 60 ] ; other baseline category scores are from the VSI-Bench [ 73 ] , VLM-3R [ 20 ] , and 3DRS [ 33 ] papers, or the methods’ own papers.
Method
Absolute Viewpoint
Relative Viewpoint
Non- Temp.
Avg.
Dis.
Dir.
Ori.
Spd.
Spd. Comp.
Dir. Pred.
Dis.
Dir.
Ori.
Spd.
Spd. Comp.
Dir. Pred.
Proprietary Models
GPT-4o [ 47 ]
18.8
29.2
26.8
29.7
21.5
26.2
24.1
23.8
17.2
32.3
22.8
24.4
34.7
26.4
GPT-5
21.1
41.5
48.7
34.5
33.3
34.7
17.2
44.3
41.9
21.2
25.0
30.9
26.7
30.8
Gemini-2.5-Flash
18.8
27.6
19.5
25.0
23.6
22.0
11.2
28.4
30.8
23.2
17.1
22.6
38.8
24.9
Gemini-2.5-Pro
20.0
44.6
53.6
27.3
38.7
30.5
23.2
32.9
43.2
17.1
28.5
27.9
34.3
31.7
TABLE IV: Dynamic 4D spatial reasoning on DSR-Bench. Baselines are from the DSR-Suite paper. Following that setting, our entry is further fine-tuned on the DSR-Bench training split; this is the only benchmark-specific fine-tuning setting in our evaluation. Avg. is sample-weighted accuracy (%). Best in bold .
Method
Input
C
EM
SQA3D
ScanQA [ 3 ]
3D
64.9
21.1
47.2
3D-LLM (FlanT5) [ 28 ]
3D
69.4
20.5
–
3D-VisTA [ 93 ]
3D
69.6
22.4
48.5
Grounded 3D-LLM [ 14 ]
3D
75.9
–
–
LL3DA [ 13 ]
3D
76.8
–
–
Scene-LLM [ 23 ]
3D
80.0
27.2
54.2
TABLE V: Indoor 3D scene question answering on ScanQA val and SQA3D test. ScanQA reports CIDEr (C) and exact match (EM); SQA3D reports accuracy (%). Input indicates explicit 3D/posed RGB-D, reconstructed BEV from 3D geometry, or RGB video. Best per column in bold ; “–” denotes not reported.
Method
SPAR
View- Spatial
CV-Bench
SPBench
Avg
Overall
2D
3D
SI
MV
LLaVA-OneVision-7B [ 37 ]
31.2
27.5
53.2
63.5
33.2
32.2
Llama-3.2-Vision-11B [ 38 ]
–
28.8
–
–
–
–
LLaVA-NeXT-Video-7B [ 38 ]
–
30.6
–
–
–
–
GPT-4o [ 47 ]
38.1
35.0
69.4
81.3
42.4
48.2
Claude-3.7-Sonnet [ 82 ]
21.8
–
–
–
–
–
TABLE VI: Image and multi-view spatial reasoning (2D input). SPAR/ViewSpatial: overall accuracy; CV-Bench: 2D/3D splits; SPBench: single-image (SI)/multi-view (MV) splits (all %). SPAR uses the full benchmark where reported; Claude-3.7-Sonnet is the tiny/API result reported by SPAR because the full result is unavailable. Best per column in bold ; “–” not reported.
Fig. 10: Qualitative spatial QA examples. Each panel shows input views, the question, ground truth, and answers from the base model and our model. The examples show cases where the base model’s answer is inconsistent with the depicted scene layout, whereas our transferred-representation model recovers the annotated answer under the same RGB input.
Fig. 11: Qualitative depth estimation. Each triplet: input image, VGGT [ 62 ] depth, and our DPT-decoded depth. Our output has the same geometric form as VGGT’s depth prediction, but the model is trained primarily to improve spatial understanding rather than to serve as a standalone geometry predictor; the decoded depth is therefore smoother, while still preserving plausible depth ordering.
Fig. 12: Qualitative 3D reconstruction (point maps) . Layout as in Fig. 11 : input image, VGGT [ 62 ] reconstruction, and our DPT-decoded point map. The point map is analogous to VGGT’s geometric output but is produced by a spatial-understanding model rather than a dedicated reconstruction model, so it is coarser while still recovering coherent room-level structure.