What Makes a Good Visual Representation for Spatial Understanding in MLLMs?

Xiaohu Huang; Yumin Guo; Kai Han

cs.CVOct 13, 2026

What Makes a Good Visual Representation for Spatial Understanding in MLLMs?

Authors: Xiaohu Huang, Yumin Guo, Kai Han

Organizations: The University of Hong Kong

Abstract

Spatial understanding is a necessary step in moving multimodal large language models (MLLMs) toward reasoning about the 3D physical world. Their visual pretraining data, however, is almost entirely 2D, so the visual tokens processed by MLLMs often lack inherent 3D structural information. A growing number of methods inject such structure by transferring representations from external vision or geometry models, and collectively establish that such transfer helps. Behind these methods lies a largely underexplored question: what makes a visual representation good for spatial understanding? To answer it, we first separate what is transferred from how it is transferred, and then trace the resulting differences back to properties of the representations themselves. What we transfer comes from 3D geometry models, self-supervised image and video models, vision-language models, and generative models, in the form of latent features or explicit depth and point maps. We transfer it either by distillation, using it as a supervision target, or by fusion, feeding it as an additional input. We find that a representation's 3D structure correlates strongly with how much it helps, measured by cross-view consistency for latent features and by depth and reconstruction accuracy for explicit geometry. We further show that the form of a representation dictates the mechanism it needs. Latent features gain as much from fusion as from distillation; explicit geometry gains little from fusion, and most of its distillation gain depends on a geometry decoding head. Guided by these findings, we distill both latent features and explicit geometry into a single MLLM. Taking only RGB video as input, it consistently improves over its base model on both static 3D and dynamic 4D spatial reasoning benchmarks.

Figures & tables

Fig. 2: Overview of the study. We transfer a representation from a pretrained model—one of five families—into an MLLM whose base model is fixed across all variants. The representation takes one of two forms, backbone features or explicit geometric outputs, and enters the MLLM by one of two mechanisms, distillation or fusion. The MLLM itself takes only RGB video and is evaluated on spatial understanding.

Fig. 3: Measuring 3D-aware cross-view consistency. Per-pixel features from different views are binned into a shared 3D voxel grid. The score $\bar{S}$ is how far genuine correspondences stand above mismatched pairs in mean cosine similarity, so a representation that fails to distinguish locations scores near zero.

Fig. 4: Backbone features: 3D-aware consistency tracks the gain. Across models, the cross-view consistency score correlates strongly with the gain from distilling the feature (Pearson

r\!\approx\!0.83

), and the trend holds across voxel resolutions.

Fig. 5: 2D feature-structure metrics do not track the gain. Unlike 3D-aware cross-view consistency (Fig. 4 ), the three 2D metrics (LDS/CDS/RMSC) are at most weakly correlated with the gain (

r\!\approx\!0.26,\,0.22,\,{-}0.11

Fig. 6: Explicit geometry: depth and reconstruction quality track the gain. Each panel scores the three geometry models’ own predictions on a standard depth or reconstruction benchmark against the gain from distilling them. The three fall in the same order on all four.

Fig. 7: Backbone features and geometric outputs specialize by task. Using distillation, we compare backbone features against geometric outputs across tasks, averaged over three geometry models (VGGT/DA3/

\pi^{3}

). (a) Mean relative improvement over the base model by VSI-Bench task category. (b) The two forms’ per-task gains are strongly correlated (Pearson

r=0.88

): single-object tasks favor geometric outputs (above the diagonal), relational/temporal tasks favor backbone features (below).

Fig. 8: Backbone-feature transfer: distillation vs. fusion. Bars are relative improvement over the base model, per model. The two mechanisms yield broadly similar gains for backbone features, with no consistent winner.

Fig. 9: Explicit-geometry transfer and its quality. (a) Distillation far outperforms fusion for geometric outputs (

\approx

+4.7% vs. +0.6%). (b) Without the pretrained DPT-style decoding head, the gain falls to

\approx

+1.4%. (c) Sorting the

288

VSI-Bench evaluation scenes (

150

from ARKitScenes,

88

from ScanNet,

50

from ScanNet++) by depth+point reconstruction error into quartiles, QA accuracy falls from Q1 (64.0%) to Q4 (53.9%).

Table 9

Method	Avg.	Count	Abs. Dist	Obj. Size	Room Size	Rel. Dist	Rel. Dir	Route	Appr. Order
Proprietary
GPT-4o [ 47 ]	34.0	46.2	5.3	43.8	38.2	37.0	41.3	31.5	28.5
Gemini-1.5 Flash	42.1	49.8	30.8	53.5	54.4	37.7	41.0	31.5	37.8
Gemini-1.5 Pro	45.4	56.2	30.9	64.1	43.6	51.3	46.3	36.0	34.6
Gemini-2.5 Pro †	53.6	46.0	37.4	68.7	54.4	62.0	43.9	47.4	68.8
GPT-5 †	55.0	53.3	34.5	73.3	47.5	63.7	48.7	50.3	68.9

TABLE III: VSI-Bench per-category accuracy (%). All methods use 2D video input. Categories are object counting (Count), absolute distance (Abs.Dist), object size (Obj.Size), room size (Room), relative distance (Rel.Dist), relative direction (Rel.Dir), route planning (Route), and appearance order (Appr.Order). Best among reported entries in bold . † Numbers are reported by GUIDE [ 60 ] ; other baseline category scores are from the VSI-Bench [ 73 ] , VLM-3R [ 20 ] , and 3DRS [ 33 ] papers, or the methods’ own papers.

Method	Absolute Viewpoint						Relative Viewpoint						Non- Temp.	Avg.
Method	Dis.	Dir.	Ori.	Spd.	Spd. Comp.	Dir. Pred.	Dis.	Dir.	Ori.	Spd.	Spd. Comp.	Dir. Pred.	Non- Temp.	Avg.
Proprietary Models
GPT-4o [ 47 ]	18.8	29.2	26.8	29.7	21.5	26.2	24.1	23.8	17.2	32.3	22.8	24.4	34.7	26.4
GPT-5	21.1	41.5	48.7	34.5	33.3	34.7	17.2	44.3	41.9	21.2	25.0	30.9	26.7	30.8
Gemini-2.5-Flash	18.8	27.6	19.5	25.0	23.6	22.0	11.2	28.4	30.8	23.2	17.1	22.6	38.8	24.9
Gemini-2.5-Pro	20.0	44.6	53.6	27.3	38.7	30.5	23.2	32.9	43.2	17.1	28.5	27.9	34.3	31.7

TABLE IV: Dynamic 4D spatial reasoning on DSR-Bench. Baselines are from the DSR-Suite paper. Following that setting, our entry is further fine-tuned on the DSR-Bench training split; this is the only benchmark-specific fine-tuning setting in our evaluation. Avg. is sample-weighted accuracy (%). Best in bold .

Method	Input	C	EM	SQA3D
ScanQA [ 3 ]	3D	64.9	21.1	47.2
3D-LLM (FlanT5) [ 28 ]	3D	69.4	20.5	–
3D-VisTA [ 93 ]	3D	69.6	22.4	48.5
Grounded 3D-LLM [ 14 ]	3D	75.9	–	–
LL3DA [ 13 ]	3D	76.8	–	–
Scene-LLM [ 23 ]	3D	80.0	27.2	54.2

TABLE V: Indoor 3D scene question answering on ScanQA val and SQA3D test. ScanQA reports CIDEr (C) and exact match (EM); SQA3D reports accuracy (%). Input indicates explicit 3D/posed RGB-D, reconstructed BEV from 3D geometry, or RGB video. Best per column in bold ; “–” denotes not reported.

Method	SPAR	View- Spatial	CV-Bench		SPBench
	Avg	Overall	2D	3D	SI	MV
LLaVA-OneVision-7B [ 37 ]	31.2	27.5	53.2	63.5	33.2	32.2
Llama-3.2-Vision-11B [ 38 ]	–	28.8	–	–	–	–
LLaVA-NeXT-Video-7B [ 38 ]	–	30.6	–	–	–	–
GPT-4o [ 47 ]	38.1	35.0	69.4	81.3	42.4	48.2
Claude-3.7-Sonnet [ 82 ]	21.8	–	–	–	–	–

TABLE VI: Image and multi-view spatial reasoning (2D input). SPAR/ViewSpatial: overall accuracy; CV-Bench: 2D/3D splits; SPBench: single-image (SI)/multi-view (MV) splits (all %). SPAR uses the full benchmark where reported; Claude-3.7-Sonnet is the tiny/API result reported by SPAR because the full result is unavailable. Best per column in bold ; “–” not reported.

Fig. 10: Qualitative spatial QA examples. Each panel shows input views, the question, ground truth, and answers from the base model and our model. The examples show cases where the base model’s answer is inconsistent with the depicted scene layout, whereas our transferred-representation model recovers the annotated answer under the same RGB input.

Fig. 11: Qualitative depth estimation. Each triplet: input image, VGGT [ 62 ] depth, and our DPT-decoded depth. Our output has the same geometric form as VGGT’s depth prediction, but the model is trained primarily to improve spatial understanding rather than to serve as a standalone geometry predictor; the decoded depth is therefore smoother, while still preserving plausible depth ordering.

Fig. 12: Qualitative 3D reconstruction (point maps) . Layout as in Fig. 11 : input image, VGGT [ 62 ] reconstruction, and our DPT-decoded point map. The point map is analogous to VGGT’s geometric output but is produced by a spatial-understanding model rather than a dedicated reconstruction model, so it is coarser while still recovering coherent room-level structure.

Multimodal Spatial Reasoning

See all 42 papers

Mar 9, 2026

Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations

While Multimodal Large Language Models (MLLMs) have achieved remarkable success in 2D visual understanding, their ability to reason about 3D space remains limited. To address this gap, we introduce geometrically referenced 3D scene representations (GR3D). Given a set of input images, GR3D annotates objects in the images with unique IDs and encodes their 3D geometric attributes as textual references indexed by these IDs. This representation enables MLLMs to interpret 3D cues using their advanced language-based skills in mathematical reasoning, while concurrently analyzing 2D visual features in a tightly coupled way. We present a simple yet effective approach based on GR3D, which requires no additional training and is readily applicable to different MLLMs. Implemented in a zero-shot setting, our approach yields substantial improvements on challenging spatial reasoning benchmarks, boosting GPT-5 performance by 9% on VSI-Bench and 12% on MindCube. Qualitative studies further demonstrate that GR3D empowers MLLMs to perform complex spatial reasoning with highly sparse input views.

Jiangye Yuan, Gowri Kumar, Baoyuan Wang

Zillow Group

3D Spatial Reasoning Multimodal Large Language Models 3D Scene Representation

Ask PDF

Apr 8, 2026

Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning

Although Multimodal Large Language Models have achieved remarkable progress, they still struggle with complex 3D spatial reasoning due to the reliance on 2D visual priors. Existing approaches typically mitigate this limitation either through computationally expensive post-training procedures on limited 3D datasets or through rigid tool-calling mechanisms that lack explicit geometric understanding and viewpoint flexibility. To address these challenges, we propose a training-free framework that introduces a Visual Chain-of-Thought mechanism grounded in explicit 3D reconstruction. The proposed pipeline first reconstructs a high-fidelity 3D mesh from a single image using MLLM-guided keyword extraction and mask generation at multiple granularities. Subsequently, the framework leverages an external knowledge base to iteratively compute optimal camera extrinsic parameters and synthesize novel views, thereby emulating human perspective-taking. Extensive experiments demonstrate that the proposed approach significantly enhances spatial comprehension. Specifically, the framework outperforms specialized spatial models and general-purpose MLLMs, including GPT-5.2 and Gemini-2.5-Flash, on major benchmarks such as 3DSRBench and Rel3D.

Jiahua Chen, Qihong Tang, Weinong Wang +1

Tsinghua University · Nanjing University · Tencent

3D Spatial Reasoning VLM Reasoning Visual CoT

Ask PDF

Sep 29, 2026

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering

Reasoning about the 3D world from multi-view images remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While modern MLLMs handle single-image inputs effectively, they struggle to integrate evidence across viewpoints into a coherent 3D understanding. A growing body of work attempts to close this gap by injecting 3D awareness into MLLMs, either by boosting fine-grained pixel-level cross-view correspondence or by fusing features from 3D geometry foundation models, yet a substantial gap to human reasoning persists. In this work, we revisit human spatial reasoning, which suggests that rather than relying on fine-grained geometry cues, humans roughly identify common objects across views, infer the relative geometry between viewpoints, and assemble a coarse 3D layout of the scene. Inspired by this process, we introduce Imagine3D-LLM, an MLLM that learns to assemble a similar compact 3D representation of the scene and conditions its answer on this representation. Concretely, we append a small set of learnable summary tokens after the image tokens, decode them into a compact 3D Gaussian Splatting representation supervised by a photometric reconstruction loss, and train jointly with the standard next-token prediction objective. Notably, although only the summary tokens receive direct reconstruction supervision, this objective also induces stronger cross-frame correspondence within the LLM's underlying image features, suggesting that learning to reconstruct propagates 3D-aware signals throughout the model. As a result, Imagine3D-LLM consistently outperforms prior approaches across multiple spatial reasoning and 3D understanding benchmarks, suggesting that imagining the scene can be more effective than being told its pixel-wise geometry.

Jaewoo Jung, Hyeonseo Yu, Honggyu An +10

KAIST AI · ETH Zürich · Google +2

3D Spatial Reasoning 3D Scene Representation 3D Scene Understanding 3D VLMs Multimodal Spatial Reasoning

Ask PDF

Abstract

Multimodal Spatial Reasoning

Explore similar work

Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations

Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning

Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering