Multimodal Large Language Models (MLLMs) have achieved remarkable progress in 2D visual tasks but still struggle to understand physical space in real-world visual streams. Recently, feed-forward geometric foundation models that implicitly extract geometric priors from visual inputs have offered a promising direction. However, many existing geometry-aware MLLMs extract features from a single deep encoder layer and perform one-shot fusion at the language-model input, potentially overlooking complementary information across geometric encoder layers, limiting access to fine-grained geometric cues, and hindering progressive cross-modal adaptation. To address these limitations, we propose GUIDE (Geometric Unrolling Inside MLLM Early-layers), a progressive framework for integrating geometric priors. GUIDE samples and aligns multi-level features from the geometric encoder and sequentially injects them into the early layers of the MLLM. This design enables the MLLM to continuously access multi-granularity geometric cues and progressively integrate them with visual representations during layer-wise semantic abstraction. GUIDE further introduces a dual context-aware gating mechanism at both the token and layer levels, selectively regulating geometric information to reduce redundant injection and interference with pretrained representations. Extensive experiments on spatial reasoning and 3D scene understanding benchmarks, including VSI-Bench, ScanRefer, and Scan2Cap, validate the effectiveness and cross-task applicability of GUIDE. Our 5B and 9B models achieve average scores of 71.5 and 72.1 on VSI-Bench, respectively.
Figures & tables
Figure 1: Overall architecture of GUIDE . GUIDE employs progressive unrolling to inject multi-granularity features from the 3D geometric encoder (green) into early LLM layers (purple) via a dual context-aware gate (grey), enabling the model to continuously access geometric cues at different granularities while mitigating interference from redundant geometric signals.
Models
Avg.
Numerical Answer
Multiple-Choice Answer
Obj. Count
Abs. Dist
Obj. Size
Room Size
Rel. Dis
Rel. Dir
Route Plan
Appr. Order
Proprietary Models
Gemini-2.5-pro ( Comanici et al. 2025 )
53.6
46.0
37.4
68.7
54.4
62.0
43.9
47.4
68.8
GPT-5 ( Singh et al. 2025 )
55.0
53.3
34.5
73.3
47.5
63.7
48.7
50.3
68.9
Open-source General Models
Bagel-7B-MoT ( Deng et al. 2025 )
31.4
30.1
29.2
35.5
25.8
34.9
41.4
30.4
24.1
Table 1: Performance on VSI-Bench. We compare GUIDE with recent general-purpose and spatially enhanced models. ∗ denotes training with the S1+S2 data configuration of VG-LLM, while † denotes our scaled-data training setting.
Models
Avg.
VSI
MMSI
ViewSp.
CV
Proprietary Models
Gemini-2.5-Pro ( Comanici et al. 2025 )
55.9
53.6
38.0
46.0
85.9
GPT-5 ( Singh et al. 2025 )
56.7
55.0
41.8
45.5
84.6
Open-Source General Models
Bagel-7B-MOT ( Deng et al. 2025 )
44.9
31.4
31.0
41.3
76.0
Qwen3-VL-8B-Instruct ( Bai et al. 2025 )
52.7
57.9
28.8
39.0
85.1
Table 2: Comparison of different models on VSI-Bench ( Yang et al. 2025a ) , MMSI-Bench ( Yang et al. 2025d ) , ViewSpatial ( Li et al. 2025 ) , and CV-Bench ( Zhu et al. 2025c ) .
Table 3: Results on ScanRefer and Scan2Cap. The best result within each input setting is highlighted in bold.
Injection Layers
Gtok
Glayer
Avg.
0
×
×
51.7
3
×
×
52.4
✓
×
52.5
✓
✓
53.8
6
×
×
50.3
✓
×
52.7
Table 4: Ablation study of the number of geometric injection layers and gating components on VSI-Bench. Gtok and Glayer denote the token-level gate and layer-level gate, respectively.
Strategy
Layer Mapping
Avg.
Single Deep Layer
{23}→{1:6}
51.6
Deep VGGT Only
{18:23}→{1:6}
52.0
Early VGGT Only
{8:13}→{1:6}
53.0
Deep LLM Injection
{8:3:23}→{1:4:21}
51.7
GUIDE
{8:3:23}→{1:6}
55.6
Table 5: VSI-Bench ablation of VGGT-to-LLM layer mappings. a:s:b denotes indices from a to b with stride s ; a:b uses unit stride.
Figure 2: Token-level gates across 6 injection layers for two scenes. L∈{1,…,6} denotes the index of the LLM layers. Warmer colors indicate stronger geometric injection
Multimodal large language models (MLLMs) have exhibited remarkable performance in various visual tasks, yet still struggle with spatial reasoning. Recent efforts mitigate this by injecting geometric features from 3D foundation models, but rely on static single-layer extractions. We identify that such an approach induces a task misalignment bias: the geometric features naturally evolve towards 3D pretraining objectives, which may contradict the heterogeneous spatial demands of MLLMs, rendering any single layer fundamentally insufficient. To resolve this, we propose GeoAlign, a novel framework that dynamically aggregates multi-layer geometric features to realign with the actual demands. GeoAlign constructs a hierarchical geometric feature bank and leverages the MLLM's original visual tokens as content-aware queries to perform layer-wise sparse routing, adaptively fetching the suitable geometric features for each patch. Extensive experiments on VSI-Bench, ScanQA, and SQA3D demonstrate that our compact 4B model effectively achieves state-of-the-art performance, even outperforming larger existing MLLMs.
Zhaochen Liu, Limeng Qiao, Guanglu Wan +1
National Engineering Research Center of Visual Technology, National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · Meituan Inc. · National Biomedical Imaging Center, Peking University
Multimodal Large Language Models (MLLMs) demonstrate exceptional semantic reasoning but struggle with 3D spatial perception when restricted to pure RGB inputs. Despite leveraging implicit geometric priors from 3D reconstruction models, image-based methods still exhibit a notable performance gap compared to methods using explicit 3D data. We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm: text-dominated fine-tuning fails to activate geometric representations within MLLMs. Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision, leading to suboptimal structural utilization. To address this limitation, we propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation. Specifically, we introduce a visual-prompted joint task that compels the MLLMs to predict sparse pointmaps alongside semantic labels, thereby enforcing geometric awareness. Furthermore, we design a multi-level progressive fusion module with a token-level gating mechanism, enabling adaptive integration of geometric priors without suppressing semantic reasoning. Extensive experiments demonstrate that GAP-MLLM significantly enhances geometric feature fusion and consistently enhances performance across 3D visual grounding, 3D dense captioning, and 3D video object detection tasks.
Jiaxin Zhang, Junjun Jiang, Haijie Li +3
Harbin Institute of Technology, China · School of Electronic and Computer Engineering, Peking University, China · Huawei, China
Multimodal Large Language Models (MLLMs) excel at 2D semantic understanding but lack intrinsic 3D awareness, resulting in representations that fail to maintain geometric and spatial consistency across video frames. Given the scarcity of large-scale 3D data, we present GeoVR, a novel framework that learns geometric representations using purely 2D video sequences. This approach effectively restructures the semantic latent space within MLLMs to unlock spatial intelligence. Rather than employing superficial feature mixing, GeoVR reshapes the internal representations of the MLLM by distilling geometry knowledge from pre-trained 3D foundation models. This is accomplished through a multi-objective learning strategy driven by four complementary geometric targets: (1) estimating inter-frame camera poses to embed varying viewpoint dynamics, (2) regressing dense depth maps to anchor physical distances, (3) predicting a metric scale factor for real-world calibration, and (4) distilling multi-scale 3D features to align the intermediate feature space. Guided by these explicit physical and geometric constraints, the model's internal representations naturally develop strong 3D awareness. Extensive experiments on spatial reasoning benchmarks demonstrate that GeoVR achieves state-of-the-art performance, establishing a new paradigm for endowing foundation models with spatial intelligence.