Multimodal Large Language Models (MLLMs) demonstrate exceptional semantic reasoning but struggle with 3D spatial perception when restricted to pure RGB inputs. Despite leveraging implicit geometric priors from 3D reconstruction models, image-based methods still exhibit a notable performance gap compared to methods using explicit 3D data. We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm: text-dominated fine-tuning fails to activate geometric representations within MLLMs. Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision, leading to suboptimal structural utilization. To address this limitation, we propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation. Specifically, we introduce a visual-prompted joint task that compels the MLLMs to predict sparse pointmaps alongside semantic labels, thereby enforcing geometric awareness. Furthermore, we design a multi-level progressive fusion module with a token-level gating mechanism, enabling adaptive integration of geometric priors without suppressing semantic reasoning. Extensive experiments demonstrate that GAP-MLLM significantly enhances geometric feature fusion and consistently enhances performance across 3D visual grounding, 3D dense captioning, and 3D video object detection tasks.
Figures & tables
Figure 1: Geometry-aligned pre-training significantly improves 3D perception in MLLMs. (A) Naive fusion without geometry-aware pre-training leads to limited geometric utilization and inaccurate 3D detection. (B) Our sparse geometry–semantics joint pre-training and multi-level fusion module progressively integrate geometric priors, activating structural perception and yielding substantial gains in 3D detection mAP.
Figure 2: Geometry–semantics imbalance in existing 3D perception paradigms. Point cloud-based methods ensure geometric accuracy but lack semantics, whereas image-based geometric encoders (GE) with VLMs retain semantics yet under-utilize geometry. As a result, both paradigms exhibit suboptimal performance in 3D perception.
Figure 3: Failure Analysis. Global attention maps of the geometric encoder are visualized using three representative tokens across layers. The same token exhibits distinct attention patterns at different layers. Prior method that relies on last-layer tokens as implicit geometric priors [ 59 ] leads to inaccurate 3D bounding box prediction.
Figure 4: Network Architecture. An image sequence is processed by parallel geometric and visual branches to extract multi-level structural and semantic tokens. After gated multi-level fusion, the final-layer fused tokens are aligned with task-related textual representations in the video LLM decoder, while selected intermediate-layer tokens are injected into early decoder blocks to preserve hierarchical geometric information.
Figure 5: Training Strategy. Sparse joint pre-training (left) activates 3D representations under a unified first-frame metric coordinate system. The learned representations are then transferred to object-level fine-tuning (right) for downstream 3D perception.
Table 6
Figure 6: Qualitative comparison on 3D visual grounding. We compare VG-LLM and GAP-MLLM against ground truth (GT). Our predictions exhibit tighter geometric alignment and more accurate box fitting under the same textual descriptions.
Model
3D Input
4-Frame Setting
6-Frame Setting
P 25
R 25
F1 25
P 25
R 25
F1 25
SpatialLM [ 36 ]
✓
40.8
23.7
29.1
42.1
26.1
31.1
VG-LLM-4B [ 59 ]
✗
41.7
35.7
38.2
39.7
34.0
36.4
VG-LLM-8B [ 59 ]
✗
43.4
39.6
41.2
43.5
38.7
40.8
VG-LLM-4B (w/ GAP)
✗
47.0
41.2
43.5
48.8
40.2
43.6
GAP-MLLM-3B (Ours)
✗
54.2
48.1
50.6
52.9
45.4
48.5
Table 3: 3D Video Object Detection Results. GAP-MLLM achieves the highest precision, recall, and F1 under both 4-frame and 6-frame settings.
Figure 7: Qualitative comparison on 3D video object detection. We compare SpatialLM [ 36 ] , VG-LLM [ 59 ] , and GAP-MLLM. Our method produces more accurate and spatially consistent 3D bounding boxes across multi-frame inputs.
Model
Aligned Evaluation
Metric Evaluation
Acc. ↓
Comp. ↓
Overall. ↓
Acc. ↓
Comp. ↓
Overall. ↓
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
Mean
Med.
CUT3R [ 49 ]
0.0302
0.0227
0.0273
0.0207
0.0287
0.0217
0.1199
0.0998
0.1045
0.0871
0.1122
0.0937
Mapanything [ 27 ]
0.0688
0.0397
0.0610
0.0480
0.0649
0.0438
0.3409
0.2721
0.2498
0.2265
0.2953
0.2493
GAP-MLLM-3B (w/o semantic)
0.0424
0.0295
0.0326
0.0246
0.0375
0.0270
0.0691
0.0580
0.0570
0.0465
0.0630
0.0523
GAP-MLLM-3B (Ours)
0.0421
0.0293
0.0327
0.0247
0.0374
0.0270
0.0675
0.0553
0.0557
0.0441
0.0616
0.0497
Table 4: PointMap Estimation Results on ScanNet (meter). We report Accuracy, Completeness, and Overall under aligned evaluation (after Sim(3) alignment to GT) and metric evaluation (direct comparison with GT without alignment).
Figure 8: Qualitative results on pointmap and semantic prediction. Despite being trained with sparse supervision, GAP-MLLM reconstructs metrically consistent 3D pointmaps and dense semantic regions. White points denote the unknown category.
A
B
VG-LLM-4B [ 59 ] (Qwen2.5-VL + VGGT)
GAP-MLLM-3B (Qwen3-VL + VGGT)
P 25
R 25
F1 25
P 25
R 25
F1 25
41.7
35.7
38.2
48.8
42.1
44.7
✓
44.1
39.6
41.4
52.7
45.9
48.7
✓
43.3
36.9
39.4
51.8
44.6
47.5
✓
✓
47.0
41.2
43.5
54.2
48.1
50.6
Table 5: Ablation on 3D video object detection. Both sparse joint pre-training (A) and multi-level gated fusion (B) contribute complementary gains.
Table 13Figure 14
Table 9: The prompt for sparse geometry–semantics joint pre-training. Given multi-frame images and a queried pixel marked by a red cross, the model predicts the semantic label and the 3D coordinate of the point in the first-frame coordinate system.
Table 10: Two-stage prompt for 3D visual grounding. The first stage selects the most informative frame for grounding, and the second stage predicts the 3D bounding box under the coordinate system of the selected frame.
Table 11: The prompt for 3D dense captioning. Given multi-frame images and a queried 3D point in the first-frame coordinate system, the model generates a natural language description of the corresponding object.
Table 12: Task prompt for 3D video object detection. Given multi-frame images, the model predicts all object categories and their 3D bounding boxes in the first-frame coordinate system.
Table 14: Effect of Two-Stage Grounding under the unified coordinate system. Introducing Two-Stage Grounding consistently improves performance across multiple 3D perception tasks under the unified coordinate system.
Setting
Precision
Recall
F1
GAP-MLLM
54.2
48.1
50.6
Blur ( σ=8 )
52.1
44.4
47.5
Blur ( σ=12 )
50.7
42.9
46.0
Occlusion (30%)
53.8
47.6
50.1
Occlusion (50%)
51.9
46.0
48.4
Table 15: Robustness under first-frame degradation. Blur severity is controlled by Gaussian kernel standard deviation σ , and occlusion severity is defined as the fraction of masked image area.
Figure 10: Qualitative results under first-frame degradation. Even when the first frame is severely corrupted (e.g., blur or occlusion), GAP-MLLM can still recover coherent object-level 3D structure by leveraging temporal information from subsequent frames, demonstrating strong robustness to unreliable initial observations.
Method
Small-object F1 (per category)
Average metrics
Soap
TP
TB
Pillow
Case
Plant
HD
Disp
P 25
R 25
F1 25
VG-LLM
11.3
18.0
2.4
14.8
34.9
15.3
0.0
31.4
18.0
15.2
16.0
GAP-MLLM
44.4
32.4
15.2
28.4
53.2
19.6
12.5
43.6
36.4
29.7
31.2
Table 16: Performance on small-object categories under patch merging. The table reports per-category F1 scores and overall averaged metrics over eight categories. TP = Toilet paper, TB = Tissue box, HD = Hair dryer, Disp = Dispenser.
Figure 11: Effect of patch merging on small-object perception. Compared with VG-LLM, GAP-MLLM better preserves small-object details under token compression. This indicates that multi-level geometric fusion effectively mitigates information loss caused by patch merging.
Figure 12: Dynamic-scene extension with sliding-window inference. Each local window predicts 3D boxes in the coordinate system of its first frame. GAP-MLLM produces plausible object-level predictions on NuScenes, showing preliminary generalization to dynamic outdoor scenes.
Figure 13: Qualitative comparison on metric reconstruction. Given multi-view RGB inputs (left), different models reconstruct 3D metric point clouds, where the red line indicates the measured distance. Our method produces metric distances closest to the ground-truth scale while preserving coherent geometry, demonstrating that geometry-aligned pre-training enables MLLMs to generate metrically pointmap.
Figure 14: Additional qualitative results on 3D visual grounding. Given a language query and multi-frame images, the model predicts the 3D bounding box of the described object. Compared with VG-LLM, our method produces more accurate and stable localization results that better match the ground truth.
Figure 15: Additional qualitative results on 3D video object detection. Compared with SpatialLM [ 36 ] and VG-LLM [ 59 ] , our method produces more accurate and consistent 3D bounding boxes for multiple objects in the scene, benefiting from improved geometric perception under the unified coordinate system.
While Multimodal Large Language Models (MLLMs) have achieved remarkable success in 2D visual understanding, their ability to reason about 3D space remains limited. To address this gap, we introduce geometrically referenced 3D scene representations (GR3D). Given a set of input images, GR3D annotates objects in the images with unique IDs and encodes their 3D geometric attributes as textual references indexed by these IDs. This representation enables MLLMs to interpret 3D cues using their advanced language-based skills in mathematical reasoning, while concurrently analyzing 2D visual features in a tightly coupled way. We present a simple yet effective approach based on GR3D, which requires no additional training and is readily applicable to different MLLMs. Implemented in a zero-shot setting, our approach yields substantial improvements on challenging spatial reasoning benchmarks, boosting GPT-5 performance by 9% on VSI-Bench and 12% on MindCube. Qualitative studies further demonstrate that GR3D empowers MLLMs to perform complex spatial reasoning with highly sparse input views.
Multimodal large language models (MLLMs) have exhibited remarkable performance in various visual tasks, yet still struggle with spatial reasoning. Recent efforts mitigate this by injecting geometric features from 3D foundation models, but rely on static single-layer extractions. We identify that such an approach induces a task misalignment bias: the geometric features naturally evolve towards 3D pretraining objectives, which may contradict the heterogeneous spatial demands of MLLMs, rendering any single layer fundamentally insufficient. To resolve this, we propose GeoAlign, a novel framework that dynamically aggregates multi-layer geometric features to realign with the actual demands. GeoAlign constructs a hierarchical geometric feature bank and leverages the MLLM's original visual tokens as content-aware queries to perform layer-wise sparse routing, adaptively fetching the suitable geometric features for each patch. Extensive experiments on VSI-Bench, ScanQA, and SQA3D demonstrate that our compact 4B model effectively achieves state-of-the-art performance, even outperforming larger existing MLLMs.
Zhaochen Liu, Limeng Qiao, Guanglu Wan +1
National Engineering Research Center of Visual Technology, National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · Meituan Inc. · National Biomedical Imaging Center, Peking University
Although Multimodal Large Language Models have achieved remarkable progress, they still struggle with complex 3D spatial reasoning due to the reliance on 2D visual priors. Existing approaches typically mitigate this limitation either through computationally expensive post-training procedures on limited 3D datasets or through rigid tool-calling mechanisms that lack explicit geometric understanding and viewpoint flexibility. To address these challenges, we propose a training-free framework that introduces a Visual Chain-of-Thought mechanism grounded in explicit 3D reconstruction. The proposed pipeline first reconstructs a high-fidelity 3D mesh from a single image using MLLM-guided keyword extraction and mask generation at multiple granularities. Subsequently, the framework leverages an external knowledge base to iteratively compute optimal camera extrinsic parameters and synthesize novel views, thereby emulating human perspective-taking. Extensive experiments demonstrate that the proposed approach significantly enhances spatial comprehension. Specifically, the framework outperforms specialized spatial models and general-purpose MLLMs, including GPT-5.2 and Gemini-2.5-Flash, on major benchmarks such as 3DSRBench and Rel3D.
Jiahua Chen, Qihong Tang, Weinong Wang +1
Tsinghua University · Nanjing University · Tencent