cs.CVMar 17, 2026
SaveGAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models
Organizations: Harbin Institute of Technology, China · School of Electronic and Computer Engineering, Peking University, China · Huawei, China
Abstract
Multimodal Large Language Models (MLLMs) demonstrate exceptional semantic reasoning but struggle with 3D spatial perception when restricted to pure RGB inputs. Despite leveraging implicit geometric priors from 3D reconstruction models, image-based methods still exhibit a notable performance gap compared to methods using explicit 3D data. We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm: text-dominated fine-tuning fails to activate geometric representations within MLLMs. Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision, leading to suboptimal structural utilization. To address this limitation, we propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation. Specifically, we introduce a visual-prompted joint task that compels the MLLMs to predict sparse pointmaps alongside semantic labels, thereby enforcing geometric awareness. Furthermore, we design a multi-level progressive fusion module with a token-level gating mechanism, enabling adaptive integration of geometric priors without suppressing semantic reasoning. Extensive experiments demonstrate that GAP-MLLM significantly enhances geometric feature fusion and consistently enhances performance across 3D visual grounding, 3D dense captioning, and 3D video object detection tasks.
Figures & tables
Figure 1: Geometry-aligned pre-training significantly improves 3D perception in MLLMs. (A) Naive fusion without geometry-aware pre-training leads to limited geometric utilization and inaccurate 3D detection. (B) Our sparse geometry–semantics joint pre-training and multi-level fusion module progressively integrate geometric priors, activating structural perception and yielding substantial gains in 3D detection mAP.
Figure 2: Geometry–semantics imbalance in existing 3D perception paradigms. Point cloud-based methods ensure geometric accuracy but lack semantics, whereas image-based geometric encoders (GE) with VLMs retain semantics yet under-utilize geometry. As a result, both paradigms exhibit suboptimal performance in 3D perception.
Figure 3: Failure Analysis. Global attention maps of the geometric encoder are visualized using three representative tokens across layers. The same token exhibits distinct attention patterns at different layers. Prior method that relies on last-layer tokens as implicit geometric priors [ 59 ] leads to inaccurate 3D bounding box prediction.
Figure 4: Network Architecture. An image sequence is processed by parallel geometric and visual branches to extract multi-level structural and semantic tokens. After gated multi-level fusion, the final-layer fused tokens are aligned with task-related textual representations in the video LLM decoder, while selected intermediate-layer tokens are injected into early decoder blocks to preserve hierarchical geometric information.
Figure 5: Training Strategy. Sparse joint pre-training (left) activates 3D representations under a unified first-frame metric coordinate system. The learned representations are then transferred to object-level fine-tuning (right) for downstream 3D perception.
Figure 6: Qualitative comparison on 3D visual grounding. We compare VG-LLM and GAP-MLLM against ground truth (GT). Our predictions exhibit tighter geometric alignment and more accurate box fitting under the same textual descriptions.
| Model | 3D Input | 4-Frame Setting | 6-Frame Setting | ||||
| P 25 | R 25 | F1 25 | P 25 | R 25 | F1 25 | ||
| SpatialLM [ 36 ] | ✓ | 40.8 | 23.7 | 29.1 | 42.1 | 26.1 | 31.1 |
| VG-LLM-4B [ 59 ] | ✗ | 41.7 | 35.7 | 38.2 | 39.7 | 34.0 | 36.4 |
| VG-LLM-8B [ 59 ] | ✗ | 43.4 | 39.6 | 41.2 | 43.5 | 38.7 | 40.8 |
| VG-LLM-4B (w/ GAP) | ✗ | 47.0 | 41.2 | 43.5 | 48.8 | 40.2 | 43.6 |
| GAP-MLLM-3B (Ours) | ✗ | 54.2 | 48.1 | 50.6 | 52.9 | 45.4 | 48.5 |
Table 3: 3D Video Object Detection Results. GAP-MLLM achieves the highest precision, recall, and F1 under both 4-frame and 6-frame settings.
Figure 7: Qualitative comparison on 3D video object detection. We compare SpatialLM [ 36 ] , VG-LLM [ 59 ] , and GAP-MLLM. Our method produces more accurate and spatially consistent 3D bounding boxes across multi-frame inputs.
| Model | Aligned Evaluation | Metric Evaluation | ||||||||||
| Acc. | Comp. | Overall. | Acc. | Comp. | Overall. | |||||||
| Mean | Med. | Mean | Med. | Mean | Med. | Mean | Med. | Mean | Med. | Mean | Med. | |
| CUT3R [ 49 ] | 0.0302 | 0.0227 | 0.0273 | 0.0207 | 0.0287 | 0.0217 | 0.1199 | 0.0998 | 0.1045 | 0.0871 | 0.1122 | 0.0937 |
| Mapanything [ 27 ] | 0.0688 | 0.0397 | 0.0610 | 0.0480 | 0.0649 | 0.0438 | 0.3409 | 0.2721 | 0.2498 | 0.2265 | 0.2953 | 0.2493 |
| GAP-MLLM-3B (w/o semantic) | 0.0424 | 0.0295 | 0.0326 | 0.0246 | 0.0375 | 0.0270 | 0.0691 | 0.0580 | 0.0570 | 0.0465 | 0.0630 | 0.0523 |
| GAP-MLLM-3B (Ours) | 0.0421 | 0.0293 | 0.0327 | 0.0247 | 0.0374 | 0.0270 | 0.0675 | 0.0553 | 0.0557 | 0.0441 | 0.0616 | 0.0497 |
Table 4: PointMap Estimation Results on ScanNet (meter). We report Accuracy, Completeness, and Overall under aligned evaluation (after Sim(3) alignment to GT) and metric evaluation (direct comparison with GT without alignment).
Figure 8: Qualitative results on pointmap and semantic prediction. Despite being trained with sparse supervision, GAP-MLLM reconstructs metrically consistent 3D pointmaps and dense semantic regions. White points denote the unknown category.
| A | B | VG-LLM-4B [ 59 ] (Qwen2.5-VL + VGGT) | GAP-MLLM-3B (Qwen3-VL + VGGT) | ||||
| P 25 | R 25 | F1 25 | P 25 | R 25 | F1 25 | ||
| 41.7 | 35.7 | 38.2 | 48.8 | 42.1 | 44.7 | ||
| ✓ | 44.1 | 39.6 | 41.4 | 52.7 | 45.9 | 48.7 | |
| ✓ | 43.3 | 36.9 | 39.4 | 51.8 | 44.6 | 47.5 | |
| ✓ | ✓ | 47.0 | 41.2 | 43.5 | 54.2 | 48.1 | 50.6 |
Table 5: Ablation on 3D video object detection. Both sparse joint pre-training (A) and multi-level gated fusion (B) contribute complementary gains.
Table 9: The prompt for sparse geometry–semantics joint pre-training. Given multi-frame images and a queried pixel marked by a red cross, the model predicts the semantic label and the 3D coordinate of the point in the first-frame coordinate system.
Table 10: Two-stage prompt for 3D visual grounding. The first stage selects the most informative frame for grounding, and the second stage predicts the 3D bounding box under the coordinate system of the selected frame.
Table 11: The prompt for 3D dense captioning. Given multi-frame images and a queried 3D point in the first-frame coordinate system, the model generates a natural language description of the corresponding object.
Table 12: Task prompt for 3D video object detection. Given multi-frame images, the model predicts all object categories and their 3D bounding boxes in the first-frame coordinate system.
| A | B | ScanRefer | Scan2Cap | 3D Video Detection | ||||
| [email protected] | [email protected] | [email protected] | [email protected] | P 25 | R 25 | F1 25 | ||
| VG-LLM-4B (Qwen2.5-VL + VGGT) | ||||||||
| 45.4 | 19.3 | 77.0 | 40.6 | 41.2 | 35.9 | 37.9 | ||
| ✓ | 48.2 | 22.6 | 79.7 | 40.8 | 44.1 | 39.6 | 41.4 | |
| ✓ | 46.8 | 20.5 | 77.5 | 40.6 | 43.3 | 36.9 | 39.4 | |
| ✓ | ✓ | 49.7 | 23.2 | 80.6 | 41.1 | 47.0 | 41.2 | 43.5 |
Table 13: Ablation on sparse joint pre-training (A) and multi-level gated fusion (B) across multiple tasks.
| Method | ScanRefer | Scan2Cap | 3D Video Detection | ||||
| [email protected] | [email protected] | [email protected] | [email protected] | P 25 | R 25 | F1 25 | |
| VG-LLM | 37.5 | 11.1 | 76.9 | 40.5 | 40.3 | 34.9 | 36.9 |
| VG-LLM + StageGrounding | 45.4 | 19.3 | 77.0 | 40.6 | 41.2 | 35.9 | 37.9 |
Table 14: Effect of Two-Stage Grounding under the unified coordinate system. Introducing Two-Stage Grounding consistently improves performance across multiple 3D perception tasks under the unified coordinate system.
| Setting | Precision | Recall | F1 |
| GAP-MLLM | 54.2 | 48.1 | 50.6 |
| Blur ( ) | 52.1 | 44.4 | 47.5 |
| Blur ( ) | 50.7 | 42.9 | 46.0 |
| Occlusion (30%) | 53.8 | 47.6 | 50.1 |
| Occlusion (50%) | 51.9 | 46.0 | 48.4 |
Table 15: Robustness under first-frame degradation. Blur severity is controlled by Gaussian kernel standard deviation , and occlusion severity is defined as the fraction of masked image area.
Figure 10: Qualitative results under first-frame degradation. Even when the first frame is severely corrupted (e.g., blur or occlusion), GAP-MLLM can still recover coherent object-level 3D structure by leveraging temporal information from subsequent frames, demonstrating strong robustness to unreliable initial observations.
| Method | Small-object F1 (per category) | Average metrics | |||||||||
| Soap | TP | TB | Pillow | Case | Plant | HD | Disp | P 25 | R 25 | F1 25 | |
| VG-LLM | 11.3 | 18.0 | 2.4 | 14.8 | 34.9 | 15.3 | 0.0 | 31.4 | 18.0 | 15.2 | 16.0 |
| GAP-MLLM | 44.4 | 32.4 | 15.2 | 28.4 | 53.2 | 19.6 | 12.5 | 43.6 | 36.4 | 29.7 | 31.2 |
Table 16: Performance on small-object categories under patch merging. The table reports per-category F1 scores and overall averaged metrics over eight categories. TP = Toilet paper, TB = Tissue box, HD = Hair dryer, Disp = Dispenser.
Figure 11: Effect of patch merging on small-object perception. Compared with VG-LLM, GAP-MLLM better preserves small-object details under token compression. This indicates that multi-level geometric fusion effectively mitigates information loss caused by patch merging.
Figure 12: Dynamic-scene extension with sliding-window inference. Each local window predicts 3D boxes in the coordinate system of its first frame. GAP-MLLM produces plausible object-level predictions on NuScenes, showing preliminary generalization to dynamic outdoor scenes.
Figure 13: Qualitative comparison on metric reconstruction. Given multi-view RGB inputs (left), different models reconstruct 3D metric point clouds, where the red line indicates the measured distance. Our method produces metric distances closest to the ground-truth scale while preserving coherent geometry, demonstrating that geometry-aligned pre-training enables MLLMs to generate metrically pointmap.
Figure 14: Additional qualitative results on 3D visual grounding. Given a language query and multi-frame images, the model predicts the 3D bounding box of the described object. Compared with VG-LLM, our method produces more accurate and stable localization results that better match the ground truth.
Figure 15: Additional qualitative results on 3D video object detection. Compared with SpatialLM [ 36 ] and VG-LLM [ 59 ] , our method produces more accurate and consistent 3D bounding boxes for multiple objects in the scene, benefiting from improved geometric perception under the unified coordinate system.
More in this topic
Multimodal Large Language Models
Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.
See all 1,091 papers