Multimodal Large Language Models (MLLMs) have made substantial progress in 3D scene understanding from multi-view RGB observations. Yet a fixed object and query can elicit inconsistent target-related responses across camera viewpoints, a phenomenon we characterize as viewpoint-induced semantic drift. We introduce LieVLM, a Lie routing framework for view-robust 3D understanding. Motivated by the rotational component of viewpoint change, Lie routing adapts visual representations by applying local SO(3) transformations to projected features within learned three-dimensional subspaces. Geometry tokens extracted by VGGT predict local rotation parameters and channel-wise routing coefficients. A closed-form Padé--Cayley map transforms projected visual features, whose identity-relative corrections are projected back and gated into the original visual and geometry features. This design retains the pretrained semantic and direct spatial context while learning view-dependent feature adjustments. Experiments on 3D captioning, grounding, detection, and spatial reasoning show consistent improvements over additive fusion across training scales and backbones. Controlled ablations examine the roles of geometric conditioning and transformation structure. Viewpoint analyses further reveal systematic variation in the predicted routing states alongside more localized target responses and stronger task performance across viewpoint changes.
Figures & tables
Figure 1: 3D scene viewpoint variation exposes semantic instability and downstream performance degradation. (a) Without routing, query-relevant activations become diffuse and less target centered across views, while Lie routing preserves stable target responses. Warmer regions indicate stronger support for the target phrase. (b) Fitted task trends show that performance degrades as sample-level viewpoint variation ΔR increases, while LieVLM maintains stronger performance across higher levels of viewpoint variation. For grounding, raw IoU visualizes the continuous quantity underlying [email protected]. Additional examples are provided in Appendix D.1 .
Figure 2: Framework of the proposed LieVLM. Lie-State Elicitation extracts latent operator states and routing coefficients from geometry tokens. Lie Group Routing transforms projected visual tokens, and routed fusion combines semantic, geometric, and operator paths before language generation.
Table 3: The quantitative results on 3D video object detection on EmbodiedScan [ 36 ] .
Figure 3: Qualitative results of LieVLM for 3D visual grounding.
Figure 4: Qualitative results for 3D video object detection.
Method
Data Size
Avg.
Low
Medium
High
CamMotion
ViewChgI
DistI-OO-MV
ObjRel-OC-MV
SpImag-OO-MV
Prior MLLMs and Spatial Models
InternVL2.5-8B [ 8 ]
N/A
36.28
29.46
31.88
43.80
28.00
6.32
56.85
51.75
39.50
LLaVA-OV-7B [ 27 ]
N/A
31.20
21.79
26.13
40.14
30.25
9.47
55.06
37.25
35.01
Qwen2.5-VL-7B [ 1 ]
N/A
33.07
28.75
22.97
40.27
28.75
6.83
51.49
44.75
31.93
SPAR-8B [ 40 ]
2M
63.25
65.53
63.01
61.29
73.00
37.14
79.76
64.00
53.50
VG LLM-4B [ 42 ]
297K
60.36
52.81
51.35
72.92
63.00
25.82
63.69
81.25
71.99
Table 4: Comparison on SPAR-Bench Full under the 3.05M setting. Prior model results follow published evaluations, and controlled rows use identical training and evaluation.
Table 7
Figure 5: Routing viewpoint correspondence. Spearman correlation ρ between physical camera rotation and internal representation change across same scene frame pairs.
Table 7: Operator structure ablation under matched training. Results are mean ± std over three seeds.
Table 10
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Data Component
Public Sources
Sampling
Examples
General video instruction
LLaVA Hound split of LLaVA Video 178K
Full released split
0.26M
Large scale spatial QA
SPAR subsets from ScanNet, ScanNet++, Structured3D, and RxR
10% of each training subset
1.63M
3D grounding and captioning
ScanRefer and Scan2Cap
Official training splits
0.07M
Other spatially grounded tasks
ScanNet, 3RScan, Matterport3D, ARKitScenes, Objectron, and SUN RGB-D
Released annotations
1.09M
Total
–
–
3.05M
Appendix
Table 10: Composition of the 3.05M generalist training mixture.
Figure 6: Additional controlled viewpoint response examples extending Fig. 1 (a). query-relevant activations are shown across reference, near, medium, and far views without and with Lie routing. Warmer regions indicate stronger support for the target phrase. Red wireframes denote projected ground truth boxes used only for visualization and [email protected].
Scene
Target
Routing
Near
Medium
Far
scene0011_00
Long table
Without Routing
44.6
2.7
49.5
Lie Routing
81.6
49.4
62.8
scene0064_00
Wooden shelf
Without Routing
82.6
64.7
34.0
Lie Routing
86.2
74.0
86.7
scene0246_00
White sink
Without Routing
73.5
40.6
15.6
Lie Routing
95.1
71.5
80.7
Appendix
Table 11: Target overlap of query-relevant activations under controlled viewpoint variation. [email protected] reports the percentage of top response locations inside the projected target box.
Object
Qwen2.5-VL
VG LLM
LieVLM (Ours)
chair
41.2
54.0
53.5
pillow
13.2
22.9
25.7
cabinet
11.6
17.1
21.3
table
36.5
46.5
56.8
lamp
9.0
18.9
23.3
couch
41.1
47.0
55.1
Appendix
Table 12: The detailed results on 3D video object detection in 4-frame setting. We report the F1 score for all categories.
Table 14: Multi-seed reliability on EmbodiedScan. Results are mean ± std over three matched random seeds.
Method
Avg.
Numerical Answer
Multiple Choice Answer
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
Prior MLLMs and Spatial Models
GPT-4o
34.0
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
Gemini-1.5-Pro
45.4
56.2
30.9
64.1
43.6
51.3
46.3
36.0
34.6
Gemini-2.5-Pro
52.7
48.2
35.6
71.3
51.7
58.9
42.4
46.8
66.7
InternVL2-8B
34.6
23.1
28.7
48.2
39.8
36.7
30.7
29.9
39.6
Appendix
Table 15: Detailed VSI-Bench comparison across model and scaling settings. Green values denote changes over the corresponding controlled VG LLM baseline.
Metric
VG LLM-8B
LieVLM-8B
Paired Gain
Positive Seeds
Obj. Count
67.88±0.26
69.58±0.14
+1.70±0.12
3/3
Abs. Dist.
37.68±0.24
40.50±0.11
+2.82±0.13
3/3
Obj. Size
58.60±0.06
58.70±0.09
+0.10±0.03
3/3
Room Size
61.98±0.21
64.69±0.13
+2.71±0.09
3/3
Rel. Dist.
46.57±0.22
49.11±0.10
+2.54±0.12
3/3
Rel. Dir.
40.69±0.22
42.90±0.15
+2.20±0.08
3/3
Appendix
Table 16: Multi seed reliability on VSI-Bench. Results are mean ± standard deviation over three matched seeds, with paired gains computed seed wise.
Table 17: Performance under decreasing input view density. Relative density is measured with respect to the task specific full input setting. Both models receive identical frame subsets sampled from the same temporal interval. The full-density results are from the runs used for the sparse-view study, whereas Table 7 reports three-seed averages.
Table 18: Inference time robustness to geometry perturbations. Values in brackets report performance retained relative to the clean setting. The clean results are from the runs used for the perturbation study, whereas the multi-seed detection results are reported in Table 14 .
Figure 7: Additional visualization results for 3D visual grounding.
Figure 8: Additional visualization results for 3D video object detection.
Multimodal Large Language Models (MLLMs) have made impressive progress in connecting vision and language, but they still struggle with spatial understanding and viewpoint-aware reasoning. Recent efforts aim to augment the input representations with geometric cues rather than explicitly teaching models to reason in 3D space. We introduce Loc3R-VLM, a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input. Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction to build a holistic representation of the scene structure, and explicit situation modeling to anchor egocentric perspective. These objectives provide direct spatial supervision that grounds both perception and language in a 3D context. To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model. Loc3R-VLM achieves state-of-the-art performance in language-based localization and outperforms existing 2D- and video-based approaches on situated and general 3D question-answering benchmarks, demonstrating that our spatial supervision framework enables strong 3D understanding. Project page: https://kevinqu7.github.io/loc3r-vlm
3D visual grounding aims to localize the target object in a 3D scene from a natural language query, requiring both fine-grained semantic understanding and viewpoint-dependent spatial reasoning. Existing methods typically formulate semantic understanding as an auxiliary closed-set object classification task and rely on multi-view feature aggregation for viewpoint reasoning, limiting semantic generalization and weakening viewpoint-specific evidence. We observe that vision-language models naturally provide complementary capabilities through open-vocabulary semantic understanding and global scene perception. Based on this insight, we propose GuideGround, a VLM-guided framework that complements rather than replaces task-specific grounding models by leveraging VLMs for semantic enhancement and viewpoint-specific hypothesis verification. Specifically, we replace auxiliary closed-set object classification with VLM-generated object semantic descriptions to enhance semantic understanding. Meanwhile, instead of directly aggregating multi-view representations, we preserve viewpoint-specific grounding hypotheses through per-view grounding and explicitly verify them using VLMs across candidate viewpoints. Extensive experiments on the ReferIt3D benchmark demonstrate that GuideGround consistently outperforms previous state-of-the-art methods. Comprehensive ablation studies further confirm the effectiveness of both the proposed semantic understanding and viewpoint reasoning strategies.
Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.