Multimodal Large Language Models (MLLMs) have made substantial progress in 3D scene understanding from multi-view RGB observations. Yet a fixed object and query can elicit inconsistent target-related responses across camera viewpoints, a phenomenon we characterize as viewpoint-induced semantic drift. We introduce LieVLM, a Lie routing framework for view-robust 3D understanding. Motivated by the rotational component of viewpoint change, Lie routing adapts visual representations by applying local SO(3) transformations to projected features within learned three-dimensional subspaces. Geometry tokens extracted by VGGT predict local rotation parameters and channel-wise routing coefficients. A closed-form Padé--Cayley map transforms projected visual features, whose identity-relative corrections are projected back and gated into the original visual and geometry features. This design retains the pretrained semantic and direct spatial context while learning view-dependent feature adjustments. Experiments on 3D captioning, grounding, detection, and spatial reasoning show consistent improvements over additive fusion across training scales and backbones. Controlled ablations examine the roles of geometric conditioning and transformation structure. Viewpoint analyses further reveal systematic variation in the predicted routing states alongside more localized target responses and stronger task performance across viewpoint changes.
Figures & tables
Figure 1: 3D scene viewpoint variation exposes semantic instability and downstream performance degradation. (a) Without routing, query-relevant activations become diffuse and less target centered across views, while Lie routing preserves stable target responses. Warmer regions indicate stronger support for the target phrase. (b) Fitted task trends show that performance degrades as sample-level viewpoint variation ΔR increases, while LieVLM maintains stronger performance across higher levels of viewpoint variation. For grounding, raw IoU visualizes the continuous quantity underlying [email protected]. Additional examples are provided in Appendix D.1 .
Figure 2: Framework of the proposed LieVLM. Lie-State Elicitation extracts latent operator states and routing coefficients from geometry tokens. Lie Group Routing transforms projected visual tokens, and routed fusion combines semantic, geometric, and operator paths before language generation.
Table 3: The quantitative results on 3D video object detection on EmbodiedScan [ 36 ] .
Figure 3: Qualitative results of LieVLM for 3D visual grounding.
Figure 4: Qualitative results for 3D video object detection.
Method
Data Size
Avg.
Low
Medium
High
CamMotion
ViewChgI
DistI-OO-MV
ObjRel-OC-MV
SpImag-OO-MV
Prior MLLMs and Spatial Models
InternVL2.5-8B [ 8 ]
N/A
36.28
29.46
31.88
43.80
28.00
6.32
56.85
51.75
39.50
LLaVA-OV-7B [ 27 ]
N/A
31.20
21.79
26.13
40.14
30.25
9.47
55.06
37.25
35.01
Qwen2.5-VL-7B [ 1 ]
N/A
33.07
28.75
22.97
40.27
28.75
6.83
51.49
44.75
31.93
SPAR-8B [ 40 ]
2M
63.25
65.53
63.01
61.29
73.00
37.14
79.76
64.00
53.50
VG LLM-4B [ 42 ]
297K
60.36
52.81
51.35
72.92
63.00
25.82
63.69
81.25
71.99
Table 4: Comparison on SPAR-Bench Full under the 3.05M setting. Prior model results follow published evaluations, and controlled rows use identical training and evaluation.
Table 7
Figure 5: Routing viewpoint correspondence. Spearman correlation ρ between physical camera rotation and internal representation change across same scene frame pairs.
Table 7: Operator structure ablation under matched training. Results are mean ± std over three seeds.
Table 10
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Data Component
Public Sources
Sampling
Examples
General video instruction
LLaVA Hound split of LLaVA Video 178K
Full released split
0.26M
Large scale spatial QA
SPAR subsets from ScanNet, ScanNet++, Structured3D, and RxR
10% of each training subset
1.63M
3D grounding and captioning
ScanRefer and Scan2Cap
Official training splits
0.07M
Other spatially grounded tasks
ScanNet, 3RScan, Matterport3D, ARKitScenes, Objectron, and SUN RGB-D
Released annotations
1.09M
Total
–
–
3.05M
Appendix
Table 10: Composition of the 3.05M generalist training mixture.
Figure 6: Additional controlled viewpoint response examples extending Fig. 1 (a). query-relevant activations are shown across reference, near, medium, and far views without and with Lie routing. Warmer regions indicate stronger support for the target phrase. Red wireframes denote projected ground truth boxes used only for visualization and [email protected].
Scene
Target
Routing
Near
Medium
Far
scene0011_00
Long table
Without Routing
44.6
2.7
49.5
Lie Routing
81.6
49.4
62.8
scene0064_00
Wooden shelf
Without Routing
82.6
64.7
34.0
Lie Routing
86.2
74.0
86.7
scene0246_00
White sink
Without Routing
73.5
40.6
15.6
Lie Routing
95.1
71.5
80.7
Appendix
Table 11: Target overlap of query-relevant activations under controlled viewpoint variation. [email protected] reports the percentage of top response locations inside the projected target box.
Object
Qwen2.5-VL
VG LLM
LieVLM (Ours)
chair
41.2
54.0
53.5
pillow
13.2
22.9
25.7
cabinet
11.6
17.1
21.3
table
36.5
46.5
56.8
lamp
9.0
18.9
23.3
couch
41.1
47.0
55.1
Appendix
Table 12: The detailed results on 3D video object detection in 4-frame setting. We report the F1 score for all categories.
Table 14: Multi-seed reliability on EmbodiedScan. Results are mean ± std over three matched random seeds.
Method
Avg.
Numerical Answer
Multiple Choice Answer
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Route Plan
Appr. Order
Prior MLLMs and Spatial Models
GPT-4o
34.0
46.2
5.3
43.8
38.2
37.0
41.3
31.5
28.5
Gemini-1.5-Pro
45.4
56.2
30.9
64.1
43.6
51.3
46.3
36.0
34.6
Gemini-2.5-Pro
52.7
48.2
35.6
71.3
51.7
58.9
42.4
46.8
66.7
InternVL2-8B
34.6
23.1
28.7
48.2
39.8
36.7
30.7
29.9
39.6
Appendix
Table 15: Detailed VSI-Bench comparison across model and scaling settings. Green values denote changes over the corresponding controlled VG LLM baseline.
Metric
VG LLM-8B
LieVLM-8B
Paired Gain
Positive Seeds
Obj. Count
67.88±0.26
69.58±0.14
+1.70±0.12
3/3
Abs. Dist.
37.68±0.24
40.50±0.11
+2.82±0.13
3/3
Obj. Size
58.60±0.06
58.70±0.09
+0.10±0.03
3/3
Room Size
61.98±0.21
64.69±0.13
+2.71±0.09
3/3
Rel. Dist.
46.57±0.22
49.11±0.10
+2.54±0.12
3/3
Rel. Dir.
40.69±0.22
42.90±0.15
+2.20±0.08
3/3
Appendix
Table 16: Multi seed reliability on VSI-Bench. Results are mean ± standard deviation over three matched seeds, with paired gains computed seed wise.
Table 17: Performance under decreasing input view density. Relative density is measured with respect to the task specific full input setting. Both models receive identical frame subsets sampled from the same temporal interval. The full-density results are from the runs used for the sparse-view study, whereas Table 7 reports three-seed averages.
Table 18: Inference time robustness to geometry perturbations. Values in brackets report performance retained relative to the clean setting. The clean results are from the runs used for the perturbation study, whereas the multi-seed detection results are reported in Table 14 .
Figure 7: Additional visualization results for 3D visual grounding.
Figure 8: Additional visualization results for 3D video object detection.