cs.CVNov 1, 2026

Lie Routing Vision Language Models for View-Robust 3D Understanding

Authors: Jingjun Yi, Chen Hu, Qi Bi, Hao Zheng, Huimin Huang, Yongkang, Wei Ji, Yawen Huang, 

Organizations: Westlake University, China · University of Amsterdam, the Netherland · Tencent, China

Abstract

Multimodal Large Language Models (MLLMs) have made substantial progress in 3D scene understanding from multi-view RGB observations. Yet a fixed object and query can elicit inconsistent target-related responses across camera viewpoints, a phenomenon we characterize as viewpoint-induced semantic drift. We introduce LieVLM, a Lie routing framework for view-robust 3D understanding. Motivated by the rotational component of viewpoint change, Lie routing adapts visual representations by applying local SO(3)\mathrm{SO}(3) transformations to projected features within learned three-dimensional subspaces. Geometry tokens extracted by VGGT predict local rotation parameters and channel-wise routing coefficients. A closed-form Padé--Cayley map transforms projected visual features, whose identity-relative corrections are projected back and gated into the original visual and geometry features. This design retains the pretrained semantic and direct spatial context while learning view-dependent feature adjustments. Experiments on 3D captioning, grounding, detection, and spatial reasoning show consistent improvements over additive fusion across training scales and backbones. Controlled ablations examine the roles of geometric conditioning and transformation structure. Viewpoint analyses further reveal systematic variation in the predicted routing states alongside more localized target responses and stronger task performance across viewpoint changes.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

    Mar 18, 2026Kevin Qu, Haozhe Qi, Mihai Dusmanu +33D VLMs3D Spatial Reasoning

  2. VLM3: Vision Language Models Are Native 3D Learners

    May 28, 2026Zhipeng Cai, Zhuang Liu, Yunyang Xiong +3Camera Pose EstimationMonocular Depth Estimation