Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

Kevin Qu; Haozhe Qi; Mihai Dusmanu; Mahdi Rad; Rui Wang; Marc Pollefeys

cs.CVMar 18, 2026

Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

Authors: Kevin Qu, Haozhe Qi, Mihai Dusmanu, Mahdi Rad, Rui Wang, Marc Pollefeys

Organizations: Microsoft Spatial AI Lab · ETH Zurich · EPFL

Abstract

Multimodal Large Language Models (MLLMs) have made impressive progress in connecting vision and language, but they still struggle with spatial understanding and viewpoint-aware reasoning. Recent efforts aim to augment the input representations with geometric cues rather than explicitly teaching models to reason in 3D space. We introduce Loc3R-VLM, a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input. Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction to build a holistic representation of the scene structure, and explicit situation modeling to anchor egocentric perspective. These objectives provide direct spatial supervision that grounds both perception and language in a 3D context. To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model. Loc3R-VLM achieves state-of-the-art performance in language-based localization and outperforms existing 2D- and video-based approaches on situated and general 3D question-answering benchmarks, demonstrating that our spatial supervision framework enables strong 3D understanding. Project page: https://kevinqu7.github.io/loc3r-vlm

Figures & tables

Figure 2: Overview of Loc3R-VLM . Our framework takes a monocular video as input and augments the vision token sequence with per-frame latent camera pose priors extracted from the 3D foundation model CUT3R [ 55 ] . The model is jointly trained using two spatial objectives: (1) layout reconstruction, which grounds vision patch tokens into a bird’s-eye-view (BEV) space to capture global scene structure, and (2) situation modeling, which utilizes dedicated localization query tokens to localize an agent from a situation description. During answer generation, the model leverages the inferred layout and location to perform viewpoint-aware 3D reasoning.

Figure 3: Spatial Supervision Framework introduces complementary training signals. For the layout reconstruction objective, the model learns to ground each vision patch token onto its corresponding BEV coordinate in a cognitive map to capture global scene structure. For localization, dedicated localization tokens explicitly model the agent’s position and orientation. The framework is trained end-to-end using a joint objective of layout, localization, and language losses.

Figure 4: Qualitative Results for language-based localization and situated QA on SQA3D [ 34 ] . Loc3R-VLM accurately grounds the described situations ( blue : prediction, green : ground truth) and provides the correct viewpoint-dependent answer. Meshes are shown for visualization only and are not used by the model.

Method	3D	2D	Localization		Orientation
Method	3D	2D	[email protected]	[email protected]	Acc@15 ∘	Acc@30 ∘
Random	–	–	7.2	25.8	8.4	16.9
SQA3D [ 34 ]	✓	–	9.5	29.6	8.7	16.5
SQA3D [ 34 ] (separate)	✓	–	10.3	31.4	17.1	22.8
3D-VisTA [ 76 ]	✓	–	11.7	34.5	16.9	24.2
SIG3D † [ 37 ]	✓	–	16.8	35.2	23.4	26.3

Table 1: Evaluation of Language-based Localization on SQA3D [ 34 ] . † indicates results reproduced by [ 63 ] . Separate means disabling other tasks to let the model focus on situation estimation only. Loc3R-VLM achieves state-of-the-art performance by a large margin and outperforms methods which operate on dense 3D point cloud input.

Model	Avg.	Numerical				Multiple-Choice
Model	Avg.	Obj. Count	Abs. Dist.	Obj. Size	Room Size	Rel. Dist.	Rel. Dir.	Route Plan	Appr. Order
Expert Models
VLM-3R [ 15 ]	60.9	70.2	49.4	69.2	67.1	65.4	80.5	45.4	40.1
2D MLLMs
GPT-4o [ 40 ]	34.0	46.2	5.3	43.8	38.2	37.0	41.3	31.5	28.5
Gemini-1.5-Pro [ 44 ]	45.4	56.2	30.9	64.1	43.6	51.3	46.3	36.0	34.6

Table 2: Evaluation of 3D QA on VSI-Bench [ 61 ] . Loc3R-VLM sets a new state-of-the-art and particularly excels at viewpoint-dependent subcategories.

Model	SQA3D ${}_{\text{test}}$		ScanQA ${}_{\text{val}}$
Model	EM	EM-R	CIDEr	METEOR	ROUGE	EM
Expert Models
SQA3D [ 34 ]	46.6	–	–	–	–	–
ScanQA [ 2 ]	–	–	64.9	13.1	33.3	21.1
3D MLLMs
LEO [ 22 ]	50.0	52.4	80.0	16.2	39.3	21.5

Table 3: Evaluation of 3D QA on SQA3D [ 34 ] and ScanQA [ 2 ] . Loc3R-VLM achieves the best performance among 2D MLLMs on both benchmarks. Our method also outperforms most 3D MLLMs on SQA3D, demonstrating strong situated reasoning.

Table 7

	Method	[email protected]	[email protected]	Acc@15°	Acc@30°
1	Situation	27.0	51.5	26.7	48.7
2	\scriptsize{1}⃝ + Layout	30.1	59.3	28.2	53.2
3	\scriptsize{1}⃝ + \scriptsize{2}⃝ + Cam	39.9	75.5	31.9	56.3

Table 6: Ablations on Language-based Localization on SQA3D [ 34 ] . Our full model achieves the best performance.

	Method	VSI-Bench	ScanQA	SQA3D	MSQA
1	LLaVA FT	49.9	92.2	58.4	54.4
2	\scriptsize{1}⃝ + Situation	50.6	98.4	59.2	55.1
3	\scriptsize{1}⃝ + Layout	50.3	99.7	58.9	55.5
4	\scriptsize{1}⃝ + \scriptsize{2}⃝ + \scriptsize{3}⃝	53.6	104.3	59.6	56.0
5	\scriptsize{4}⃝ + Cam	54.3	107.3	59.9	56.6

Table 7: Ablations on 3D QA . Consistent gains from each component validate their individual utility, while the full model’s peak performance confirms the complementary nature and effectiveness of our framework.

3D Features	VSI-Bench	SQA3D
3D Features	Avg.	EM	Acc@1m	Acc@30 ∘
Cam. + Geom.	59.5	59.0	71.8	59.8
Cam. (Ours)	63.2	62.8	75.9	63.0

Table 8: Ablations on 3D Foundation Model Feature Selection. Integrating only the CUT3R camera token as a spatial prior proves superior to including both camera and geometry tokens.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Source	# samples	Datasets	Question Length	Answer Length
ScanQA [ 2 ]	26,515	ScanNet	8.7	2.4
SQA3D [ 34 ]	79,445	ScanNet	28.0	1.1
MSQA [ 32 ]	49,747	ScanNet	33.8	8.5
VSI-Bench [ 61 ]	9,969	ScanNet, ScanNet++, ARKitScenes	39.4	1.0
VSI-Bench (VLM-3R) [ 15 ]	106,890	ScanNet, ScanNet++, ARKitScenes	47.9	1.0

Appendix

Table 0.A.1 : Detailed training data statistics.

Method	VSI-Bench	SQA3D
Method	Avg.	EM-R	Acc@1m	Acc@30 ∘
Ours w/ CUT3R	53.0	63.5	74.2	57.1
Ours w/ VGGT	54.2	63.9	74.4	59.4

Appendix

Table 0.B.1 : Evaluation with different 3D Foundation Model. Replacing our default CUT3R encoder with VGGT yields similar performance, demonstrating that our framework is not dependent on a specific 3D representation backbone.

Method	Latency (s)		Peak VRAM (GB)
Method	CUT3R Enc.	Total	Peak VRAM (GB)
LLaVA-Video-7B	–	1.3	19.0
Ours	1.2	2.6	20.3

Appendix

Table 0.C.1 : Efficiency Analysis. Inference latency (TTFT) and peak VRAM usage for 32 input frames on a single RTX 4090 (fp16). The CUT3R encoder introduces a one-time per-video encoding cost of 1.2s and a modest increase of +6.8% in peak VRAM. Once computed, CUT3R tokens can be cached and reused across multiple queries for the same video without incurring additional encoding latency.

Figure 0.D.1 : Correlation Between Localization Accuracy, QA Performance, and Predicted Uncertainty in Loc3R-VLM on SQA3D [ 34 ] . Localization is considered accurate for a position error

\leq 1.0

m and an orientation error

\leq 45^{\circ}

. Left : QA accuracy (EM-R) is substantially higher when Loc3R-VLM correctly localizes the agent, demonstrating that its learned situation representation enables effective viewpoint-aware reasoning. Right : The predicted positional uncertainty

\sigma_{\text{pos}}

is notably higher for instances where localization is inaccurate, demonstrating that the model’s uncertainty estimates can reflect the reliability of the predicted situation.

Mean Error (m)	RMSE (m)	[email protected]	[email protected]	[email protected]
0.40	0.48	40.6	74.8	94.4

Appendix

Table 0.F.1 : Evaluation of Global Layout Reconstruction on 67 unseen ScanNet [ 11 ] test scenes. Loc3R-VLM is able to ground image patches with high spatial accuracy in the BEV space.

Layout Representation	ScanQA	MSQA	SQA3D
Layout Representation	CIDEr	Overall	EM-R	[email protected]	Acc@30 ∘
3D	101.1	58.3	62.0	77.2	55.4
2D BEV (Ours)	101.7	58.6	63.5	74.2	57.1

Appendix

Table 0.F.2 : Ablation on Layout Representation. We compare our 2D BEV representation for global layout reconstruction against directly predicting 3D coordinates. The BEV-based model achieves stronger QA performance while maintaining comparable localization accuracy.

Frame Count	VSI-Bench	SQA3D
Frame Count	Avg.	EM-R	Acc@1m	Acc@30 ∘
16	60.3	64.1	70.7	59.8
32	63.2	65.0	75.9	63.0
40	63.3	64.8	75.5	62.6

Appendix

Table 0.G.1 : Evaluation for Varying Input Frame Count. The model maintains stability across different numbers of input frames.

Panel of Figure 0 — Figure 0.J.1 : Qualitative Examples of language-based localization and situated question answering on SQA3D [ 34 ] ( blue : predicted situation, green : ground-truth situation).

3D VLMs

Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.

See all 204 papers

Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning
Feb 24, 2026Haoyi Jiang, Liu Liu, Xinjie Wang +53D Spatial Reasoning 3D VLMs
School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · Horizon Robotics, Beijing 100094, China · D-Robotics, Shenzhen 518055, China
Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning
May 28, 2026Chun-Hsiao Yeh, Shengyi Qian, Manchen Wang +3Vision-Language Models 3D Spatial Reasoning
FAIR at Meta · UC Berkeley · HKU
3D-Aware VLMs with Implicit and Explicit Geometries
Jul 23, 2026Wenhao Li, Xueying Jiang, Quanhao Qian +43D Spatial Reasoning 3D Object Detection
1Nanyang Technological University · 2DAMO Academy, Alibaba Group · 3HuPan Lab +1

Feb 24, 2026J/K move · Enter open · S save

Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning

Haoyi Jiang, Liu Liu, Xinjie Wang, Yonghao He+4

School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China · Horizon Robotics, Beijing 100094, China · D-Robotics, Shenzhen 518055, China

3D Spatial Reasoning 3D VLMs 3D Scene Representation 3D Representation Learning Self-Supervised Learning

Open paper AskPDF arXiv

Vision-language models excel at 2D visual understanding but remain limited in 3D spatial reasoning. Existing approaches either depend on explicit 3D modalities, which limits scalability, or inject partial, view-conditioned geometric priors and leave the language model to recover global scene structure from sparse cues. We introduce Spa3R, a self-supervised framework that learns a unified, view-invariant spatial representation from unposed multi-view RGB images. Its Predictive Spatial Field Modeling objective compresses context views into a compact latent representation and predicts aligned geometric and semantic feature fields at novel viewpoints, thereby encouraging coherent encoding of scene geometry and layout. We integrate the pre-trained Spa3R Encoder into a vision-language model through a lightweight residual cross-attention adapter, yielding Spa3-VLM and grounding language reasoning in global spatial context. Spa3-VLM achieves an average score of 58.6% on VSI-Bench and delivers leading or competitive performance across three additional spatial reasoning benchmarks. These results demonstrate that predictive spatial representation learning provides an effective visual foundation for 3D reasoning.

Abstract

3D VLMs

Explore similar work

Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning

Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning

3D-Aware VLMs with Implicit and Explicit Geometries