Retrieve-to-Localize: Bridging Large Language Models and LiDAR Geometry for Spatial Grounding

Byounggun Park; Giyong Moon; Jusung Kim; Soonmin Hwang

cs.CVSep 24, 2026

Retrieve-to-Localize: Bridging Large Language Models and LiDAR Geometry for Spatial Grounding

Authors: Byounggun Park, Giyong Moon, Jusung Kim, Soonmin Hwang

Organizations: Department of Automotive Engineering (Automotive-Computer Convergence), Hanyang University, Seoul, South Korea. · Department of Automotive Engineering, Hanyang University, Seoul, South Korea.

Abstract

LiDAR provides precise geometric information for spatial perception tasks such as object detection in autonomous driving and outdoor robotics. However, recognizing and localizing individual objects is not sufficient to answer questions that require composing spatial relations and grounding the intended target. Motivated by recent advances in large language models (LLMs) for autonomous driving, we leverage their language priors to interpret complex spatial questions and ground the referred target in LiDAR geometry. To support this spatial grounding capability, we introduce SpatialLiDAR-QA, which combines single- and multi-step relational grounding with complementary spatial understanding tasks. We further propose SpatialLiDAR-LM, which aligns LiDAR point features with an LLM and grounds target coordinates through language-conditioned, position-aware proposal retrieval and local point refinement. This design derives target coordinates directly from local LiDAR geometry rather than through textual language decoding. Experiments demonstrate substantial improvements over representative LiDAR--language models and multi-camera VLMs on precise coordinate prediction tasks. Our dataset and model training code will be publicly released.

Figures & tables

Fig. 1: LiDAR provides precise geometry for object detection, but spatial questions require more than localizing individual objects: relations must be composed to ground the intended target.

Fig. 2: Overview of the five spatial grounding tasks in SpatialLiDAR-QA: Direct Localization, Object Identification, Single/Multi-Step Relational Grounding, and Maneuver Feasibility, all formulated from LiDAR observations.

Task Category	Training	Validation	Total
Localization	24,586	5,273	29,859
Object Identification	5,998	1,303	7,301
Single-step Relational	27,050	5,826	32,876
Multi-step Relational	8,152	2,400	10,552
Maneuver Feasibility	23,762	4,461	28,223
Total	89,548	19,263	108,811

TABLE I: Distribution of QA pairs in SpatialLiDAR-QA.

Fig. 3: Overview of SpatialLiDAR-LM . LiDAR point tokens and questions are processed in a shared LLM embedding space. For coordinate prediction, PRL constructs spatial proposals through proposal scoring and $xy$ -NMS, retrieves a question-relevant proposal using the retrieval query and position-aware proposal keys, and refines its LiDAR-frame coordinate from local point features.

Fig. 4: Point-Retrieved Localization (PRL) on a validation example. The retrieval query first selects a question-relevant spatial proposal using proposal-level attention $\alpha_{j}$ , after which local attention $\beta_{i}$ over the point tokens in $\mathcal{N}_{\mathrm{ref}}$ refines the final coordinate. The red box denotes the ground-truth bounding box.

Base LLM	Method	Modality	Localization			Single-Step			Multi-Step
			@0.5m	@1.0m	@2.0m	@0.5m	@1.0m	@2.0m	@0.5m	@1.0m	@2.0m
SmolVLM	SmolVLM-500M-Instruct	Multi-Camera	14.9	30.5	52.3	8.2	24.5	51.5	3.9	11.7	26.8
	BEV-LLM [ 14 ]	Multi-Camera	2.4	4.1	6.5	8.5	23.5	49.1	3.8	11.2	25.3
	BEV-LLM [ 14 ]	LiDAR	2.6	4.3	6.5	8.1	24.0	49.2	3.4	12.2	24.1
	LiDAR-LLM [ 9 ]	LiDAR	0.5	1.6	5.3	3.1	11.5	34.7	0.9	5.8	21.8
	SpatialLiDAR-LM (Language CE)	LiDAR	6.7	15.6	31.9	3.6	9.3	22.4	2.2	8.7	20.0

TABLE II: Quantitative results (accuracy, %, within

0.5

1.0

, and

2.0\,\text{m}

) on Coordinate prediction tasks. Language CE predicts coordinates through language decoding instead of PRL; best results per base LLM are in bold .

Base LLM	Task	Threshold	Language CE	Smooth L1	PRL	PRL + Key PE
SmolVLM	Localization	@0.5m	6.7	1.0	54.8	57.2
		@1.0m	15.6	4.1	76.5	77.0
		@2.0m	31.9	14.7	84.6	85.0
	Single-Step	@0.5m	3.6	5.9	14.5	28.4
		@1.0m	9.3	16.9	25.7	46.3
		@2.0m	22.4	42.8	38.7	59.9

TABLE III: Ablation of the grounding mechanism and Key PE. Accuracy (%) is reported within

0.5

1.0

, and

2.0\,\text{m}

of the ground-truth location.

Base LLM	Method	Modality	Maneuver	Object Identification	Single-Step	Multi-Step
SmolVLM	SmolVLM-500M-Instruct	C	89.7	62.2	64.0	71.5
	BEV-LLM [ 14 ]	C	60.6	24.0	58.8	70.4
	BEV-LLM [ 14 ]	L	60.7	26.0	59.0	70.8
	LiDAR-LLM [ 9 ]	L	69.0	16.0	57.0	64.6
	SpatialLiDAR-LM (Ours)	L	87.2	38.0	59.4	71.8
Qwen3.5	Qwen3.5-0.8B	C	92.2	77.1	75.9	80.0

TABLE IV: Quantitative results (exact-match accuracy, %) on Class prediction tasks and Maneuver Feasibility. C and L denote multi-camera and LiDAR.

Eval. dataset	Model	Localization			Single-Step			Multi-Step
Eval. dataset	Model	@0.5	@1.0	@2.0	@0.5	@1.0	@2.0	@0.5	@1.0	@2.0
nuScenes (source)	Ours	57.6	78.2	85.4	34.7	56.2	71.4	44.1	62.9	74.0
nuScenes (source)	VLM	38.5	63.1	79.3	12.4	32.5	58.9	8.5	21.5	42.4
AV2 [ 29 ]	Ours	47.1	67.8	76.2	36.3	55.4	65.3	31.6	50.4	57.9
AV2 [ 29 ]	VLM	12.1	28.9	54.6	7.6	20.7	44.2	1.7	6.7	23.6
NAVSIM [ 30 ]	Ours	67.8	77.3	79.9	43.5	57.4	63.2	41.4	54.4	59.5

TABLE V: Cross-dataset grounding accuracy (%). Both models use Qwen3.5-0.8B and are fine-tuned on nuScenes-based SpatialLiDAR-QA; evaluation on AV2 and NAVSIM is zero-shot. Distance thresholds are in meters.

Explore similar work

Do LiDAR Language Models Really Understand Spatio-temporal Relationships?
Sep 21, 2026Runyi Yang, Murat Akkoyun, Di Wen +8Spatio-Temporal Reasoning Lidar
INSAIT, Sofia University “St. Kliment Ohridski”, Bulgaria · Karlsruhe Institute of Technology, Germany · Hunan University, China
OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs
Jun 21, 2026Hao Vo, Phu Loc Nguyen, Khoa Vo +7Multimodal Large Language Models Efficient Planning
University of Arkansas, USA · Google Research, Google · University of Liverpool, UK +1
SSR3D-LLM: Structured Spatial Reasoning via Latent Steps for Fine-Grained Grounding in Unified 3D-LLMs
May 27, 2026Jiawei Li, Ziyi Liu, Weijie Shi +33D Visual Grounding 3D Spatial Reasoning
The Hong Kong University of Science and Technology · Soochow University

Sep 21, 2026, cs.CVJ/K move · Enter open · S save

Do LiDAR Language Models Really Understand Spatio-temporal Relationships?

Runyi Yang, Murat Akkoyun, Di Wen, Ruiping Liu+7

INSAIT, Sofia University “St. Kliment Ohridski”, Bulgaria · Karlsruhe Institute of Technology, Germany · Hunan University, China

Spatio-Temporal Reasoning Lidar Spatiotemporal

Open paper AskPDF arXiv

Recent 4D LiDAR language models aim to reason about objects and their evolving spatial relationships. Yet, in our evaluation, always selecting the same option nearly matches the multiple-choice accuracy of two B4DL-derived configurations. We introduce LiDAR-Hallu, a geometry-referenced benchmark and diagnostic protocol with 10,000 questions across 150 nuScenes scenes. It covers object existence, ego-relative position, distance ordering, relative motion, and temporal localization, with explicit rules for selecting objects, comparing times, and determining reference answers. Our protocol combines fixed-answer and candidate-content controls, cross-scene pairs with identical prompts but opposite reference answers, and relation-specific recall. Analysis of 100,000 recorded responses reveals failures hidden by aggregate accuracy. Candidate duration alone makes temporal answers predictable without observing LiDAR. On paired questions, the models frequently give the same answer to scenes requiring opposite answers. Relation-specific analysis further shows that both configurations miss every positive lateral-motion case across all tested conditions. Temporal-shuffle contrastive decoding provides little net improvement, as repairs are largely offset by new errors and the main failures persist. These results show that evaluating spatio-temporal reasoning requires testing whether models distinguish the queried physical relationships, rather than relying on individual-answer accuracy alone. The source code, checkpoints, and data are released at https://github.com/Awesome4D/4DMLLM_Hallucination_Bench.

Abstract

Explore similar work

Do LiDAR Language Models Really Understand Spatio-temporal Relationships?

OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs

SSR3D-LLM: Structured Spatial Reasoning via Latent Steps for Fine-Grained Grounding in Unified 3D-LLMs