LiDAR Prompted Spatio-Temporal Multi-View Stereo for Autonomous Driving

Qihao Sun; Jiarun Liu; Ziqian Ni; Jianyun Xu; Tao Xie; Lijun Zhao; Ruifeng Li; Sheng Yang

cs.CVMar 4, 2026

LiDAR Prompted Spatio-Temporal Multi-View Stereo for Autonomous Driving

Authors: Qihao Sun, Jiarun Liu, Ziqian Ni, Jianyun Xu, Tao Xie, Lijun Zhao, Ruifeng Li, Sheng Yang

Organizations: Unmanned Vehicle Dept., CaiNiao Inc., Alibaba Group · Harbin Institute of Technology

Abstract

Accurate metric depth is critical for autonomous driving perception and simulation, yet current approaches struggle to achieve high metric accuracy, multi-view and temporal consistency, and cross-domain generalization. To address these challenges, we present DriveMVS, a novel multi-view stereo framework that reconciles these competing objectives through two key insights: (1) Sparse but metrically accurate LiDAR observations can serve as geometric prompts to anchor depth estimation in absolute scale, and (2) deep fusion of diverse cues is essential for resolving ambiguities and enhancing robustness, while a spatio-temporal decoder ensures consistency across frames. Built upon these principles, DriveMVS embeds the LiDAR prompt in two ways: as a hard geometric prior that anchors the cost volume, and as soft feature-wise guidance fused by a triple-cue combiner. Regarding temporal consistency, DriveMVS employs a spatio-temporal decoder that jointly leverages geometric cues from the MVS cost volume and temporal context from neighboring frames. Experiments show that DriveMVS achieves state-of-the-art performance on multiple benchmarks, excelling in metric accuracy, temporal stability, and zero-shot cross-domain transfer, demonstrating its practical value for scalable, reliable autonomous driving systems.

Figures & tables

Figure 1: Overview of MVS-Pro. We introduce a Prompt-Anchored Cost Volume ( Sec. 3.2 ) mechanism to fuse the absolute depth metric prompt into the multi-view cost volume. To fuse heterogeneous features from images, depth, and cost volume, we propose a Triple-Cues Combiner ( Sec. 3.3 ) to combine the cues. Finally, a Spatio-Temporal Decoder ( Sec. 3.4 ) produces continuous, consistent depth results.

Split	Name	Scenes	# Total	# Total	Sparse
			scenes	images	prompt
Training	TartanAir [ 62 ]	Indoor, Outdoor	347	1M	Synthetic
	TartanGround [ 43 ]	Indoor, Outdoor	789	1M
	VKITTI2 [ 4 , 16 ]	Outdoor, Driving	50	21K
	MVS-Synth [ 25 ]	Outdoor, Driving	117	12K
Testing	KITTI [ 17 ]	Outdoor, Driving	61	42K	LiDAR

Table 1: We train on four MVS datasets from a variety of domains. All these datasets are synthetically rendered, providing perfect ground-truth depths and camera calibration. All the sparse prompts are synthesized directly from the ground-truth depths.

Method	Venue	KITTI			DDAD			Waymo
Method	Venue	MAE ( $\downarrow$ )	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	MAE ( $\downarrow$ )	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	MAE ( $\downarrow$ )	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )
a) Feed-forward reconstruction
VGGT [ 58 ]	CVPR’25	$13.19$	$77.58$	$2.21$	$31.30$	$98.17$	$0.00$	-	-	-
MapAnything [ 30 ]	arXiv’25	$1.45$	$10.14$	$94.45$	$11.05$	$40.57$	$2.82$	-	-	-
MapAnything † [ 30 ]	arXiv’25	$1.27$	$8.45$	$95.28$	$4.60$	$13.57$	$81.06$	-	-	-
b) Monocular depth (w/o prompts)

Table 2: Benchmark study of depth estimation methods. All methods are evaluated on the official validation splits of KITTI [ 17 ] , DDAD [ 19 ] , and Waymo [ 54 ] . Each dataset reports three metrics. MAE is measured in meters (m), while AbsRel and

\tau

are reported as percentages (%). The best and second best scores are highlighted in bold and underline .

Figure 2: The qualitative results of the estimated depth by different methods on KITTI [ 17 ] (row 1 & 2), DDAD [ 19 ] (row 3) and Waymo [ 54 ] (row 4 & 5). The best and second best are highlighted with green and yellow borders, respectively. Please check the red boxes in the figure for a detailed comparison.

Method	Venue	KITTI
Method	Venue	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	TAE ( $\downarrow$ )
VideoDA-B [ 8 ]	CVPR’25	16.64	83.17	0.767
MVSAnywhere [ 26 ]	CVPR’25	10.37	91.05	0.338
Ours	-	2.56	98.78	0.296

Table 3: Comparison of temporal consistency on KITTI [ 17 ] for video depth estimation . All scores are given in percentage (%). The best and the second best results are highlighted in bold and underline .

#	PACV	TCC	STD	$\mathcal{L}_{\text{s}}$	$\mathcal{L}_{\text{t}}$	KITTI
#	PACV	TCC	STD	$\mathcal{L}_{\text{s}}$	$\mathcal{L}_{\text{t}}$	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	TAE ( $\downarrow$ )
1				✓		$10.37$	$91.05$	$0.338$
2	✓			✓		$4.86$	$94.21$	$0.338$
3	✓	✓		✓		$2.76$	$98.72$	$0.338$
4			✓	✓	✓	$10.18$	$91.30$	$0.297$
5	✓	✓	✓	✓		$2.58$	$98.76$	$0.335$

Table 4: Ablation studies on: [left] Prompt-Anchored Cost Volume (PACV), Triple-Cues Combiner (TCC), Spatio-Temporal Decoder (STD), and [middle] temporal loss.

Method	Rainy		Dark		Static
Method	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )
DepthPro [ 2 ]	$34.96$	$19.30$	$43.63$	$13.43$	$34.03$	$11.23$
PromptDA [ 36 ]	$86.15$	$6.44$	$81.60$	$10.06$	$88.15$	$2.46$
PriorDA [ 64 ]	$11.92$	$86.56$	$11.25$	$83.59$	$6.78$	$93.30$
MVSAnywhere [ 26 ]	$18.48$	$82.85$	$16.97$	$79.74$	$55.56$	$43.23$
Ours	7.21	94.02	4.97	94.84	4.93	95.56

Table 5: Ablation studies on extreme cases, including adverse weather, low-lighting, and ego-static situations. The best and second best scores are highlighted in bold and underline .

Figure 3: Visualization of depth estimation on a static scene. The result demonstrates our robustness in challenging, low-parallax scenarios.

Figure 4: Ablations on (a) different LiDAR laser beams and (b) different LiDAR occlusion rate (from bottom). Lower AbsRel refers to a better result.

Figure 5: Ablation study on Lidar prompt absence. We estimate the depth of the back-view image while only the front-view image is provided by LiDAR. Our method remains an accurate metric under such a configuration.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Illustrations of spatio-temporal decoder. Please refer to Appendix A for more details.

Metric	Definition
MAE	$\frac{1}{N}\sum_{i=1}^{N}\|D_{gt}-\hat{D}\|$
AbsRel	$\frac{1}{N}\sum_{i=1}^{N}\|D_{gt}-\hat{D}\|/D_{gt}$
$\tau$	$\frac{1}{N}\sum_{i=1}^{N}\left(\max\left(\frac{\hat{D}}{D_{gt}},\frac{D_{gt}}{\hat{D}}\right)<1.25\right)$

Appendix

Table 6: Depth metric definitions.

D_{gt}

and

\hat{D}

are the ground-truth and predicted depth, respectively.

Case	Scene ID	# Total
Case	Scene ID	Images
Static	2011_09_29_drive_0026_sync	148
	13941626351027979229	199
	14127943473592757944	197
Rainy	10448102132863604198	183
	11356601648124485814	199

Appendix

Table 7: Extreme Cases IDs

Figure 7: Qualitative comparison of 3D reconstruction results on KITTI [ 17 ] . Rows show different methods: our DriveMVS model, MVSA [ 26 ] , and PriorDA [ 64 ] . (a) : Reconstructed colored point clouds of the driving scene. (b) : Zoomed-in views highlighting fine-grained details of vegetation and structural boundaries. (c) : Corresponding $z$ -axis visualizations.

#	Time (ms)	Memory (GB)
PriorDA	$246.79$	$2.45$
PromptDA	$300.60$	$3.75$
MVSA	$38.84$	$4.02$
Ours	$65.61$	$4.47$

Appendix

Table 8: Speed and memory consumption. Number benchmarked on an A100 GPU.

Figure 8: Qualitative comparison of depth prediction results on KITTI [ 17 ] . Rows show different methods: our DriveMVS model, MVSA [ 26 ] , PriorDA [ 64 ] , MapAnything [ 30 ] , and PromptDA [ 36 ] , along with RGB and prompt inputs ( $\boldsymbol{I}_{r}$ , $\boldsymbol{P}_{r}$ ). [left] : 2011_09_26_drive_0095_sync . [Middle] : 2011_09_26_drive_0023_sync . [Right] : 2011_09_26_drive_0002_sync . The best and second best are highlighted with green and yellow borders, respectively.

Figure 9: Qualitative comparison of depth prediction errors on KITTI [ 17 ] , corresponding to Fig. 8 . Rows show different methods: our DriveMVS model, MVSA [ 26 ] , PriorDA [ 64 ] , MapAnything [ 30 ] , and PromptDA [ 36 ] , along with RGB inputs ( $\boldsymbol{I}_{r}$ ) and prompt ( $\boldsymbol{P}_{r}$ ).

Figure 10: Qualitative comparison of depth prediction results on DDAD [ 19 ] and Waymo [ 54 ] . Rows show different methods: our DriveMVS model, MVSA [ 26 ] , PriorDA [ 64 ] , MapAnything [ 30 ] , and PromptDA [ 36 ] , along with RGB and prompt inputs ( $\boldsymbol{I_{r}}$ , $\boldsymbol{P_{r}}$ ). For left to right: 000156 , 000155 , 000194 , 1024360143612057520 , 12866817684252793621 . The best and second best are highlighted with green and yellow borders, respectively.

Figure 11: Qualitative comparison of depth prediction errors on DDAD [ 17 ] and Waymo [ 54 ] , corresponding to Fig. 10 . Rows show different methods: our DriveMVS model, MVSA [ 26 ] , PriorDA [ 64 ] , MapAnything [ 30 ] , and PromptDA [ 36 ] , along with RGB inputs ( $\boldsymbol{I}_{r}$ ) and prompt ( $\boldsymbol{P}_{r}$ ).

Figure 12: Qualitative comparison of depth prediction results and depth prediction errors across extreme cases for autonomous driving in our sampled dataset (most Waymo [ 54 ] ). Columns show different methods: our DriveMVS model, MVSAnywhere [ 26 ] , and PriorDA [ 64 ] , along with RGB & prompt inputs ( $\boldsymbol{I}_{r},\boldsymbol{P}_{r}$ ) and ground-truth depths (GT). [Top] : 11356601648124485814 . [Middle] : 14107757919671295130 . [Bottom] : 14127943474592757944 .

Multi-View Depth Estimation

Multi-view depth estimation recovers consistent 3D structure from multiple images using camera geometry, learned priors, and cross-view evidence.

See all 23 papers

Jun 30, 2026

DrivingDepth: Sparse-Prompted Pixel-wise Scale Correction for Driving Depth Estimation

Dense depth estimation for autonomous driving faces a geometry-scale conflict: depth foundation models deliver pixel-aligned dense visual geometry without reliable metric scale, while projected LiDAR provides metric anchors that are sparse, noisy, and misaligned with image structures. Existing sparse-prompted methods incorporate LiDAR by regenerating depth from scratch, overriding the foundation model's coherent geometry and producing structural artifacts on visually continuous surfaces. Our key insight is that foundation models already capture geometrically coherent relative depth; no additional surface structure learning is required-only a per-pixel scale factor mapping relative geometry to metric coordinates. Based on this, we propose DrivingDepth, which treats sparse LiDAR as geometric prompts that locally calibrate a frozen foundation prior through residual pixel-wise scale correction, preserving dense visual geometry by construction. On nuScenes with 4-frame surround-view input, DrivingDepth achieves an AbsRel of 11.19 and an EdgeCR of 5.741, outperforming MapAnything (11.99/1.914) by simultaneously delivering SOTA metric accuracy and geometric consistency.

Chi Huang, Wenhao Zhang, Hang Yin +5

Baidu Inc.

Metric Depth Estimation Depth Estimation Camera-LiDAR Fusion Autonomous Driving Perception

Ask PDF

Jun 15, 2026

SurroundNEXO: Ego-Centric Metric Bridging for Spatially Consistent Geometry in Autonomous Driving

Modern autonomous driving depends on accurate metric 3D understanding for perception, reconstruction, and planning, which in turn requires reliable multi-camera depth prediction. However, the outward-facing nature of vehicle-mounted surround-view camera rigs inherently limits visual overlap across views, challenging the correspondence-based assumptions that underpin conventional multi-view geometry. To bridge this gap, we present SurroundNEXO, named after the Spanish word nexo for a geometric link, a low-overlap multi-camera metric depth framework that grounds cross-view reasoning in ego-centric geometry rather than dense visual correspondences. Instead of directly enforcing early global fusion, SurroundNEXO first assigns image tokens globally comparable ego-frame viewing directions through Ego-Ray Positional Encoding, then uses sparse LiDAR measurements as metric anchors to propagate absolute scale cues, and finally expands feature interaction progressively from view-local modeling to decomposed spatio-temporal reasoning and global integration. This design enables metric-scale depth prediction with improved spatial consistency across weakly overlapping cameras. Across low-overlap autonomous driving benchmarks, including NuScenes, Waymo and DDAD, SurroundNEXO reduces single-view error by 33.2%, improves cross-view consistency by 10.5%, and enhances metric reconstruction quality by 25.6% compared with SOTA methods. It further remains robust under extremely sparse depth prompts and exhibits strong zero-shot generalization to unseen camera layouts.

Shuai Yuan, Runxi Tang, Yuzhou Ji +8

School of Artificial Intelligence, Shanghai Jiao Tong University · 2Hello Inc.

Multi-View Depth Estimation Multi-View Consistency Metric Depth Estimation Depth Estimation Camera-LiDAR Fusion Autonomous Driving Perception

Ask PDF

Jul 21, 2026

D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models

Recent advances in Multimodal Large Language Models (MLLMs) have triggered the development of end-to-end MLLMs for autonomous driving. However, the main emphasis to date has been for MLLMs using 2D images and videos. In contrast, this paper considers MLLM effectiveness using 3D sensors, particularly LiDAR and stereo cameras. LiDAR presents unique challenges to integration within an MLLM, largely because of data sparsity and lack of a grid structure for the data. For similar reasons, fusion of camera and LiDAR data within an MLLM pipeline is also uncommon. However, most autonomous systems rely on LiDAR-based sensing, and incorporating 3D data has been proven to improve performance in traditional 3D scene perception tasks. This paper presents D3VL, a novel MLLM framework that integrates 2D and 3D time-series data in a single but simple architecture. The model aims to answer questions involving traffic scene understanding and safety. D3VL shows an 11% improvement in the KITTI Question-Answering (QA) dataset compared to baseline methods in processing 2D and 3D time-series data. This paper further introduces the Waymo QA dataset extension, which assesses models' capabilities in processing 3D and time-series data under diverse driving conditions. D3VL implementation code and WaymoQA extension can be found on our supplemental website: https://automotivesafety-lvlm.github.io

Heesang Han, A. Lynn Abbott, Abhijit Sarkar

Bradley Department of Electrical and Computer Engineering, Virginia Tech, USA · Sanghani Center for Artificial Intelligence and Data Analytics, USA · Virginia Tech Transportation Institute, USA

VLMs for Autonomous Driving Camera-LiDAR Fusion Question Answering 3D VLMs

Ask PDF

Abstract

Multi-View Depth Estimation

Explore similar work

DrivingDepth: Sparse-Prompted Pixel-wise Scale Correction for Driving Depth Estimation

SurroundNEXO: Ego-Centric Metric Bridging for Spatially Consistent Geometry in Autonomous Driving

D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models