LiDAR Prompted Spatio-Temporal Multi-View Stereo for Autonomous Driving

Qihao Sun; Jiarun Liu; Ziqian Ni; Jianyun Xu; Tao Xie; Lijun Zhao; Ruifeng Li; Sheng Yang

cs.CVMar 4, 2026

LiDAR Prompted Spatio-Temporal Multi-View Stereo for Autonomous Driving

Authors: Qihao Sun, Jiarun Liu, Ziqian Ni, Jianyun Xu, Tao Xie, Lijun Zhao, Ruifeng Li, Sheng Yang

Organizations: Unmanned Vehicle Dept., CaiNiao Inc., Alibaba Group · Harbin Institute of Technology

Abstract

Accurate metric depth is critical for autonomous driving perception and simulation, yet current approaches struggle to achieve high metric accuracy, multi-view and temporal consistency, and cross-domain generalization. To address these challenges, we present DriveMVS, a novel multi-view stereo framework that reconciles these competing objectives through two key insights: (1) Sparse but metrically accurate LiDAR observations can serve as geometric prompts to anchor depth estimation in absolute scale, and (2) deep fusion of diverse cues is essential for resolving ambiguities and enhancing robustness, while a spatio-temporal decoder ensures consistency across frames. Built upon these principles, DriveMVS embeds the LiDAR prompt in two ways: as a hard geometric prior that anchors the cost volume, and as soft feature-wise guidance fused by a triple-cue combiner. Regarding temporal consistency, DriveMVS employs a spatio-temporal decoder that jointly leverages geometric cues from the MVS cost volume and temporal context from neighboring frames. Experiments show that DriveMVS achieves state-of-the-art performance on multiple benchmarks, excelling in metric accuracy, temporal stability, and zero-shot cross-domain transfer, demonstrating its practical value for scalable, reliable autonomous driving systems.

Figures & tables

Figure 1: Overview of MVS-Pro. We introduce a Prompt-Anchored Cost Volume ( Sec. 3.2 ) mechanism to fuse the absolute depth metric prompt into the multi-view cost volume. To fuse heterogeneous features from images, depth, and cost volume, we propose a Triple-Cues Combiner ( Sec. 3.3 ) to combine the cues. Finally, a Spatio-Temporal Decoder ( Sec. 3.4 ) produces continuous, consistent depth results.

Split	Name	Scenes	# Total	# Total	Sparse
			scenes	images	prompt
Training	TartanAir [ 62 ]	Indoor, Outdoor	347	1M	Synthetic
	TartanGround [ 43 ]	Indoor, Outdoor	789	1M
	VKITTI2 [ 4 , 16 ]	Outdoor, Driving	50	21K
	MVS-Synth [ 25 ]	Outdoor, Driving	117	12K
Testing	KITTI [ 17 ]	Outdoor, Driving	61	42K	LiDAR

Table 1: We train on four MVS datasets from a variety of domains. All these datasets are synthetically rendered, providing perfect ground-truth depths and camera calibration. All the sparse prompts are synthesized directly from the ground-truth depths.

Method	Venue	KITTI			DDAD			Waymo
Method	Venue	MAE ( $\downarrow$ )	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	MAE ( $\downarrow$ )	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	MAE ( $\downarrow$ )	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )
a) Feed-forward reconstruction
VGGT [ 58 ]	CVPR’25	$13.19$	$77.58$	$2.21$	$31.30$	$98.17$	$0.00$	-	-	-
MapAnything [ 30 ]	arXiv’25	$1.45$	$10.14$	$94.45$	$11.05$	$40.57$	$2.82$	-	-	-
MapAnything † [ 30 ]	arXiv’25	$1.27$	$8.45$	$95.28$	$4.60$	$13.57$	$81.06$	-	-	-
b) Monocular depth (w/o prompts)

Table 2: Benchmark study of depth estimation methods. All methods are evaluated on the official validation splits of KITTI [ 17 ] , DDAD [ 19 ] , and Waymo [ 54 ] . Each dataset reports three metrics. MAE is measured in meters (m), while AbsRel and

\tau

are reported as percentages (%). The best and second best scores are highlighted in bold and underline .

Figure 2: The qualitative results of the estimated depth by different methods on KITTI [ 17 ] (row 1 & 2), DDAD [ 19 ] (row 3) and Waymo [ 54 ] (row 4 & 5). The best and second best are highlighted with green and yellow borders, respectively. Please check the red boxes in the figure for a detailed comparison.

Method	Venue	KITTI
Method	Venue	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	TAE ( $\downarrow$ )
VideoDA-B [ 8 ]	CVPR’25	16.64	83.17	0.767
MVSAnywhere [ 26 ]	CVPR’25	10.37	91.05	0.338
Ours	-	2.56	98.78	0.296

Table 3: Comparison of temporal consistency on KITTI [ 17 ] for video depth estimation . All scores are given in percentage (%). The best and the second best results are highlighted in bold and underline .

#	PACV	TCC	STD	$\mathcal{L}_{\text{s}}$	$\mathcal{L}_{\text{t}}$	KITTI
#	PACV	TCC	STD	$\mathcal{L}_{\text{s}}$	$\mathcal{L}_{\text{t}}$	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	TAE ( $\downarrow$ )
1				✓		$10.37$	$91.05$	$0.338$
2	✓			✓		$4.86$	$94.21$	$0.338$
3	✓	✓		✓		$2.76$	$98.72$	$0.338$
4			✓	✓	✓	$10.18$	$91.30$	$0.297$
5	✓	✓	✓	✓		$2.58$	$98.76$	$0.335$

Table 4: Ablation studies on: [left] Prompt-Anchored Cost Volume (PACV), Triple-Cues Combiner (TCC), Spatio-Temporal Decoder (STD), and [middle] temporal loss.

Method	Rainy		Dark		Static
Method	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )	AbsRel ( $\downarrow$ )	$\tau$ ( $\uparrow$ )
DepthPro [ 2 ]	$34.96$	$19.30$	$43.63$	$13.43$	$34.03$	$11.23$
PromptDA [ 36 ]	$86.15$	$6.44$	$81.60$	$10.06$	$88.15$	$2.46$
PriorDA [ 64 ]	$11.92$	$86.56$	$11.25$	$83.59$	$6.78$	$93.30$
MVSAnywhere [ 26 ]	$18.48$	$82.85$	$16.97$	$79.74$	$55.56$	$43.23$
Ours	7.21	94.02	4.97	94.84	4.93	95.56

Table 5: Ablation studies on extreme cases, including adverse weather, low-lighting, and ego-static situations. The best and second best scores are highlighted in bold and underline .

Figure 3: Visualization of depth estimation on a static scene. The result demonstrates our robustness in challenging, low-parallax scenarios.

Figure 4: Ablations on (a) different LiDAR laser beams and (b) different LiDAR occlusion rate (from bottom). Lower AbsRel refers to a better result.

Figure 5: Ablation study on Lidar prompt absence. We estimate the depth of the back-view image while only the front-view image is provided by LiDAR. Our method remains an accurate metric under such a configuration.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Illustrations of spatio-temporal decoder. Please refer to Appendix A for more details.

Metric	Definition
MAE	$\frac{1}{N}\sum_{i=1}^{N}\|D_{gt}-\hat{D}\|$
AbsRel	$\frac{1}{N}\sum_{i=1}^{N}\|D_{gt}-\hat{D}\|/D_{gt}$
$\tau$	$\frac{1}{N}\sum_{i=1}^{N}\left(\max\left(\frac{\hat{D}}{D_{gt}},\frac{D_{gt}}{\hat{D}}\right)<1.25\right)$

Appendix

Table 6: Depth metric definitions.

D_{gt}

and

\hat{D}

are the ground-truth and predicted depth, respectively.

Case	Scene ID	# Total
Case	Scene ID	Images
Static	2011_09_29_drive_0026_sync	148
	13941626351027979229	199
	14127943473592757944	197
Rainy	10448102132863604198	183
	11356601648124485814	199

Appendix

Table 7: Extreme Cases IDs

Figure 7: Qualitative comparison of 3D reconstruction results on KITTI [ 17 ] . Rows show different methods: our DriveMVS model, MVSA [ 26 ] , and PriorDA [ 64 ] . (a) : Reconstructed colored point clouds of the driving scene. (b) : Zoomed-in views highlighting fine-grained details of vegetation and structural boundaries. (c) : Corresponding $z$ -axis visualizations.

#	Time (ms)	Memory (GB)
PriorDA	$246.79$	$2.45$
PromptDA	$300.60$	$3.75$
MVSA	$38.84$	$4.02$
Ours	$65.61$	$4.47$

Appendix

Table 8: Speed and memory consumption. Number benchmarked on an A100 GPU.

Figure 8: Qualitative comparison of depth prediction results on KITTI [ 17 ] . Rows show different methods: our DriveMVS model, MVSA [ 26 ] , PriorDA [ 64 ] , MapAnything [ 30 ] , and PromptDA [ 36 ] , along with RGB and prompt inputs ( $\boldsymbol{I}_{r}$ , $\boldsymbol{P}_{r}$ ). [left] : 2011_09_26_drive_0095_sync . [Middle] : 2011_09_26_drive_0023_sync . [Right] : 2011_09_26_drive_0002_sync . The best and second best are highlighted with green and yellow borders, respectively.

Figure 9: Qualitative comparison of depth prediction errors on KITTI [ 17 ] , corresponding to Fig. 8 . Rows show different methods: our DriveMVS model, MVSA [ 26 ] , PriorDA [ 64 ] , MapAnything [ 30 ] , and PromptDA [ 36 ] , along with RGB inputs ( $\boldsymbol{I}_{r}$ ) and prompt ( $\boldsymbol{P}_{r}$ ).

Figure 10: Qualitative comparison of depth prediction results on DDAD [ 19 ] and Waymo [ 54 ] . Rows show different methods: our DriveMVS model, MVSA [ 26 ] , PriorDA [ 64 ] , MapAnything [ 30 ] , and PromptDA [ 36 ] , along with RGB and prompt inputs ( $\boldsymbol{I_{r}}$ , $\boldsymbol{P_{r}}$ ). For left to right: 000156 , 000155 , 000194 , 1024360143612057520 , 12866817684252793621 . The best and second best are highlighted with green and yellow borders, respectively.

Figure 11: Qualitative comparison of depth prediction errors on DDAD [ 17 ] and Waymo [ 54 ] , corresponding to Fig. 10 . Rows show different methods: our DriveMVS model, MVSA [ 26 ] , PriorDA [ 64 ] , MapAnything [ 30 ] , and PromptDA [ 36 ] , along with RGB inputs ( $\boldsymbol{I}_{r}$ ) and prompt ( $\boldsymbol{P}_{r}$ ).

Figure 12: Qualitative comparison of depth prediction results and depth prediction errors across extreme cases for autonomous driving in our sampled dataset (most Waymo [ 54 ] ). Columns show different methods: our DriveMVS model, MVSAnywhere [ 26 ] , and PriorDA [ 64 ] , along with RGB & prompt inputs ( $\boldsymbol{I}_{r},\boldsymbol{P}_{r}$ ) and ground-truth depths (GT). [Top] : 11356601648124485814 . [Middle] : 14107757919671295130 . [Bottom] : 14127943474592757944 .

Multi-View Depth Estimation

Multi-view depth estimation recovers consistent 3D structure from multiple images using camera geometry, learned priors, and cross-view evidence.

See all 23 papers

DrivingDepth: Sparse-Prompted Pixel-wise Scale Correction for Driving Depth Estimation
Jun 30, 2026Chi Huang, Wenhao Zhang, Hang Yin +5Metric Depth Estimation Depth Estimation
Baidu Inc.
SurroundNEXO: Ego-Centric Metric Bridging for Spatially Consistent Geometry in Autonomous Driving
Jun 15, 2026Shuai Yuan, Runxi Tang, Yuzhou Ji +8Multi-View Depth Estimation Multi-View Consistency
School of Artificial Intelligence, Shanghai Jiao Tong University · 2Hello Inc.
D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models
Jul 21, 2026Heesang Han, A. Lynn Abbott, Abhijit SarkarVLMs for Autonomous Driving Camera-LiDAR Fusion
Bradley Department of Electrical and Computer Engineering, Virginia Tech, USA · Sanghani Center for Artificial Intelligence and Data Analytics, USA · Virginia Tech Transportation Institute, USA

Jun 30, 2026J/K move · Enter open · S save

DrivingDepth: Sparse-Prompted Pixel-wise Scale Correction for Driving Depth Estimation

Chi Huang, Wenhao Zhang, Hang Yin, YuAn Wang+4

Baidu Inc.

Metric Depth Estimation Depth Estimation Camera-LiDAR Fusion Autonomous Driving Perception

Open paper AskPDF arXiv

Dense depth estimation for autonomous driving faces a geometry-scale conflict: depth foundation models deliver pixel-aligned dense visual geometry without reliable metric scale, while projected LiDAR provides metric anchors that are sparse, noisy, and misaligned with image structures. Existing sparse-prompted methods incorporate LiDAR by regenerating depth from scratch, overriding the foundation model's coherent geometry and producing structural artifacts on visually continuous surfaces. Our key insight is that foundation models already capture geometrically coherent relative depth; no additional surface structure learning is required-only a per-pixel scale factor mapping relative geometry to metric coordinates. Based on this, we propose DrivingDepth, which treats sparse LiDAR as geometric prompts that locally calibrate a frozen foundation prior through residual pixel-wise scale correction, preserving dense visual geometry by construction. On nuScenes with 4-frame surround-view input, DrivingDepth achieves an AbsRel of 11.19 and an EdgeCR of 5.741, outperforming MapAnything (11.99/1.914) by simultaneously delivering SOTA metric accuracy and geometric consistency.

Abstract

Multi-View Depth Estimation

Explore similar work

DrivingDepth: Sparse-Prompted Pixel-wise Scale Correction for Driving Depth Estimation

SurroundNEXO: Ego-Centric Metric Bridging for Spatially Consistent Geometry in Autonomous Driving

D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models