DepthEvidence: Unifying Metric Depth Prediction and Geometric Reasoning in Multimodal Language Models

Jiangning Wei; Yuan Yao; Miaomiao Cui; Mingsheng Li; Humen Zhong; Shuai Bai; Zhibo Yang

cs.CVSep 25, 2026

DepthEvidence: Unifying Metric Depth Prediction and Geometric Reasoning in Multimodal Language Models

Authors: Jiangning Wei, Yuan Yao, Miaomiao Cui, Mingsheng Li, Humen Zhong, Shuai Bai, Zhibo Yang

Organizations: Alibaba Token Hub, Alibaba Group · Beijing University of Posts and Telecommunications

Abstract

Spatial reasoning with metric constraints requires linking objects to geometric measurements and preserving their numerical content during language reasoning. We present DepthEvidence, a 4B model that uses its own dense metric predictions as object-grounded evidence for language generation. A camera-conditioned decoder predicts full-resolution metric depth using multi-scale visual features and high-resolution RGB refinement. A dense-to-language interface converts predicted depths and decoder features into object-aligned continuous geometry tokens anchored to object identifiers. Geometric supervision encourages metric information to remain recoverable before and after language-context interaction, while instruction tuning supports object measurement and compositional reasoning. We introduce a Depth-VQA benchmark evaluating object-depth queries, relative comparisons, and decisions combining spatial and numerical constraints. Across nine datasets, DepthEvidence achieves the highest average dense $δ_1$ among evaluated methods, competitive with specialized estimators. It also leads the evaluated methods in instance-level metric depth estimation and overall accuracy on both relative and metric reasoning tracks, while broadly preserving general VQA performance and improving spatial understanding relative to the base model.

Figures & tables

Figure 1: Overview of DepthEvidence’s capabilities. (a) A metric-spatial question requires selecting objects by image location and mean camera depth; the dense-to-language interface rejects out-of-range distractors. (b) Dense metric depth results ( $\delta_{1}$ ) on four representative datasets and the nine-dataset average, compared with specialized estimators. (c) Level 1 metric estimation and Level 2 relative and metric reasoning, compared with general-purpose and depth-aware VLMs as well as the no-interface variant. (d) General VQA retention on MMMU, MMBench, and OCRBench, and spatial understanding on Ego3D. All scores are percentages and higher is better.

Figure 2: Overview of DepthEvidence. The dense decoder supplies metric predictions and features to the dense-to-language interface, which converts them into object-aligned geometry tokens for answer generation. This path connects Level-0 dense prediction to Level-1 instance measurement and Level-2 compositional reasoning within a shared VLM.

Track	Subbench	Question summary
Relative Depth	Spatial Pair	Compare two objects, optionally after resolving a spatial reference.
	Spatial Group	Spatially filter candidates, then select the nearest or farthest.
	Depth Ordering	Select an extreme or order a subset or all referenced objects.
Metric Depth	Metric Pair	Predict a mean-depth gap or compare it with a threshold.
	Spatial Metric Pair	Resolve a spatial reference, then predict an absolute mean-depth gap.
	Spatial Metric Group	Spatially filter candidates, then apply range or bound constraints.

Table 1: Level-2 tracks and subbench goals. Appendix Table 7 provides all question types and reasoning requirements.

Method	Outdoor				Indoor					Avg.
Method	Argoverse2	Waymo	NuScenes	DDAD	Taskonomy	ScanNet++	HyperSim	SUN RGB-D	Booster	Avg.
VLMs Trained on Metric Depth Estimation
Youtu-VL-4B-Instruct	0.6290	0.5121	0.6443	0.2995	0.7561	0.6006	0.4285	0.3525	0.3471	0.5077
DepthVLM-4B	0.7696	0.8056	0.8433	0.8232	0.6992	0.8568	0.4536	0.5089	0.2452	0.6673
DepthLM-12B †	0.7768	0.7758	0.8310	0.6638	0.8218	0.7962	0.4358	0.5508	0.5934	0.6939
Specialized Vision Models

Table 2: Level-0 dense metric depth comparison across outdoor and indoor benchmarks. We report

\delta_{1}\uparrow

and its unweighted average over all nine datasets. The best and second-best results in each column are shown in bold and underlined, respectively. Rankings are determined using the unrounded values. † DepthLM-12B follows its original sparse point-query protocol ( Cai et al., 2026 ) , using 8,192 randomly sampled pixels per dataset.

Figure 3: Qualitative dense depth on Argoverse2, Waymo, ScanNet++, and HyperSim. Columns show RGB, ground truth (GT), DepthVLM-4B, Youtu-VL-4B-Instruct, MetricAnything, UniDepth V2, and DepthEvidence. Outdoor GT is densified only for visualization; Table 2 uses the original sparse sensor GT. Each row shares one color scale; Figure 5 shows the other five datasets.

General-Purpose VLMs
Method	Level-1: Instance Metric Depth			Level-2: Compositional Depth Reasoning
	Level-1: Instance Metric Depth			Relative				Metric
	$\delta_{1}\uparrow$	RMSE $\downarrow$	AbsRel $\downarrow$	Spatial Pair	Spatial Group	Ordering	Overall Acc	Metric Pair Acc	Metric Pair $\delta_{1}\uparrow$	Spatial Pair $\delta_{1}\uparrow$	Spatial Group	Set Decision	Overall Acc
Qwen3.5-4B	0.0159	206.1221	9.5185	0.0930	0.0568	0.0102	0.0405	0.6304	0.2500	0.2234	0.3085	0.5532	0.4701
Gemini-3.1-Pro-Preview	0.4108	52.4680	2.0927	0.6395	0.5795	0.3724	0.4838	0.6304	0.2292	0.2340	0.3085	0.5957	0.4872
Qwen3.7-plus	0.0766	107.2556	3.0477	0.5814	0.6136	0.3827	0.4838	0.6522	0.2917	0.1489	0.3511	0.4681	0.4573

Table 3: Instance-level depth estimation and reasoning results. The interface is enabled or disabled at both instance levels. Relative Overall Acc is computed over 370 questions, and Metric Overall Acc includes the 234 questions scored by accuracy. Bold and underlined entries denote the best and second-best results in each column.

Figure 4: Representative Level-2 cases corrected by the dense-to-language interface. The three panels test complete depth ordering, spatially filtered metric-range selection, and margin-aware ambiguity decisions, respectively. Each panel contrasts matched variants without and with the interface; green answers match the ground truth. Aggregate results are reported in Table 3 .

Method	STEM				General VQA
Method	MathVision	MathVista	MMMU	MMMU-Pro	RealWorldQA	MMStar	MMBench	HallusionBench
Youtu-VL-4B-Instruct	–	76.50	61.10	43.00	74.60	71.10	83.90	59.10
GLM-4.1V-9B-Thinking	54.40	80.70	68.00	57.10	–	72.90	85.80	63.20
Qwen3.5-4B	74.60	85.10	77.60	66.30	79.50	78.30	89.40	65.00
DepthEvidence (Ours)	74.10	84.90	78.71	64.05	78.06	78.30	89.52	68.21

Table 4: General and spatial capability evaluation. Metrics follow their native scales; all are higher-is-better except Ego3D RMSE. Bold marks the best result, and “–” denotes unavailable results.

Table 9

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Track	Subbench	Raw question type	Required reasoning
Relative Depth	Spatial Pair	closer_farther	Decide which of two objects is closer to or farther from the camera.
		front_behind	Decide which of two objects is in front of or behind the other.
		spatial_depth	Resolve a left/right reference, then compare the selected object with another object in depth.
	Spatial Group	spatial_depth_group	Filter objects by a spatial relation, then select the nearest or farthest member.
	Depth Ordering	depth_ordering_closest	Select the closest object from all candidates.
		depth_ordering_farthest	Select the farthest object from all candidates.

Appendix

Table 7: Detailed composition of the Relative Depth and Metric Depth tracks in the Level-2 benchmark. “Subset” orders a specified subset of objects, whereas “all” orders every referenced object.

Component	Sampling unit	Scale or coverage	Task composition
Level-0 dense depth	Image	4,228 images from nine datasets	500 images from each of eight datasets and all 228 Booster images
Level-1 instance depth	Grounded-object query	757 queries from seven datasets	Center, nearest, farthest, mean, and median metric depth; box or referring prompts
Level-2 Relative	Multi-object question	370 questions	Spatial Pair (86), Spatial Group (88), and Depth Ordering (196)
Level-2 Metric	Multi-object question	376 questions	Metric Pair (46 threshold + 48 numerical gap), Spatial Metric Pair (94), Spatial Metric Group (94), and Metric Set Decision (94)

Appendix

Table 8: Compact statistics for the dense and instance-level depth benchmarks. Depth-VQA contains 1,503 questions in total.

Variant	Relative Overall	Metric Overall
No interface	0.5838	0.7692
No interface + metric supervision	0.5865	0.7949
Interface + $\mathcal{L}_{\mathrm{ans}}$	0.5838	0.7735
Full interface	0.6270	0.8504

Appendix

Table 9: Disentangling direct metric supervision from the dense-to-language interface on Level-2 Relative and Metric Overall Acc.

Mode	Generation latency (s) $\downarrow$	End-to-end latency (s) $\downarrow$	Throughput (tokens/s) $\uparrow$	Peak allocated memory (GiB) $\downarrow$
No-interface	4.174	4.289	15.333	10.36
Two-pass	4.535	4.710	14.111	13.56
Single-prefill	4.323	4.491	14.804	13.73

Appendix

Table 10: Inference efficiency with a fixed output length of 64 tokens and end-of-sequence stopping disabled. Single-prefill is the cached geometry-continuation scheme used by DepthEvidence.

Figure 5: Additional qualitative dense metric depth comparisons on NuScenes, DDAD, Taskonomy, SUN RGB-D, and Booster. Columns show RGB, ground truth (GT), DepthVLM-4B, Youtu-VL-4B-Instruct, MetricAnything, UniDepth V2, and DepthEvidence. Outdoor GT is densely completed for visualization only; quantitative evaluation uses the original sparse sensor GT. All depth maps within a row use the same color scale.

3D VLMs

Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.

See all 213 papers

Unlocking Dense Metric Depth Estimation in VLMs
May 15, 2026Hanxun Yu, Xuan Qu, Yuxin Wang +2Vision-Language Models Metric Depth Estimation
Zhejiang University · Tencent Hunyuan LLM · HKUST +1
When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoning
Jul 13, 2026Quynh Vo, Phuc Dao, Cong-Duy Nguyen +1Vision-Language Models Visual Spatial Reasoning
National University of Singapore · Center of AI Research, VinUniversity
Learning and Reasoning with Geometry: How Generation Helps Spatial Understanding
Oct 26, 2026Rui Yang, Yanwei Li, Sili Chen +3Spatial Reasoning 3D Spatial Reasoning
1The University of Hong Kong · 2Shanghai Jiao Tong University · 3ByteDance Seed

May 15, 2026J/K move · Enter open · S save

Unlocking Dense Metric Depth Estimation in VLMs

Hanxun Yu, Xuan Qu, Yuxin Wang, Jianke Zhu+1

Zhejiang University · Tencent Hunyuan LLM · HKUST+1

Vision-Language Models Metric Depth Estimation Depth Estimation Monocular Depth Estimation Multimodal Foundation Models 3D Scene Understanding

Open paper AskPDF arXiv

Vision-Language Models (VLMs) excel at 2D tasks such as grounding and captioning, yet remain limited in 3D understanding. A key limitation is their text-only supervision paradigm, which under-constrains fine-grained visual perception and prevents the recovery of dense geometry. Prior methods either distill geometry from external vision models, introducing error accumulation, or enable direct prediction with inefficient per-pixel query or coarse token-level outputs. In this paper, we propose DepthVLM, a simple yet effective framework that transforms a single VLM into a native dense geometry predictor while preserving its multimodal capability. By attaching a lightweight depth head to the LLM backbone and training under a unified vision-text supervision paradigm with a two-stage schedule, DepthVLM generates full-resolution depth maps alongside language outputs in a single forward pass. We further introduce a unified indoor-outdoor metric depth benchmark in a VLM-compatible format. Experiments show that DepthVLM significantly outperforms existing VLMs with higher inference efficiency, surpasses leading pure vision models, and improves complex 3D spatial reasoning, moving toward a truly unified multimodal foundation model. The project page is available at https://depthvlm.github.io/

Abstract

3D VLMs

Explore similar work

Unlocking Dense Metric Depth Estimation in VLMs

When Depth Is Better Told Than Shown: Depth-Ordinal Prompting for Vision-Language Spatial Reasoning

Learning and Reasoning with Geometry: How Generation Helps Spatial Understanding