Seeing Is Not Measuring: Tool-Augmented Metric Spatial Reasoning for Vision-Language Models

Kai Glantz; Clemens Grange

cs.CVSep 24, 2026

Seeing Is Not Measuring: Tool-Augmented Metric Spatial Reasoning for Vision-Language Models

Authors: Kai Glantz, Clemens Grange

Organizations: Technical University of Munich

Abstract

Vision-Language Models (VLMs) describe scenes well but reason poorly about metric 3D structure such as absolute distances, physical sizes, or egocentric directions. We present a modular, predictor agnostic, tool-augmented framework that equips a small VLM (Qwen3.5-4B) with geometric tools: 3D object detection, metric depth estimation, and deterministic solvers for distance, size and bearing. Each object is detected in the camera frame of its own best view, and the tools use that frame's pose to lift every detection into one shared world frame. Moving metric computation out of the model's weights and into explicit solvers yields large gains on three of four ReVSI-Bench tasks: with a strong monocular detector (WildDet3D), absolute distance rises from 0.46 to 0.74 Mean Relative Accuracy (MRA), relative distance from 39.1% to 67.4%, and relative direction from a below-chance 25.9% to 73.4%. Because any detector can be swapped in behind the tool interface, comparing real detectors against ground-truth boxes separates perception error from reasoning error: orchestration costs only 0.03 MRA. Object size is bounded by the detector: the tools are near-exact on groundtruth boxes (0.97) yet the best real detector barely beats the no-tool baseline (0.61 vs. 0.58), because size reads straight off a box extent monocular detectors get wrong. Without a predefined recipe, the model already sequences the tools correctly on its own, matching a scripted pipeline on three of four tasks.

Figures & tables

Figure 1 : Example of tool-augmented metric spatial reasoning. The VLM orchestrates tools to answer a spatial distance question from video frames: detect_object_3d returns an oriented 3D box in the camera frame of each object’s own best view, project_box_to_world lifts those boxes into a single shared metric world frame using the frames’ camera poses, and a deterministic solver returns the surface-to-surface distance.

No-tool baselines, absolute distance (MRA $\uparrow$ )
Model	visual	blind	+intrinsics
Qwen3.5-0.8B	0.00	0.00	0.02
Qwen3.5-2B	0.21	0.00	0.03
Qwen3.5-4B	0.46	0.29	0.17

Table 1 : Native (no-tool) VLMs on absolute distance. Only the 4B model is a usable baseline; passing camera intrinsics as text does not help.

Absolute distance (401 Q)
Method	MRA $\uparrow$	Ceiling $\uparrow$
No tools (visual)	0.46	–
Tools, 2D + depth	0.32	0.29
Tools, 3D boxes (OVMono3D)	0.26	0.26
Tools, 3D boxes (Cube R-CNN)	0.32	0.28
Tools, 3D boxes (WildDet3D)	0.74	0.74

Table 2 : Absolute distance and object size results (ReVSI-Bench ARKitScenes); the toolset behind each row is given in Tab. A4 . “Ceiling” runs the geometric tools directly on the boxes, without a VLM, so it measures what the perception backend allows before any orchestration.

Relative distance (215 Q, chance $25.0\%$ )
Method	Acc. $\uparrow$	Parse	Ceiling
No tools (blind)	30.2%	100.0%	–
No tools (visual)	39.1%	100.0%	–
Tools, 3D boxes (WildDet3D)	67.4%	100.0%	69.3%
Tools, 3D boxes (GT oracle)	92.1%	100.0%	94.9%
Relative direction (290 Q, chance $29.3\%$ )

Table 3 : Multiple-choice results on ReVSI-Bench. “Parse” is the share of runs emitting a usable option letter ( Sec. A ); unparseable answers score wrong. Both no-tool baselines parse at

100\%

, so their at-or-below-chance relative-direction accuracy is a real reasoning failure and not a formatting artifact.

	MRA $\uparrow$		Acc. $\uparrow$
Method	Dist.	Size	R. dist	R. dir
No recipe (generic prompt)	0.94	0.97	92.5	70.0
+ few-shot order examples	0.89	1.00	80.0	75.0
Scripted recipe ( Tabs. 2 and 3 )	0.94	0.97	92.1	80.0

Table 4 : Autonomous probe: every tool exposed at once, no per-task recipe (

160

questions,

40

per task, GT backend). Distance and size are scored by MRA, the two relative tasks by accuracy. The scripted-recipe row repeats the GT-oracle results of Tabs. 2 and 3 , measured on the full splits rather than on this subsample.

Backend	Objects found	Questions fully covered
GT Oracle	100.0% (802/802)	100.0% (401/401)
OVMono3D	100.0% (802/802)	100.0% (401/401)
Cube R-CNN	91.8% (736/802)	85.0% (341/401)
WildDet3D	100.0% (802/802)	100.0% (401/401)
Grounding DINO (2D)	93.6% (751/802)	87.8% (352/401)

Table A1 : Detector coverage on the absolute-distance split (401 questions, 802 queried object instances). Objects found : the backend returns a box for the queried name. Questions fully covered : boxes exist for both objects, i.e. the tool chain can run end-to-end. Cube R-CNN and the 2D grounding front-end miss objects, which is why their VLM MRA can exceed their own ceiling; OVMono3D is fully covered yet still the weakest backend ( Tab. 2 ), so coverage and accuracy are independent failure modes.

Pipeline	task / answer_format
bbox_3d_abs_dist gdino_depthpro_abs_dist	distances between objects a single number in metres, e.g.: 1.5
bbox_3d_size gdino_depthpro_size	the size (longest dimension) of objects a single number in centimetres, e.g.: 120
bbox_3d_reldist	which of several candidate objects is closest or farthest to a reference object the letter (A, B, C, or D)
bbox_3d_reldir	the egocentric direction (left/right/back, or a front/back-left/right quadrant) of a target from a viewer the letter (A, B, C, or D)
autonomous_3d	spatial questions about objects (all four tasks mixed) the exact format the question asks for

Table A2 : Per-pipeline task and answer_format strings substituted into the scaffold of . These reuse the same tool interface across numeric (MRA) and multiple-choice tasks.

Tool	Role (in/out)
detect_object_3d	name $\to$ camera-space OBB (predictor-agnostic backend)
project_box_to_world	name $\to$ world-space OBB (pose lift)
calculate_object_distance	two names $\to$ surface distance (m)
calculate_object_size	name $\to$ longest dim (cm)
relative_direction	three names $+$ mode $\to$ 3-/4-way label
detect_object	name $\to$ 2D bbox (Grounding DINO)

Table A3 : Tool registry. Top block: 3D OBB tools (used by the bbox_3d_* and autonomous_3d pipelines). Bottom block: the 2D + depth contrast tools ( gdino_depthpro_* ). Every tool takes object names ; geometry is cached host-side.

Task	Pipeline (Toolset)	Geometry Representation	Min. Steps	Median Steps
Abs. Distance	gdino_depthpro_abs_dist	2D bbox + Depth (center-to-center)	7	7
Abs. Distance	bbox_3d_abs_dist	3D OBB (surface-to-surface)	5	5
Object Size	gdino_depthpro_size	Apparent 2D projection	3	3
Object Size	bbox_3d_size	3D OBB extent (rigid invariant)	2	2
Rel. Distance	bbox_3d_reldist	3D OBB (nearest-instance surface)	14	16
Rel. Direction	bbox_3d_reldir	3D OBB (egocentric bearing angle)	7	8

Table A4 : The four core toolsets, plus the 2D + depth baselines. Direct 3D box pipelines compute surface-to-surface relationships and rigid invariants. Median Steps ( Sec. F ; GT backend for the 3D pipelines, real detector for the 2D) equals Min. Steps on every numeric pipeline and exceeds it only on the two multiple-choice tasks, a benign overhead: a premature bearing call that errors and is retried (rel. direction), and redundant distance calls (rel. distance).

Figure A1 : Detector backends behind detect_object_3d on one object (dishwasher, ARKitScenes 45663164, best frame 10). Top four panels: each backend’s predicted 3D box (solid) with the ground-truth box (dashed cyan) reprojected onto the best frame; $\Delta$ is the world-space center error. Bottom: the same boxes as top-down ground-plane footprints. WildDet3D ( $\Delta{=}0.04$ m) nearly coincides with GT, OVMono3D ( $0.31$ m) is oversized and offset, and Cube R-CNN ( $0.69$ m) drifts onto the sink. The tool-calling chain is identical across backends; only the box origin changes, so this gap is the perception bottleneck of Sec. 4 .

Figure A2 : Intermediate signals of the 2D + depth pipeline ( gdino_depthpro_abs_dist ) on a wardrobe/wall-picture distance query (ARKitScenes 41069042). Top: each object’s best frame with its Grounding DINO 2D box. Bottom: the metric depth map Depth Pro predicts for that frame (bright $=$ near, each map scaled to its own range), with the same box and the sampled center pixel (“ $+$ ”); the pipeline keeps only the median depth inside the box ( $2.52$ m for the wardrobe, $1.02$ m for the wall picture). Back-projecting the two box centers yields a center-to-center distance ( $3.10$ m) that overshoots the surface-to-surface ground truth ( $2.4$ m), the systematic bias the OBB pipeline corrects. Depth Pro also reads the bedspread in the lower right as far (dark) although it is the nearest surface: monocular depth degrades on large textureless regions, the weakness behind the error tail in Fig. A4 .

Figure A3 : Mean vs. median MRA across every absolute-distance pipeline (401 questions); the gap between the bars measures tail sensitivity. Only WildDet3D beats the no-tool baseline: OVMono3D ( $0.26$ ), Cube R-CNN ( $0.32$ ) and the 2D+depth pipeline ( $0.32$ ) all land below it ( $0.46$ ), so a poor box is worse than no box. OVMono3D’s median is $0.00$ , i.e. more than half its answers are off by over $50\%$ . WildDet3D (mean $0.74$ , median $0.90$ ) approaches the GT oracle ( $0.94$ / $1.00$ ).

Figure A4 : The same pipelines in metric error , which corroborates the MRA ranking. The weak 3D backends are worse than the no-tool baseline (MAE $1.56$ m for OVMono3D and $1.25$ m for Cube R-CNN, against $0.94$ m), while WildDet3D ( $0.48$ m) and the GT oracle ( $0.13$ m) sit far below it. Every OBB pipeline stays bounded (max error $\leq 6.2$ m), so their means are trustworthy. The 2D+depth pipeline is the exception: monocular depth blows up on boundary boxes to a worst error of $10\,187$ m, leaving an MAE ( $29.8$ m) thirty times its own median ( $0.99$ m).

Figure A5 : Per-sample MRA distributions on absolute distance (401 questions) for the no-tool visual baseline and each detect_object_3d backend. The GT oracle concentrates at $1.0$ (median $1.00$ ) and WildDet3D tracks it (median $0.90$ ), whereas OVMono3D and Cube R-CNN pile up at zero: their medians ( $0.00$ and $0.20$ ) sit below the no-tool baseline’s ( $0.40$ ), so over half of OVMono3D’s answers are more than $50\%$ off. Tool augmentation only pays off once the detector is strong enough.

Figure A6 : Object size from a 2D box ( gdino_depthpro_size ) on six ReVSI objects: the Grounding DINO box (red) and mask (cyan) with the longest image-plane axis (yellow). Panel titles give GT and both back-projected estimates in cm, mask-based and box-based ( bbox , the reported variant). Back-projection sees only the image-plane dimensions, so it is accurate when the longest axis lies in that plane (kitchen sink $104\to 103$ , couch $165\to 168$ ) and fails in two ways: the box covers only part of the object (floor lamp $195\to 80$ : only the shade is grounded), or it lands on the wrong surface (dresser $90\to 118$ ). This variance, not a constant bias, caps the 2D size pipeline at $0.34$ MRA.

Figure A7 : Minimal correct chain vs. observed tool-call count per question type (GT backend). Stars mark the minimal chain ( $2n_{\text{obj}}+n_{\text{meas}}$ , structural per question); boxes summarize the tool- using runs (median line, IQR), and jittered points show every run including the zero-tool band (annotated %). Numeric tasks run the minimal chain exactly; relative direction runs $+1$ (a premature relative_direction call that errors and is retried) and relative distance $+2$ (redundant distance calls). The $23\%$ zero-tool rate on relative direction is the orchestration discipline gap, separate from this overhead.

Subtype	N	GT Oracle	WildDet3D
Backward Easy (3-way)	53	79.2%	75.5%
Backward Hard (4-way)	69	84.1%	68.1%
Forward Easy (3-way)	96	82.3%	74.0%
Forward Hard (4-way)	72	73.6%	76.4%
Overall	290	80.0%	73.4%

Table A5 : Relative-direction accuracy by subtype (“easy” = 3-way left/right/back; “hard” = 4-way quadrant; “backward” = the away facing mode). The GT oracle averages

6.1

tool calls per question over all runs, or

8.0

(median

8

) over the runs that call any tool at all; the gap to the

100\%

geometric ceiling is the

23\%

zero-tool guessing rate.

Condition	MRA	Dist.	Size	Errors	Zero-tool
1. Terse tool errors	0.729	0.575	0.883	138	5/80
2. + actionable errors	0.800	0.878	0.723	36	14/80
3. + few-shot examples	0.933	0.938	0.928	13	0/80

Table A6 : The staged ablation on tool feedback (80 questions: 40 absolute distance, 40 object size; GT backend). Errors counts tool results returned as errors across all

80

traces; Zero-tool counts runs that answered without calling a single tool. Better feedback removes the looping, and here the few-shot examples remove the remaining guessing; on the four-task probe ( Tab. 4 ) they no longer help.

Explore similar work

Sep 28, 2026cs.CV

Geometric Encoding for Spatial Reasoning in Vision-Language Models

Vision-Language Models (VLMs) are far more reliable at recognizing what appears in a video than at reasoning about its spatial and temporal properties, such as metric distances, object dimensions, and consistent object identities across frames. We present Geometric Code, a perception-to-geometry pipeline that computes explicit spatial structure from video and supplies it to VLMs as context to augment reasoning. A perception layer segments and classifies objects and recovers depth, camera pose, and intrinsics from monocular RGB video. A deterministic geometric engine then back-projects, merges, and cleans these outputs into a spatial code, including per-object positions, dimensions, counts, inter-object distances, appearance order, and room geometry. The code is serialized into VLMs' prompts, either alongside the video or replacing it entirely. Specifically, there is no component trained or fine-tuned in our approach. On VSI-Bench, augmenting 2B and 4B open models with the spatial code improves average accuracy by +4.1 points over the frames-only baseline, with the largest gains on numeric estimation tasks such as absolute distance (+24.1 points). The results suggest that explicitly computed geometry, delivered through the language channel, recovers spatial competence that small VLMs cannot extract from pixels alone.

Antonio Jun, Haoshui Yu, Zhengyi Lu +2

School of Arts and Sciences New York University New York City, United States · Dept. of Engineering and Computer Science Oakland University Rochester, United States

Spatial Reasoning 3D Layout Generation

Ask PDF

May 15, 2026cs.CV

Unlocking Dense Metric Depth Estimation in VLMs

Vision-Language Models (VLMs) excel at 2D tasks such as grounding and captioning, yet remain limited in 3D understanding. A key limitation is their text-only supervision paradigm, which under-constrains fine-grained visual perception and prevents the recovery of dense geometry. Prior methods either distill geometry from external vision models, introducing error accumulation, or enable direct prediction with inefficient per-pixel query or coarse token-level outputs. In this paper, we propose DepthVLM, a simple yet effective framework that transforms a single VLM into a native dense geometry predictor while preserving its multimodal capability. By attaching a lightweight depth head to the LLM backbone and training under a unified vision-text supervision paradigm with a two-stage schedule, DepthVLM generates full-resolution depth maps alongside language outputs in a single forward pass. We further introduce a unified indoor-outdoor metric depth benchmark in a VLM-compatible format. Experiments show that DepthVLM significantly outperforms existing VLMs with higher inference efficiency, surpasses leading pure vision models, and improves complex 3D spatial reasoning, moving toward a truly unified multimodal foundation model. The project page is available at https://depthvlm.github.io/

Hanxun Yu, Xuan Qu, Yuxin Wang +2

Zhejiang University · Tencent Hunyuan LLM · HKUST +1

Depth Estimation 3D Representation

Ask PDF

Aug 3, 2026cs.CV

SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models

Vision-Language Models (VLMs) perform well on commonsense reasoning tasks but struggle with visual spatial reasoning. Most existing solutions introduce extra 3D prior inputs or external spatial encoders, which increase complexity and degrade the underlying VLMs' general-purpose capabilities after spatial fine-tuning. To this end, we propose a parameter-efficient \textit{\textbf{Spatio}-vision \textbf{L}anguage \textbf{M}odels (SpatioLM)}, that enhances spatial intelligence without extra 3D prior inputs or third-party spatial encoders. Concretely, we design a plug-and-play and non-invasive spatio-vision module that elicits the spatial knowledge inherent in VLMs. Furthermore, we innovatively leverage pseudo depth and camera information as supervision to guide the model in learning physically coherent representations. Extensive experiments show that SpatioLM achieves significant improvements in diverse tasks, including spatial perception and understanding while effectively limiting the degradation of general capabilities. Notably, the model achieves an impressive score of 71.6 on the VSI-Bench (the first model to surpass 70). In addition, it attains competitive performance when transferred to embodied manipulation tasks. Code is available at \faGithub~spatio-lm.

Jing Wu, Jianhua Wu, Jiayi Guan +5

Xiaomi EV, Beijing, China · College of Automotive and Energy Engineering, Tongji University, Shanghai, China · Independent Researcher

Recent Vision-Language Models Stable Spatial Understanding Spatial Encoding

Ask PDF