cs.CVSep 24, 2026
SaveSeeing Is Not Measuring: Tool-Augmented Metric Spatial Reasoning for Vision-Language Models
Organizations: Technical University of Munich
Abstract
Vision-Language Models (VLMs) describe scenes well but reason poorly about metric 3D structure such as absolute distances, physical sizes, or egocentric directions. We present a modular, predictor agnostic, tool-augmented framework that equips a small VLM (Qwen3.5-4B) with geometric tools: 3D object detection, metric depth estimation, and deterministic solvers for distance, size and bearing. Each object is detected in the camera frame of its own best view, and the tools use that frame's pose to lift every detection into one shared world frame. Moving metric computation out of the model's weights and into explicit solvers yields large gains on three of four ReVSI-Bench tasks: with a strong monocular detector (WildDet3D), absolute distance rises from 0.46 to 0.74 Mean Relative Accuracy (MRA), relative distance from 39.1% to 67.4%, and relative direction from a below-chance 25.9% to 73.4%. Because any detector can be swapped in behind the tool interface, comparing real detectors against ground-truth boxes separates perception error from reasoning error: orchestration costs only 0.03 MRA. Object size is bounded by the detector: the tools are near-exact on groundtruth boxes (0.97) yet the best real detector barely beats the no-tool baseline (0.61 vs. 0.58), because size reads straight off a box extent monocular detectors get wrong. Without a predefined recipe, the model already sequences the tools correctly on its own, matching a scripted pipeline on three of four tasks.
Figures & tables
Figure 1 : Example of tool-augmented metric spatial reasoning. The VLM orchestrates tools to answer a spatial distance question from video frames: detect_object_3d returns an oriented 3D box in the camera frame of each object’s own best view, project_box_to_world lifts those boxes into a single shared metric world frame using the frames’ camera poses, and a deterministic solver returns the surface-to-surface distance.
| No-tool baselines, absolute distance (MRA ) | |||
|---|---|---|---|
| Model | visual | blind | +intrinsics |
| Qwen3.5-0.8B | 0.00 | 0.00 | 0.02 |
| Qwen3.5-2B | 0.21 | 0.00 | 0.03 |
| Qwen3.5-4B | 0.46 | 0.29 | 0.17 |
Table 1 : Native (no-tool) VLMs on absolute distance. Only the 4B model is a usable baseline; passing camera intrinsics as text does not help.
| Method | MRA | Ceiling |
|---|---|---|
| Absolute distance (401 Q) | ||
| No tools (visual) | 0.46 | – |
| Tools, 2D + depth | 0.32 | 0.29 |
| Tools, 3D boxes (OVMono3D) | 0.26 | 0.26 |
| Tools, 3D boxes (Cube R-CNN) | 0.32 | 0.28 |
| Tools, 3D boxes (WildDet3D) | 0.74 | 0.74 |
Table 2 : Absolute distance and object size results (ReVSI-Bench ARKitScenes); the toolset behind each row is given in Tab. A4 . “Ceiling” runs the geometric tools directly on the boxes, without a VLM, so it measures what the perception backend allows before any orchestration.
| Method | Acc. | Parse | Ceiling |
|---|---|---|---|
| Relative distance (215 Q, chance ) | |||
| No tools (blind) | 30.2% | 100.0% | – |
| No tools (visual) | 39.1% | 100.0% | – |
| Tools, 3D boxes (WildDet3D) | 67.4% | 100.0% | 69.3% |
| Tools, 3D boxes (GT oracle) | 92.1% | 100.0% | 94.9% |
| Relative direction (290 Q, chance ) | |||
Table 3 : Multiple-choice results on ReVSI-Bench. “Parse” is the share of runs emitting a usable option letter ( Sec. A ); unparseable answers score wrong. Both no-tool baselines parse at , so their at-or-below-chance relative-direction accuracy is a real reasoning failure and not a formatting artifact.
| MRA | Acc. | |||
|---|---|---|---|---|
| Method | Dist. | Size | R. dist | R. dir |
| No recipe (generic prompt) | 0.94 | 0.97 | 92.5 | 70.0 |
| + few-shot order examples | 0.89 | 1.00 | 80.0 | 75.0 |
| Scripted recipe ( Tabs. 2 and 3 ) | 0.94 | 0.97 | 92.1 | 80.0 |
Table 4 : Autonomous probe: every tool exposed at once, no per-task recipe ( questions, per task, GT backend). Distance and size are scored by MRA, the two relative tasks by accuracy. The scripted-recipe row repeats the GT-oracle results of Tabs. 2 and 3 , measured on the full splits rather than on this subsample.
| Backend | Objects found | Questions fully covered |
|---|---|---|
| GT Oracle | 100.0% (802/802) | 100.0% (401/401) |
| OVMono3D | 100.0% (802/802) | 100.0% (401/401) |
| Cube R-CNN | 91.8% (736/802) | 85.0% (341/401) |
| WildDet3D | 100.0% (802/802) | 100.0% (401/401) |
| Grounding DINO (2D) | 93.6% (751/802) | 87.8% (352/401) |
Table A1 : Detector coverage on the absolute-distance split (401 questions, 802 queried object instances). Objects found : the backend returns a box for the queried name. Questions fully covered : boxes exist for both objects, i.e. the tool chain can run end-to-end. Cube R-CNN and the 2D grounding front-end miss objects, which is why their VLM MRA can exceed their own ceiling; OVMono3D is fully covered yet still the weakest backend ( Tab. 2 ), so coverage and accuracy are independent failure modes.
| Pipeline | task / answer_format |
|---|---|
| bbox_3d_abs_dist gdino_depthpro_abs_dist | distances between objects a single number in metres, e.g.: 1.5 |
| bbox_3d_size gdino_depthpro_size | the size (longest dimension) of objects a single number in centimetres, e.g.: 120 |
| bbox_3d_reldist | which of several candidate objects is closest or farthest to a reference object the letter (A, B, C, or D) |
| bbox_3d_reldir | the egocentric direction (left/right/back, or a front/back-left/right quadrant) of a target from a viewer the letter (A, B, C, or D) |
| autonomous_3d | spatial questions about objects (all four tasks mixed) the exact format the question asks for |
Table A2 : Per-pipeline task and answer_format strings substituted into the scaffold of . These reuse the same tool interface across numeric (MRA) and multiple-choice tasks.
| Tool | Role (in/out) |
|---|---|
| detect_object_3d | name camera-space OBB (predictor-agnostic backend) |
| project_box_to_world | name world-space OBB (pose lift) |
| calculate_object_distance | two names surface distance (m) |
| calculate_object_size | name longest dim (cm) |
| relative_direction | three names mode 3-/4-way label |
| detect_object | name 2D bbox (Grounding DINO) |
Table A3 : Tool registry. Top block: 3D OBB tools (used by the bbox_3d_* and autonomous_3d pipelines). Bottom block: the 2D + depth contrast tools ( gdino_depthpro_* ). Every tool takes object names ; geometry is cached host-side.
| Task | Pipeline (Toolset) | Geometry Representation | Min. Steps | Median Steps |
|---|---|---|---|---|
| Abs. Distance | gdino_depthpro_abs_dist | 2D bbox + Depth (center-to-center) | 7 | 7 |
| Abs. Distance | bbox_3d_abs_dist | 3D OBB (surface-to-surface) | 5 | 5 |
| Object Size | gdino_depthpro_size | Apparent 2D projection | 3 | 3 |
| Object Size | bbox_3d_size | 3D OBB extent (rigid invariant) | 2 | 2 |
| Rel. Distance | bbox_3d_reldist | 3D OBB (nearest-instance surface) | 14 | 16 |
| Rel. Direction | bbox_3d_reldir | 3D OBB (egocentric bearing angle) | 7 | 8 |
Table A4 : The four core toolsets, plus the 2D + depth baselines. Direct 3D box pipelines compute surface-to-surface relationships and rigid invariants. Median Steps ( Sec. F ; GT backend for the 3D pipelines, real detector for the 2D) equals Min. Steps on every numeric pipeline and exceeds it only on the two multiple-choice tasks, a benign overhead: a premature bearing call that errors and is retried (rel. direction), and redundant distance calls (rel. distance).
Figure A1 : Detector backends behind detect_object_3d on one object (dishwasher, ARKitScenes 45663164, best frame 10). Top four panels: each backend’s predicted 3D box (solid) with the ground-truth box (dashed cyan) reprojected onto the best frame; is the world-space center error. Bottom: the same boxes as top-down ground-plane footprints. WildDet3D ( m) nearly coincides with GT, OVMono3D ( m) is oversized and offset, and Cube R-CNN ( m) drifts onto the sink. The tool-calling chain is identical across backends; only the box origin changes, so this gap is the perception bottleneck of Sec. 4 .
Figure A2 : Intermediate signals of the 2D + depth pipeline ( gdino_depthpro_abs_dist ) on a wardrobe/wall-picture distance query (ARKitScenes 41069042). Top: each object’s best frame with its Grounding DINO 2D box. Bottom: the metric depth map Depth Pro predicts for that frame (bright near, each map scaled to its own range), with the same box and the sampled center pixel (“ ”); the pipeline keeps only the median depth inside the box ( m for the wardrobe, m for the wall picture). Back-projecting the two box centers yields a center-to-center distance ( m) that overshoots the surface-to-surface ground truth ( m), the systematic bias the OBB pipeline corrects. Depth Pro also reads the bedspread in the lower right as far (dark) although it is the nearest surface: monocular depth degrades on large textureless regions, the weakness behind the error tail in Fig. A4 .
Figure A3 : Mean vs. median MRA across every absolute-distance pipeline (401 questions); the gap between the bars measures tail sensitivity. Only WildDet3D beats the no-tool baseline: OVMono3D ( ), Cube R-CNN ( ) and the 2D+depth pipeline ( ) all land below it ( ), so a poor box is worse than no box. OVMono3D’s median is , i.e. more than half its answers are off by over . WildDet3D (mean , median ) approaches the GT oracle ( / ).
Figure A4 : The same pipelines in metric error , which corroborates the MRA ranking. The weak 3D backends are worse than the no-tool baseline (MAE m for OVMono3D and m for Cube R-CNN, against m), while WildDet3D ( m) and the GT oracle ( m) sit far below it. Every OBB pipeline stays bounded (max error m), so their means are trustworthy. The 2D+depth pipeline is the exception: monocular depth blows up on boundary boxes to a worst error of m, leaving an MAE ( m) thirty times its own median ( m).
Figure A5 : Per-sample MRA distributions on absolute distance (401 questions) for the no-tool visual baseline and each detect_object_3d backend. The GT oracle concentrates at (median ) and WildDet3D tracks it (median ), whereas OVMono3D and Cube R-CNN pile up at zero: their medians ( and ) sit below the no-tool baseline’s ( ), so over half of OVMono3D’s answers are more than off. Tool augmentation only pays off once the detector is strong enough.
Figure A6 : Object size from a 2D box ( gdino_depthpro_size ) on six ReVSI objects: the Grounding DINO box (red) and mask (cyan) with the longest image-plane axis (yellow). Panel titles give GT and both back-projected estimates in cm, mask-based and box-based ( bbox , the reported variant). Back-projection sees only the image-plane dimensions, so it is accurate when the longest axis lies in that plane (kitchen sink , couch ) and fails in two ways: the box covers only part of the object (floor lamp : only the shade is grounded), or it lands on the wrong surface (dresser ). This variance, not a constant bias, caps the 2D size pipeline at MRA.
Figure A7 : Minimal correct chain vs. observed tool-call count per question type (GT backend). Stars mark the minimal chain ( , structural per question); boxes summarize the tool- using runs (median line, IQR), and jittered points show every run including the zero-tool band (annotated %). Numeric tasks run the minimal chain exactly; relative direction runs (a premature relative_direction call that errors and is retried) and relative distance (redundant distance calls). The zero-tool rate on relative direction is the orchestration discipline gap, separate from this overhead.
| Subtype | N | GT Oracle | WildDet3D |
|---|---|---|---|
| Backward Easy (3-way) | 53 | 79.2% | 75.5% |
| Backward Hard (4-way) | 69 | 84.1% | 68.1% |
| Forward Easy (3-way) | 96 | 82.3% | 74.0% |
| Forward Hard (4-way) | 72 | 73.6% | 76.4% |
| Overall | 290 | 80.0% | 73.4% |
Table A5 : Relative-direction accuracy by subtype (“easy” = 3-way left/right/back; “hard” = 4-way quadrant; “backward” = the away facing mode). The GT oracle averages tool calls per question over all runs, or (median ) over the runs that call any tool at all; the gap to the geometric ceiling is the zero-tool guessing rate.
| Condition | MRA | Dist. | Size | Errors | Zero-tool |
|---|---|---|---|---|---|
| 1. Terse tool errors | 0.729 | 0.575 | 0.883 | 138 | 5/80 |
| 2. + actionable errors | 0.800 | 0.878 | 0.723 | 36 | 14/80 |
| 3. + few-shot examples | 0.933 | 0.938 | 0.928 | 13 | 0/80 |
Table A6 : The staged ablation on tool feedback (80 questions: 40 absolute distance, 40 object size; GT backend). Errors counts tool results returned as errors across all traces; Zero-tool counts runs that answered without calling a single tool. Better feedback removes the looping, and here the few-shot examples remove the remaining guessing; on the four-task probe ( Tab. 4 ) they no longer help.