Where and Why is Grounded Reasoning Useful?

David Yan; Le Xue; Forrest Huang; Qiuyi Zhang; Dustin Tran; Andrew M. Dai; Yinfei Yang

cs.CVNov 5, 2026

Where and Why is Grounded Reasoning Useful?

Authors: David Yan, Le Xue, Forrest Huang, Qiuyi Zhang, Dustin Tran, Andrew M. Dai, Yinfei Yang

Organizations: Elorian · Princeton University

Abstract

Grounded reasoning, where a model outputs text coordinates of relevant locations in chain-of-thought reasoning, is a common strategy for solving complex visual reasoning problems. However, the benefits and limitations of grounded reasoning are understudied. We study where and why grounded reasoning is most useful. To do so, we introduce a chain-of-thought analysis method to measure how much a model uses its outputted grounding in its thinking trace and perform a detailed ablation study of a grounded reasoning training recipe. We find that grounded reasoning improves counting but shows no significant gain over baselines without coordinate grounding across a wide range of spatial reasoning and perception benchmarks. Using several synthetic tasks, we show that grounded reasoning is most useful when a task requires undescribable references, which are visual concepts that are difficult to describe in language. However, we observe that many existing visual reasoning benchmarks are solvable with only describable text references. To close this gap, we introduce BrickVision, a new benchmark that requires undescribable visual references to solve.

Figures & tables

Figure 1: Visual grounding is useful as an undescribable reference . When objects in a task are uniquely identifiable in text (e.g. through a label, descriptor, or position), we say that a textual reference is available. When textual references are available, grounded reasoning performs no better than text reasoning. When no unique text description is available (e.g. in dense counting or tracing tasks), we say the task requires maintaining undescribable references . Grounded reasoning significantly outperforms text reasoning when the task requires undescribable references.

Figure 2: Decode Grounding Corruption . We introduce a CoT analysis technique to analyze whether a model uses its grounding. Instead of editing CoT after generation, we randomize the model’s outputted points during decoding and reset the KV-cache.

Open models
Model	MultihopSpatial	SpatialMQA	V ∗	CountQA
TreeVGR-7B-CI	$-0.62\pm 0.33$	$-0.50\pm 0.28$	$+0.55\pm 0.71$	$-0.18\pm 0.26$
TreeVGR-7B-RL	$-0.17\pm 0.22$	$-0.21\pm 0.19$	$-0.08\pm 0.35$	$-3.19\pm 0.22$
iVGR-8B	$-0.93\pm 0.29$	$-0.41\pm 0.32$	$-0.99\pm 0.51$	$-2.64\pm 0.29$
DeepEyes-7B	$+0.63\pm 0.24$	$+0.02\pm 0.26$	$-0.68\pm 0.38$	$-0.50\pm 0.24$
Ours (format type)

Table 1: Grounding usage experiments. We report change in mean performance (% acc) for our coordinate corruption experiments. Many open models don’t rely on their grounding for spatial reasoning and perception. Our retrained models (below) are more reliant on their grounding outputs.

Figure 3: Training Coordinate Randomization . We randomize varying percentages of coordinates in the SFT set and report task performance and grounding usage. While grounding usage decreases with more randomization, task performance remains relatively flat, except on counting.

Benchmark	Appended	Reflection	$\Delta$
SMQA	0.529	0.527	$-0.2\pm 1.2$
MHS	0.544	0.539	$-0.5\pm 1.2$
V ∗	0.839	0.844	$+0.5\pm 4.5$
MME-RW	0.533	0.531	$-0.2\pm 1.0$
TreeBench	0.451	0.467	$+1.6\pm 2.0$
CountQA	0.300	0.327	$\mathbf{+2.7\pm 1.9}$

Table 2: Appended vs reflection performance . The reflection reasoning style yields significantly better performance on counting.

		SFT			RL
Benchmark	$n$	Ungrounded	Grounded	$\Delta$	Ungrounded	Grounded	$\Delta$
Spatial reasoning
SMQA	1076	0.537	0.527	$-1.0\pm 2.1$	0.571	0.562	$-0.9\pm 1.8$
MHS	1500	0.551	0.539	$-1.3\pm 1.3$	0.615	0.588	$\mathbf{-2.7\pm 1.9}$
EmbSpatial	3640	0.730	0.733	$+0.2\pm 0.7$	0.791	0.775	$\mathbf{-1.7\pm 0.9}$
VSR	1222	0.803	0.801	$-0.2\pm 1.5$	0.789	0.733	$\mathbf{-5.5\pm 1.3}$

Table 3: Training recipe comparison . We report the performance of grounded reasoning and text-only reasoning training recipes after SFT and RL training phases. Deltas that exclude 0 from the confidence interval are highlighted. We find that grounding provides a statistically significant benefit on counting, but not on spatial reasoning or perception.

Figure 4: Synthetic Probe Tasks . We use three synthetic tasks to study where grounded reasoning improves over pure text reasoning. We find that grounding performs significantly better than text reasoning when no text-labels are available, but worse when given text labels. Zoom in for detail.

Figure 5: BrickVision . Samples from each task distribution of our BrickVision evaluation set. BrickVision tasks involve iterative comparison across objects difficult to describe in language, and require precise answers in the form of a point or integer.

Model	Crop ID	Fine-grained counting	Duplicate	Unpaired	Overall
GPT-6 Astra ( OpenAI, 2026b )	90.0	25.2	74.8	84.8	68.7
Claude Opus 5.5 ( Anthropic, 2026a )	40.4	16.8	19.6	27.6	26.1
GPT-5.6 Sol ( OpenAI, 2026a )	35.2	8.4	11.6	20.4	18.9
Gemini 3.8 Flash ( Google DeepMind, 2026b )	20.4	15.2	8.8	17.6	15.5
Claude Opus 5 ( Anthropic, 2026b )	26.8	10.4	8.4	11.6	14.3
Qwen3.8-Max ( Qwen Team, 2026b )	6.4	2.0	13.2	12.4	8.5

Table 4: VLM performance on BrickVision . Many frontier models struggle on BrickVision, suggesting limited ability to maintain undescribable visual references. Models trained with in-domain grounding data perform better than text reasoning, demonstrating the benefit of grounded reasoning.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	SFT	RL
Epochs	1	5
Batch	64	512 $\times$ 8
LR	$10^{-5}$	$10^{-5}$
Warmup	cosine, 3%	constant
Weight decay	0	0.01
KL / clip	–	0.01 / 0.2

Appendix

Table 5: Training hyperparameters .

Bench	$n$	Point	Box	Box $-$ Point
Spatial Reasoning
SMQA	1076	0.527	0.509	$\mathbf{-1.9\pm 1.5}$
MHS	1500	0.539	0.525	$\mathbf{-1.4\pm 1.2}$
EmbSpatial	3640	0.733	0.728	$-0.4\pm 1.2$
VSR	1222	0.801	0.797	$-0.4\pm 2.1$
OmniSpatial	1533	0.470	0.467	$-0.3\pm 1.7$

Appendix

Table 6: Point versus box grounding after SFT. We report the box minus point accuracy delta with 95% confidence intervals computed as in Appendix A . Bold deltas exclude 0.

Figure 6: SFT format ablation . We compare the appended and reflection SFT formats. We find that the reflection format generally leads to better grounding usage, and use this format in our SFT data.

Figure 7: Incorrect TreeVGR samples . Selected samples from the TreeVGR dataset illustrating errors in data such as incorrect box labeling or uninformative boxes.

Figure 8: Probe reasoning traces . We provide examples of our reasoning data used in our “Arrow Following” task to illustrate the difference between text traces and grounded traces.

Figure 9: Benchmark construction . Our benchmark combines procedural generation with conditional image generation to produce realistic images aligned with the original synthetic ground truth.

Figure 10: Restyling drift . Without locking the aspect ratio or prompting Nano-Banana with bounding boxes, we find that the restyled image drifts from the original, making the segmentation map ground truth (visualized as an overlay) incorrect.

Visually Grounded Reasoning

Visually grounded reasoning links visual evidence to multi-step conclusions and actions, aiming to make multimodal systems accurate, auditable, and robust.

See all 349 papers

Thinking with Visual Grounding
Jun 15, 2026Junkai Zhang, Yihe Deng, Kai-Wei Chang +1Visual Spatial Reasoning Vision-Language Grounding
University of California, Los Angeles
Beyond Referring Expressions: Scenario Comprehension Visual Grounding
Apr 2, 2026Ruozhen He, Nisarg A. Shah, Qihua Dong +3Vision-Language Grounding Visually Grounded Reasoning
Rice University · Johns Hopkins University · Northeastern University
SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Models
Sep 30, 2026Rafi Ibn Sultan, Xiangyu Zhou, Md. Sajid Alam Chowdhury +4Visual Spatial Reasoning Large Vision-Language Models
Department of Computer Science, Wayne State University · Department of Radiation Oncology, Henry Ford Health · Department of Electrical and Computer Engineering, The Ohio State University +1

Jun 15, 2026J/K move · Enter open · S save

Thinking with Visual Grounding

Junkai Zhang, Yihe Deng, Kai-Wei Chang, Wei Wang

University of California, Los Angeles

Visual Spatial Reasoning Vision-Language Grounding VLM Reasoning Visually Grounded Reasoning RL for VLMs

Open paper AskPDF arXiv

Visual thinking should not only sound right; it should show its evidence. While recent vision-language models (VLMs) can produce natural-language reasoning traces, these traces often leave the supporting image regions implicit, making them hard to verify and difficult to supervise. We introduce visually grounded thinking, a reasoning process in which models interleave natural-language thoughts with explicit point or box groundings of the visual evidence used at each step. This lets the model express intermediate reasoning in language while grounding key objects in the image regions they refer to. To train this behavior, we construct a scalable synthesis pipeline that distills correct visual reasoning traces, extracts the visual objects required by the traces, grounds them with a SAM3-based agent, and derives aligned point and box supervision from the resulting masks. We further propose grounding-aware reinforcement learning, which combines answer correctness rewards with dense grounding rewards that score whether generated object references match the correct image evidence. Across two counting benchmarks and four spatial reasoning benchmarks, adding visually grounded thinking to Gemma3-4B-IT consistently improves performance over the original model and the non-grounded thinking baseline. On spatial reasoning, the visually grounded thinking 4B models match, and in some cases surpass, Gemma3-27B-IT from the same model family. Our analysis shows that point grounding is well suited to counting, while box grounding benefits most from explicit grounding rewards on spatial tasks. Overall, our results show that VLMs think better when their intermediate thoughts are tied to the image regions that make them true.

Abstract

Visually Grounded Reasoning

Explore similar work

Thinking with Visual Grounding

Beyond Referring Expressions: Scenario Comprehension Visual Grounding

SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Models