See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs

Yongchang Zhang; Oliver Ma; Tianyi Liu; Guangquan Zhou; Yang Chen

cs.CVFeb 25, 2026

See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs

Authors: Yongchang Zhang, Oliver Ma, Tianyi Liu, Guangquan Zhou, Yang Chen

Organizations: Southeast University · AIIA, Ministry of Education, China · University of Oxford

Abstract

Recent large vision-language models (LVLMs) have demonstrated impressive reasoning ability by generating long chain-of-thought (CoT) responses. However, CoT reasoning in multimodal contexts is highly vulnerable to visual hallucination propagation: once an intermediate reasoning step becomes inconsistent with the visual evidence, subsequent steps-even if logically valid-can still lead to incorrect final answers. Existing solutions attempt to mitigate this issue by training models to "think with images" via reinforcement learning (RL). While effective, these methods are costly, model-specific, and difficult to generalize across architectures. Differently, we present a lightweight method that bypasses RL training and provides an iterative, training-free, plug-and-play framework for visually-grounded multimodal reasoning. Our key idea is to supervise each reasoning step at test time with visual evidence, ensuring that every decoded token is justified by corresponding visual cues. Concretely, we construct a textual visual-evidence pool that guides the model's reasoning generation. When existing evidence is insufficient, a visual decider module dynamically extracts additional relevant evidence from the image based on the ongoing reasoning context, expanding the pool until the model achieves sufficient visual certainty to terminate reasoning and produce the final answer. Extensive experiments on multiple LVLM backbones and benchmarks demonstrate the effectiveness of our approach. Our method achieves 16.5%-29.5% improvements on TreeBench and 13.7% RH-AUC gains on RH-Bench, substantially reducing hallucination rates while improving reasoning accuracy without additional training.

Figures & tables

Figure 1: Reasoning pattern comparison. (a) Greedy decoding: the base VLM selects the top-1 token at each step; any hallucination in an intermediate step propagates to an incorrect final answer. (b) RLHF-based “think-with-images”: the model learns when to call tools to zoom or crop the image and re-inject cropped regions into the reasoning context—effective but costly and model-specific. (c) Ours: a lightweight, training-free, model-agnostic framework. A supervisor maintains a dynamic visual-evidence pool to detect and correct hallucination steps. When uncertainty arises, it invokes a visual decider to extract new evidence, enabling visually grounded reasoning throughout the chain.

Figure 2: Overview of evidence-constrained reweighting decoding (ECRD) at decoding step $i$ . The base VLM emits a top-k candidate set; the supervisor builds an evidence-induced distribution from the current evidence pool and negotiates with the base probabilities to reweight candidates. If confidence remains low, the visual decider reads the image with the current prefix, commits a token, and adds a short textual evidence for later steps.

		Perception					Reasoning
	Overall	Attr.	Mater.	Phys.	ObjRet.	OCR	Persp.	Order.	Cont.&Oc.	Contain.	Compar.
Private Models
Gemini-2.5-Flash-0520 [ 2 ]	45.9	48.3	53.9	69.6	68.8	75.0	15.3	19.3	56.1	72.4	43.2
GPT-4o-1120 [ 15 ]	46.9	51.7	61.5	65.2	43.8	69.1	18.8	38.6	48.8	72.4	43.2
Gemini-2.5-Pro-0605 [ 3 ]	54.1	51.7	61.5	56.5	75.0	83.8	20.0	36.8	65.9	86.2	54.6
o3-0416 [ 16 ]	54.8	69.0	69.2	65.2	68.8	79.4	22.4	38.6	61.0	86.2	50.0

Table 1: Results of different models on TreeBench. Best performances for open-source models are highlighted in bold . ECRD consistently improves open-source general models across architectures and scales, confirming training-free plug-and-play applicability.

Method	Base	Woodpecker	ViperGPT	ControlMLLM	Beam	Self-cons.	Diverse	ECRD
Overall	37.0	36.3	38.3	40.5	38.8	39.0	38.3	47.9

Table 2: Comparison with training-free baselines and decoding alternatives on TreeBench. Best result are highlighted in bold . ECRD performs markedly better among all plug-and-play settings.

		Perception					Reasoning
	Overall	Attr.	Mater.	Phys.	ObjRet.	OCR	Persp.	Order.	Cont.&Oc.	Contain.	Compar.
GRIT-3B [ 4 ]	30.1	31.0	23.1	56.5	25.0	39.7	21.2	17.5	36.6	48.3	20.5
Qwen2.5-VL-7B [ 1 ]	37.0	55.2	53.8	56.5	62.5	27.9	20.0	35.1	39.0	44.8	43.2
Qwen2.5-VL-7B + VDGD	39.5	58.6	53.8	52.2	62.5	50.0	17.6	29.8	39.0	48.3	40.9
Qwen2.5-VL-7B + supervisor	40.7	58.6	53.8	56.5	62.5	51.5	18.8	31.6	41.5	44.8	43.2
Qwen2.5-VL-7B + ECRD (Qwen2.5-VL-3B as visual decider)	43.7	62.1	53.8	69.6	62.5	52.9	20.0	31.6	43.9	62.1	43.2

Table 3: TreeBench ablations on Qwen2.5-VL-7B. Best performances are highlighted in bold . The supervisor provides a stable boost and the visual decider adds the remaining lift; both components are necessary and complementary.

Model	RH-Bench
	Reas.	Perc.	RH-AUC
Qwen2.5-VL-7B [ 1 ]	39.6	50.2	0.51
Qwen2.5-VL-7B + supervisor	42.0	53.3	0.54
Qwen2.5-VL-7B + ECRD	46.4	57.1	0.58

Table 4: Results on RH-Bench. Reas. indicates Reasoning , and Perc. indicates Perception . Best performances are highlighted in bold . ECRD improves both Reasoning and Perception while increasing RH-AUC, indicating a better balance between reasoning and hallucination over longer chains.

Model	V* Bench			MathVista	ChartQA	OCRBench	HallusionBench
	Attr	Spatial	Overall
LLaVA-OneVision-7B [ 8 ]	73.0	60.5	68.1	63.2	80.0	62.2	55.1
LLaVA-OneVision-7B + ECRD	74.8	65.8	71.2	67.9	86.3	73.8	63.7
Qwen2.5-VL-7B [ 1 ]	73.9	67.1	71.2	68.2	84.4	82.3	61.3
Qwen2.5-VL-7B + supervisor	73.9	71.1	72.8	69.8	86.8	85.7	67.5
Qwen2.5-VL-7B + ECRD	74.8	75.0	74.9	72.3	88.3	90.7	72.5

Table 5: Results on five general multimodal benchmarks. Better performances are highlighted in bold . ECRD brings consistent gains across diverse tasks on both Qwen2.5-VL-7B and LLaVA-OneVision-7B, demonstrating backbone-agnostic, task-general effectiveness.

Figure 3: Analysis of the uncertainty threshold

\delta

: accuracy as a function of

\delta

across five benchmarks, and the average visual decider invocation rate (calls per question) as

\delta

varies. The gray dashed line marks

\delta=0.08

Benchmark	V* Bench	MathVista	ChartQA	OCRBench	HallusionBench
$t_{0}$	8.98	12.92	9.76	3.24	11.67
$l_{0}$	1.32	1.46	1.30	1.12	1.43

Table 6: Average time per question at

\delta=0

(

t_{0}

, seconds per question) and global average latency of a single visual decider call (

l_{0}

, seconds). Note that

l_{0}

is computed as an overall mean across all benchmarks and

\delta

settings, and is not restricted to the

\delta=0

condition. All tests are conducted on a single H20-NVLink GPU.

Figure 4: Two typical application cases of ECRD.

Figure 5: Breakdown of ECRD’s gain on TreeBench based on Qwen2.5-VL-7B.

Explore similar work

Distilling Visual Reasoning into Text Space
Sep 28, 2026Wenhan Yang, Nilay Naharas, Ali Payani +1CoT Distillation VLM Distillation
University of California, Los Angeles · Cisco Research
SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows
Oct 1, 2026Albert Gao, Bing Xue, Andrea ZanetteVisual Reasoning Latent Visual Reasoning
Carnegie Mellon University · Meta
Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification
Mar 27, 2026Shuai Lv, Chang Liu, Feng Tang +5Multimodal Large Language Models Multimodal Reasoning
University of Science and Technology of China, Hefei, China · Huawei Foundation Model Department, Shanghai, China · The Chinese University of Hong Kong, Hong Kong, China +1

Sep 28, 2026J/K move · Enter open · S save

Distilling Visual Reasoning into Text Space

Wenhan Yang, Nilay Naharas, Ali Payani, Baharan Mirzasoleiman

University of California, Los Angeles · Cisco Research

CoT Distillation VLM Distillation Large Vision-Language Models VLM Reasoning

Open paper AskPDF arXiv

Large Vision-Language Models (LVLMs) have shown strong promise for multimodal reasoning, yet often struggle with tasks requiring concepts beyond what is directly observable in the input image. Existing methods generate intermediate images or latent visual tokens to guide reasoning, but these representations can introduce errors and increasingly interfere with textual reasoning as reasoning progresses. We propose Visual-to-Text Chain-of-Thought Distillation (V2T), a framework that enables LVLMs to internalize visual reasoning without generating intermediate visual representations at inference time. V2T first trains a teacher LVLM using interleaved visual and textual chains of thought, and then uses knowledge distillation to train a student LVLM using the teacher's logits and cross-entropy supervision from ground-truth textual reasoning. When reasoning images can be mapped to the original image, V2T can additionally distill the teacher's attention to corresponding regions, while ground-truth bounding boxes can further guide a subsequent reinforcement learning stage. Experiments across multiple multimodal reasoning benchmarks show that V2T consistently outperforms the teacher and existing baselines, improving average accuracy by 14.3% on a held-out set and 2.7% on the broader visual evaluation suite. Moreover, lightweight SFT and substantially reduced RL make V2T up to 42x faster to train than state-of-the-art baselines.

Abstract

Explore similar work

Distilling Visual Reasoning into Text Space

SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows

Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Verification