See, Think, Use: Diagnosing Perception and Reasoning Failures in VLMs

Jiacheng Ding; Yanliang Qi; Cong Guo; Peihua Mai

cs.CVNov 1, 2026

See, Think, Use: Diagnosing Perception and Reasoning Failures in VLMs

Authors: Jiacheng Ding, Yanliang Qi, Cong Guo, Peihua Mai

Organizations: University of Memphis · National University of Singapore

Abstract

We present See, Think, Use, a protocol that traces each answer of a vision-language model to perception, reasoning or the use of perception. The diagnosis shows a developer which stage to improve and directs test-time compute to the stage that fails. Benchmarks score the final answer, and an answer that rests on the image earns the same credit as a guess or a memorized prior. The protocol asks the model two extra questions per item and needs no access to its weights. A localization probe asks for a box around every instance and measures seeing, and a text-replacement probe answers from a description and measures thinking. Use is the rate at which the answer follows the model's own perception across counterfactual image pairs. The rate needs no answer labels, because the question and its prior are the same on both images. On two model families and six datasets the protocol shows that the knob that helps follows the task family. Answers follow the model's perception except on images that contradict a common prior, and asking the model to localize first repairs this failure when its boxes are right. Allocating compute by the measured stages comes close to the best uniform policy at lower cost and beats a random allocation of the same compute wherever a knob raises the short stage. We will release the code, the prompts and the per-question outputs.

Figures & tables

Figure 2: The protocol on one counterfactual pair. The model answers the question on the edited image $v$ and on its original $v^{\prime}$ . The model also boxes every instance in probe A and answers from a text in place of the image in probe B. The probes place each answer into one of five outcomes, and each outcome names the knob for its short stage. On the pig the boxes follow the edit and the answer does not, which is a failure of use .

Figure 3: Three ways to diagnose two wrong answers of Qwen3-VL-8B. Output-only scoring marks both as wrong. An image-removal control sees the answer change on a grey image and counts the image as used in both cases. See, Think, Use compares the model’s own boxes across the counterfactual pair. The pig’s boxes follow the edit while the answer stays, and a box remains where the BEAF person was removed. Blue boxes are probe A, and the red dashed box marks a box on an object that is no longer there.

dataset	hard part	questions	groups	image pair
BEAF	object presence	14,275	500	object removed
BEAF subset		1,018	150	object removed
V*	small objects	191	191	none
VLMs-are-Biased	count vs. prior	332	284	original
MathVerse	geometry	436	436	none
DynaMath	math	500	100	none

Table 1: Datasets, all taken from the public releases without new annotation. Groups are the clusters of the bootstrap.

dataset	model	$s_{0}r_{0}$	$s_{1}r_{0}$	$s_{0}r_{1}$	$s_{1}r_{1}$	oracle	$s_{0}$	$s_{1}$	$r_{0}$	$r_{1}$	$r_{0}$	$r_{1}$	no image
		accuracy					probe A		probe B		$B_{\text{self}}$
BEAF subset	Qwen	84.1	86.5	79.7	81.6	91.0	74.9	79.6	–	–	–	–	50.6
BEAF subset	InternVL	75.4	78.1	76.3	79.0	90.4	53.7	74.1	–	–	–	–	50.6
V*	Qwen	59.2	82.7	58.1	82.7	89.5	25.1	69.1	–	–	–	–	33.0
V*	InternVL	52.9	69.6	48.7	51.8	78.5	0.5	2.1	–	–	–	–	35.1
VLMs-are-Biased	Qwen	4.2	8.1	9.6	10.8	18.7	28.6	43.7	100	99.7	–	–	0.3

Table 2: Accuracy in percent in the four cells of the grid, the per-question oracle over these cells, the three probes and the no-image control. Bold marks the best cell of a row. A green or red cell marks a paired change from

s_{0}r_{0}

whose 95 percent interval lies above or below zero. Probe A is right when its boxes lie on the object or match the count.

Figure 4: Localize-then-answer on edited VLMs-are-Biased items for Qwen3-VL-8B at the high budget. Blue boxes are probe A, which the model sees before the question under $G$ . On the first three items the boxes are right, and $G$ corrects the answer that the direct answer $C$ takes from the prior. On the last two the boxes miss a row of the board and a leg of the bird, and $G$ repeats their count.

questions where	model	$n$	$s_{0}r_{0}$	$s_{1}r_{0}$	$s_{0}r_{1}$	$s_{1}r_{1}$
BEAF subset [-1pt] probe A wrong	Qwen	256	46.9	55.9	37.9	48.0
BEAF subset [-1pt] probe A wrong	InternVL	471	64.3	69.4	65.8	67.3
V* [-1pt] probe A wrong	Qwen	143	49.0	77.6	48.3	76.9
V* [-1pt] probe A wrong	InternVL	190	52.6	69.5	48.4	52.1
VLMs-are-Biased [-1pt] probe A wrong	Qwen	237	3.0	7.6	10.1	10.1
VLMs-are-Biased [-1pt] probe A wrong	InternVL	317	7.3	9.5	7.3	3.8

Table 3: Accuracy in percent on the questions where one stage is short at

(s_{0},r_{0})

. A tinted cell marks the knob of the short stage, blue for pixels and orange for chain-of-thought, and bold marks the best cell of a row.

				BEAF		V*		VLMs-Biased
outcome	A	B	C	Qwen	Intern	Qwen	Intern	Qwen	Intern
genuine	✓		✓	75.4	48.6	22.5	0.5	2.1	0.9
right without seeing	✗		✓	10.7	35.5	36.6	52.4	2.1	6.9
seen, not used	✓	✓	✗	2.1	4.3	2.6	0.0	26.5	3.6
think error	✓	✗	✗	–	–	–	–	0.0	0.0
see error	✗		✗	11.8	11.6	38.2	47.1	69.3	88.6

Table 4: Share of questions in percent in each outcome at the cheapest cell

(s_{0},r_{0})

. Bold marks the largest share among the four outcomes other than genuine, and a dash marks a probe that was not run.

	Qwen			InternVL			Qwen-32B
pairs or problems	$r_{0}$	$r_{1}$	$G$	$r_{0}$	$r_{1}$	$G$	$r_{0}$	$G$
BEAF subset	92.2	85.9	99.7	78.0	79.6	99.0	84.2	100
NaturalBench	85.0	–	–	67.8	–	–	–	–
MathVerse	81.9	92.4	–	81.1	86.0	–	–	–
DynaMath	90.2	94.9	–	78.4	87.0	–	–	–
VLMs-are-Biased	10.9	13.6	60.9	24.6	30.4	62.3	19.6	65.2

Table 5: use in percent at

s_{0}

for the direct answer, chain-of-thought and localize-then-answer. The BEAF, NaturalBench and VLMs-are-Biased rows use the image pairs of Sec. 3.3 , and the math rows give the accuracy from the image on the problems that the model answers right from

B_{\text{self}}

	Qwen			InternVL
items at $s_{1}$	$C$	$G$	$G_{\text{crop}}$	$C$	$G$	$G_{\text{crop}}$
V*, all	82.7	82.7	88.5	69.6	69.6	67.0
V*, box drawn	85.5	86.0	92.4	77.1	76.3	72.5
BEAF removed, all	82.1	70.9	76.3	61.9	64.9	64.9
BEAF removed, box drawn	46.7	0.0	18.7	8.4	0.0	0.0
VLMs-are-Biased edited	8.1	29.8	8.1	10.5	7.2	6.0

Table 6: Accuracy in percent at the high budget when the model answers directly, localizes first, or zooms into its own box. A box drawn on a BEAF removed item marks an object that is no longer there. Bold marks the best way per model.

dataset	model	cell	accuracy	tokens	accuracy	tokens	moved	oracle	short stage	$B_{\text{self}}$
		best uniform cell			two-pointer rule				AUC before the step
BEAF subset	Qwen	$s_{1}r_{0}$	86.5	1057	86.3	440	25%	91.0	0.83	–
BEAF subset	InternVL	$s_{1}r_{1}$	79.0	2136	77.8	938	46%	90.4	0.73	–
V*	Qwen	$s_{1}r_{0}$	82.7	2509	80.6	1941	75%	89.5	0.74	–
V*	InternVL	$s_{1}r_{0}$	69.6	1699	69.6	1691	99%	78.5	–	–
VLMs-Biased	Qwen	$s_{1}r_{1}$	10.8	2771	7.5	935	71%	18.7	0.80	–

Table 7: Accuracy in percent and mean tokens per question for the best uniform cell, the offline two-pointer rule with the share of questions it moves, and the per-question oracle. The last two columns give the cross-validated AUC with which signals from the cheapest cell predict the short stage. The short stage is a failed probe A on the first three datasets and a failed probe B on MathVerse, and the last column predicts a failed

B_{\text{self}}

. Bold marks the tokens of the rule.

	BEAF subset		V*		VLMs-Biased		MathVerse
allocation	Qwen	Intern	Qwen	Intern	Qwen	Intern	Qwen	Intern
two-pointer rule	86.3	77.8	80.6	69.6	7.5	9.9	72.0	65.8
knobs swapped	81.8	76.1	58.6	48.7	9.3	7.8	56.0	60.1
random, same mix	84.7	76.7	76.8	69.5	7.0	10.4	63.7	62.3
no $B_{\text{self}}$ step	–	–	–	–	–	–	72.2	64.4

Table 8: Ablations. On the left, accuracy in percent of the offline allocation in three variants. The same questions receive the other knob, random questions receive the same mix of cells at the same expected tokens, or MathVerse drops the

B_{\text{self}}

step. On the right, three ways to measure use at

(s_{0},r_{0})

on all BEAF pairs and on VLMs-are-Biased. The first two rows give the pair measure with a pair seen by any box or only by correct boxes. The third row gives the share of answers that a grey image changes, and the last two give the share of edited items that receive the familiar count. A dash marks a quantity that does not apply or rests on fewer than 20 pairs.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 1,017 papers

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
Sep 28, 2026Rong Yu Xu, Prayag Tiwari, Shaolei ZhangEfficient VLM Inference VLM Interpretability
Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China
From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models
May 19, 2026Juncheng Wu, Hardy Chen, Haoqin Tu +6Vision-Language Models VLM Reasoning
Amazon · UC Santa Cruz · University of Waterloo +1
Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning
May 13, 2026Haozhe Wang, Qixin Xu, Changpeng Wang +4Multimodal Reasoning VLM Reasoning
1Hong Kong University of Science and Technology (HKUST) · 2Tsinghua University · 3Zhejiang University +2

Sep 28, 2026J/K move · Enter open · S save

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

Rong Yu Xu, Prayag Tiwari, Shaolei Zhang

Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China

Efficient VLM Inference VLM Interpretability Latent Visual Reasoning VLM Reasoning Reasoning for VLMs Compositional Reasoning in VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.

Abstract

VLM Evaluation

Explore similar work

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models

Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning