Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act

Yuyang Dai; Bofei Huang; Hongbo Zhang; Haoran Xie

cs.CVSep 18, 2026

Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act

Authors: Yuyang Dai, Bofei Huang, Hongbo Zhang, Haoran Xie

Organizations: Florida State University

Abstract

Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they have already identified correctly. We distinguish perceptual failure, where relevant evidence is not recognized, from process failure, where recognized evidence fails to constrain the final decision. We introduce VPAC-Bench, a benchmark spanning nine real-image process families, with each image annotated by its current activity stage and nearby stage transition. We also propose State-Relevance-Target (SRT), a family of structured process-prior interventions that requires models to connect visible evidence to the relevant process state before answering. Across multiple VLMs, process failure is widespread: models that correctly enumerate visual candidates still over-commit to a single answer in more than 95% of ambiguous cases. An explicit process-structured intervention reduces this rate to below 13% without degrading performance on unambiguous cases. However, the transfer of process priors is model-dependent, and generic SRT does not consistently outperform strong chain-of-thought baselines. When the relevant stage transition is known, boundary-aligned SRT substantially outperforms generic process prompting and all tested chain-of-thought baselines across assembly, physical state transition, navigation and traffic, and object-use affordance tasks. These results show that process priors are most useful when aligned with the scene's specific decision boundary, motivating boundary-aware prior selection for process-grounded visual reasoning.

Figures & tables

Figure 1: Overview of our framework. Top left : a motivating example showing process failure, the model correctly identifies multiple car candidates yet still commits to a single answer, violating its own observations. Top center : our benchmark ( VPAC-Bench ) covers nine real-image process families (sports action, mobility, tool use and repair, household activities, drawing & craft, assembly, physical transition, navigation, and object-use). Bottom : the SRT framework applied to a skiing image. State extraction (S) identifies the decision-relevant visible entities (two skiers, equipment, motion cues). Relevance checking (R) filters to decision-relevant relations only (person–ski contact ✓, body posture ✓) and suppresses irrelevant ones (number of ski tracks, backpack brand

\oslash

). Target commitment (T) commits only to the stage supported by visible evidence ( pre_action_setup ), ruling out active_execution because no strong motion cue or dynamic posture is visible.

Component	S1: Referent Disambiguation	S2: Spatial Reasoning	S3: Process Families
S State Extraction	Visible candidate referents and distinguishing features	Subject, reference object, and visible geometric layout	Actor, tool, manipulated object, and object-state cues
R Relevance Checking	Candidate uniqueness	Relation-specific evidence (support, enclosure, orientation, …)	Stage-boundary compatibility (actor–tool, tool–object, object–state)
T Target Decision	Answer, enumerate, or clarify	True, false, or uncertain	Current stage label
Main failure	Over-answering under referent ambiguity	Relation guessing without layout checking	Activity-label guessing without stage checking

Table 1: The three SRT invariants across the three VPAC stages. In S1, this structure is embodied by an existing inspect–judge–decide method whose effectiveness motivates SRT; S2 and S3 evaluate SRT directly.

Figure 2: Process failure is distinct from perceptual failure (COCO Task 1, five models). Providing visual annotations without a decision process ( marked ) leaves policy accuracy near the default baseline, while adding an inspect–judge–decide structure on top of the same visual access ( VS ) raises it above 87% for four of five models.

Model	Data	Default	Evidence	Marked	VS
GPT-5.4-mini	COCO	0.0/98.5	45.1/54.7	0.7/98.9	88.8 /9.2
	GQA	0.0/97.0	35.0/65.0	3.0/97.0	91.0 /6.0
Qwen2.5-VL-72B	COCO	3.5/95.8	44.7/48.9	9.8/90.2	91.0 /8.5
	GQA	5.0/91.0	52.0/47.0	12.0/88.0	86.0 /13.0
GLM-4.6V	COCO	0.0/99.0	37.0/62.0	5.0/95.0	87.0 /12.0
	GQA	0.0/100.0	42.0/57.0	31.0/69.0	92.0 /7.0

Table 2: S1 Task 1 policy accuracy / over-answer rate (%) on COCO and GQA. Bold indicates the highest policy accuracy in each row. VS = visual sketchpad .

Figure 3: Process-prior transfer is model-sensitive across five frontier VLMs on VSR. Each line traces one model’s accuracy as the SRT condition progresses from default through generic, relation-routed, and model-specific instantiations. Cross-model points are dev100 (

n

=100); Qwen’s relation-routed point is test400 (

n

=400).

Figure 4: Oracle boundary-sensitive SRT versus generic SRT on three of the four standardized process families (

n

=60 per family per model, five-model averages); results for the fourth family (object-use affordance) are reported in Table C4. The rightmost panel shows a single process-overload case (GLM, boundary P7) and is not a family-level result.

Condition	Phys.	Assem.
Default	29.4	44.4
Visual evidence only	51.1	45.6
Policy only ( $T$ )	15.6	33.9
$S+T$	27.8	40.6
$R+T$	41.1	41.6
$S+R$ , no $T$	35.0	40.0

Table 3: Ablation accuracy (%), averaged across three models (

n=180

per condition and family). † denotes boundary-alignment ablation; ‡ denotes routing-feasibility ablation. Wilson 95% intervals are reported in Table D1.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Dataset / Family	Items	Models
S1	COCO Task 1 (ambiguous)	457 ∗	5
S1	COCO Task 2 (unique)	330 ∗	5
S1	GQA Task 1 (ambiguous)	100	5
S1	GQA Task 2 (unique)	96	5
S2	VSR dev100	100	5
S2	VSR test400 (Qwen)	400	1

Appendix

Table 4: Nominal evaluation sample sizes per stage and dataset. S3 family-level results (

n

=60 per model) are treated as diagnostic evidence; aggregate trends across families and models form the primary basis for our conclusions.

Field	Meaning
gold_step_label	Current visible process stage.
boundary_family	Process boundary being tested.
visible_actor	Whether the relevant actor is visible.
neighbor_step_label	Most confusable adjacent process stage.
why_not_neighbor	Visible evidence ruling out adjacent stage.
issue_type	Ambiguity, occlusion, or image-quality flag.

Appendix

Table 5: Main annotation fields used for process-family examples.

Family	Generic SRT	Self-router	Oracle router	Self $-$ Generic
Physical	46.1	47.8	86.1	+1.7
Assembly	42.2	42.8	85.0	+0.6

Appendix

Table 6: Oracle boundary routing versus self-routing (accuracy %, averaged across GPT-5.4-mini, Qwen2.5-VL-72B, and Mistral-Small-3.2; parseable-only convention). Self-router gains over generic SRT are not statistically established; intervals in Table D1.

Family	Default	Label-only	Oracle SRT	CSR	$\bm{\Delta}$
Standardized (paired)
Physical	26.3 [21.7,31.6]	55.7 [50.0,61.2]	76.7 [71.6,81.1]	+29.3	+21.0
Assembly	46.2 [40.1,52.6]	55.8 [49.5,62.0]	78.8 [73.1,83.5]	0 +9.6	+22.9
Traffic	47.1 [40.9,53.4]	43.3 [37.2,49.7]	71.2 [65.2,76.6]	0 $-$ 3.8	+27.9
Affordance	37.1 [31.2,43.4]	33.3 [27.7,39.5]	57.5 [51.2,63.6]	0 $-$ 3.8	+24.2
Supporting (unpaired, parseable-only)

Appendix

Table 7: B1: Decomposition of the oracle boundary router gain. The four standardized families are computed on the three-way paired intersection with unparseable = incorrect (Appendix A.1); physical uses five models (

n

=300) and the other three use four (

n

=240). Supporting families use the parseable-only convention and are not paired here; see Table B1 ′ for the paired version of Cleaning, Craft, and Mobility. CSR = Label-only

-

Default;

\Delta

= Oracle SRT

-

Label-only.

Family	$n$	$b$	$c$	$c-b$	$\chi^{2}$	$p$
Physical	300	29	92	63	31.8	$<$ 0.001
Assembly	240	20	75	55	30.7	$<$ 0.001
Traffic	240	14	81	67	45.9	$<$ 0.001
Affordance	240	18	76	58	34.6	$<$ 0.001

Appendix

Table 8: B2: McNemar test, label-only vs. oracle SRT, computed from item-level paired responses (unparseable = incorrect).

b

= label-only correct and oracle wrong;

c

= oracle correct and label-only wrong;

c-b

equals the net item-count gain in the paired set.

\chi^{2}

with continuity correction.

Family	$n$	Label-only	Oracle SRT	$\bm{\Delta}$
Cleaning	280	46.4 [40.7,52.3]	43.2 [37.5,49.1]	$-$ 3.2
Craft	231	79.2 [73.5,84.0]	42.9 [36.6,49.3]	$-$ 36.3
Mobility	530	53.4 [49.1,57.6]	46.4 [42.2,50.7]	$-$ 7.0

Appendix

Table 9: B1 ′ : Label-only versus oracle SRT on the three reversed families, paired at the item level (unparseable = incorrect, five models), the same convention as Table B1.

Family	$n$	$b$	$c$	$\chi^{2}$	$p$
Cleaning	280	75	66	0.5	0.50
Craft	231	88	4	74.9	$<$ 0.001
Mobility	530	159	122	4.6	0.032

Appendix

Table 10: B2 ′ : McNemar test, label-only vs. oracle SRT, three reversed families.

b

= label-only correct and oracle wrong;

c

= oracle correct and label-only wrong; note

b>c

throughout, the reverse of Table B2. Exact binomial test for Craft given the extreme imbalance;

\chi^{2}

with continuity correction otherwise.

Four of five models show a positive margin; GLM is the exception, consistent
Model	Default	Label-only	Oracle SRT	$\bm{\Delta}$
GPT-5.4-mini	35.5	73.8	84.6	+10.9
Qwen2.5-VL-72B	37.1	69.5	76.7	+7.2
Gemini-3.1-Pro	32.5	57.5	67.7	+10.2
GLM-4.6V	44.6	61.8	60.4	$-$ 1.4
Mistral-Small-3.2	38.3	63.3	80.0	+16.7

Appendix

Table 11: B3: Label-only vs. oracle SRT by model, micro-averaged across the standardized families for which that model has oracle outputs (parseable-only convention). Mistral covers physical state transition and assembly.

Family	Def.	Best CoT	Generic	Oracle	$\bm{\Delta}$
Physical	26.7	44.3	38.3	80.0	+35.7
Assembly	44.7	50.0	44.0	79.7	+29.7
Traffic	52.7	52.7	48.3	74.3	+21.7
Affordance	37.0	39.3	32.7	59.7	+20.4
Aggregate	40.3 [37.6,43.1]	46.6 [43.8,49.4]	40.8 [38.1,43.6]	73.4 [70.8,75.8]	+26.9

Appendix

Table 12: B4: SRT vs. CoT-style baselines (accuracy %, five-model averages, parseable-only convention). Best CoT = best of five CoT conditions. Wilson 95% CIs given for the pooled row (

n

=1,200).

Family	App. B (paired)	Main text / App. C
Physical	76.7 ( $n$ =300)	80.0 ( $n$ =300)
Assembly	78.8 ( $n$ =240)	79.7 ( $n$ =300)
Traffic	71.2 ( $n$ =240)	74.3 ( $n$ =300)
Affordance	57.5 ( $n$ =240)	59.7 ( $n$ =300)

Appendix

Table 13: C1: Correspondence between the two accuracy conventions, for the oracle SRT condition. Appendix B pairs items across three conditions and counts unparseable responses as incorrect; the main text and the remainder of Appendix C use all parseable responses from all available models.

Set	Task	$n$	Default	Evidence	Marked	VS
COCO	T1	1214–1219 †	2.5 [1.7,3.5]	45.3 [42.4,48.2]	10.7 [9.1,12.6]	90.4 [88.6,91.9]
COCO	T2	865–961 ‡	96.3 [94.8,97.4]	96.1 [94.7,97.2]	99.5 [98.8,99.8]	99.6 [98.9,99.8]

Appendix

Table 14: C2: COCO T1/T2 pooled accuracy under the parseable-only convention. Actual pooled

n

per condition, not the nominal

5{\times}457{=}2{,}285

5{\times}330{=}1{,}650

, because three models’ evaluation files are capped at 100 items; see footnote. T1 correct policy = not-answer (clarify or enumerate); T2 correct policy = answer.

Model	Split	Condition	Accuracy [95% CI]
Qwen2.5-VL-72B	dev100	default	76.0 [66.8, 83.3]
Qwen2.5-VL-72B	dev100	evidence	85.0 [76.7, 90.7]
Qwen2.5-VL-72B	dev100	verify	80.0 [71.1, 86.7]
Qwen2.5-VL-72B	test400	default	78.5 [74.2, 82.2]
Qwen2.5-VL-72B	test400	evidence	78.5 [74.2, 82.2]
Qwen2.5-VL-72B	test400	verify	77.5 [73.2, 81.3]

Appendix

Table 15: C3: S2 VSR accuracy with Wilson 95% confidence intervals. Each row is a condition that was evaluated; conditions not run for a model are omitted rather than listed as empty. All cross-model comparisons use dev100; test400 was run for Qwen only. Gemini’s default-to-verify difference is the only S2 comparison whose intervals are disjoint.

Family	Default	Generic SRT	Scenario SRT	Boundary Router
Physical	26.7 [22.0, 31.9]	38.3 [33.0, 43.9]	37.7 [32.4, 43.3]	80.0 [75.1, 84.1]
Assembly	44.7 [39.1, 50.3]	44.0 [38.5, 49.7]	39.7 [34.3, 45.3]	79.7 [74.8, 83.8]
Traffic	52.7 [47.0, 58.2]	48.3 [42.7, 54.0]	41.7 [36.2, 47.3]	74.3 [69.1, 78.9]
Affordance	37.0 [31.7, 42.6]	32.7 [27.6, 38.2]	30.7 [25.7, 36.1]	59.7 [54.0, 65.1]
Cleaning	38.2 [32.7, 44.0]	29.3 [24.3, 34.9]	28.9 [23.9, 34.5]	45.4 [39.6, 51.2]
Craft †	44.9 [40.0, 49.9]	45.5 [40.6, 50.4]	51.2 [46.2, 56.1]	49.1 [44.1, 54.1]

Appendix

Table 16: C4: S3 process-family accuracy with Wilson 95% confidence intervals, five-model averages, parseable-only convention (

n

=300 per cell for the four standardized families). These are the values reported in the main text. See Table B1 ′ for a paired re-analysis of Cleaning, Craft, and Mobility.

Family	Model	Default	Targeted	$\bm{\Delta}$	Condition
Sports	GPT-5.4-mini	83.9	90.3	+6.5	Generic
	Qwen2.5-VL-72B	93.5	93.5	+0.0	Generic
	Gemini-3.1-Pro	71.0	74.2	+3.2	Generic
	GLM-4.6V	77.4	74.2	$-$ 3.2	Generic
	Mistral-3.2	77.4	71.0	$-$ 6.5	Generic
Tool-use	GPT-5.4-mini	63.5	73.1	+9.6	Scenario

Appendix

Table 17: C5: Supporting families, per-model results (%). Targeted is the intervention condition designated for that family before evaluation (named in the last column), not a per-model maximum;

\Delta

is therefore negative where the designated intervention underperforms the default. Point estimates on per-model subsets.

Family	Scenario SRT v2	Interpretation
Tool-use	61.9 [55.9, 67.6]	below v1 scenario SRT
Cleaning	41.1 [35.5, 46.9]	overlaps default
Craft	48.3 [43.4, 53.3]	overlaps v1
Mobility	50.8 [46.5, 55.0]	below default

Appendix

Table 18: C6: Exploratory Scenario SRT v2 variants (accuracy %, Wilson 95% CI). Introduced after the primary conditions were frozen; excluded from all main-paper aggregate and comparative claims.

Condition	Physical	Assembly
Default	29.4 [23.3,36.5]	44.4 [37.4,51.7]
Visual evidence only	51.1 [43.9,58.3]	45.6 [38.4,52.8]
Policy only (T)	15.6 [11.0,21.6]	33.9 [27.4,41.1]
S+T	27.8 [21.8,34.7]	40.6 [33.7,47.9]
R+T	41.1 [34.2,48.4]	41.6 [34.7,49.0]
S+R, no T	35.0 [28.4,42.2]	40.0 [33.1,47.3]

Appendix

Table 19: D1: Ablation results with Wilson 95% CI (accuracy %, three-model average: GPT-5.4-mini, Qwen2.5-VL-72B, Mistral-Small-3.2;

n

=180 per condition per family; parseable-only convention). Point estimates match Table 3 of the main text. † = boundary alignment ablation; ‡ = routing feasibility ablation.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
Sep 28, 2026Rong Yu Xu, Prayag Tiwari, Shaolei ZhangEfficient VLM Inference VLM Interpretability
Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China
Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts
Apr 10, 2026Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2Vision-Language Grounding Visual Evidence Sensitivity in VLMs
Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.
Composition, Not Conversation: VLMs Lose the Scene, Not the Thread
Sep 29, 2026L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5VLM Evaluation Visual Question Answering
NIT Silchar · MBZUAI · BML Munjal University +1

Sep 28, 2026J/K move · Enter open · S save

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

Rong Yu Xu, Prayag Tiwari, Shaolei Zhang

Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China

Efficient VLM Inference VLM Interpretability Latent Visual Reasoning VLM Reasoning Reasoning for VLMs Compositional Reasoning in VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.

Abstract

VLM Reasoning

Explore similar work

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

Composition, Not Conversation: VLMs Lose the Scene, Not the Thread