Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they have already identified correctly. We distinguish perceptual failure, where relevant evidence is not recognized, from process failure, where recognized evidence fails to constrain the final decision. We introduce VPAC-Bench, a benchmark spanning nine real-image process families, with each image annotated by its current activity stage and nearby stage transition. We also propose State-Relevance-Target (SRT), a family of structured process-prior interventions that requires models to connect visible evidence to the relevant process state before answering. Across multiple VLMs, process failure is widespread: models that correctly enumerate visual candidates still over-commit to a single answer in more than 95% of ambiguous cases. An explicit process-structured intervention reduces this rate to below 13% without degrading performance on unambiguous cases. However, the transfer of process priors is model-dependent, and generic SRT does not consistently outperform strong chain-of-thought baselines. When the relevant stage transition is known, boundary-aligned SRT substantially outperforms generic process prompting and all tested chain-of-thought baselines across assembly, physical state transition, navigation and traffic, and object-use affordance tasks. These results show that process priors are most useful when aligned with the scene's specific decision boundary, motivating boundary-aware prior selection for process-grounded visual reasoning.
Figures & tables
Figure 1: Overview of our framework. Top left : a motivating example showing process failure, the model correctly identifies multiple car candidates yet still commits to a single answer, violating its own observations. Top center : our benchmark ( VPAC-Bench ) covers nine real-image process families (sports action, mobility, tool use and repair, household activities, drawing & craft, assembly, physical transition, navigation, and object-use). Bottom : the SRT framework applied to a skiing image. State extraction (S) identifies the decision-relevant visible entities (two skiers, equipment, motion cues). Relevance checking (R) filters to decision-relevant relations only (person–ski contact ✓, body posture ✓) and suppresses irrelevant ones (number of ski tracks, backpack brand ⊘ ). Target commitment (T) commits only to the stage supported by visible evidence ( pre_action_setup ), ruling out active_execution because no strong motion cue or dynamic posture is visible.
Component
S1: Referent Disambiguation
S2: Spatial Reasoning
S3: Process Families
S State Extraction
Visible candidate referents and distinguishing features
Subject, reference object, and visible geometric layout
Actor, tool, manipulated object, and object-state cues
Table 1: The three SRT invariants across the three VPAC stages. In S1, this structure is embodied by an existing inspect–judge–decide method whose effectiveness motivates SRT; S2 and S3 evaluate SRT directly.
Figure 2: Process failure is distinct from perceptual failure (COCO Task 1, five models). Providing visual annotations without a decision process ( marked ) leaves policy accuracy near the default baseline, while adding an inspect–judge–decide structure on top of the same visual access ( VS ) raises it above 87% for four of five models.
Model
Data
Default
Evidence
Marked
VS
GPT-5.4-mini
COCO
0.0/98.5
45.1/54.7
0.7/98.9
88.8 /9.2
GQA
0.0/97.0
35.0/65.0
3.0/97.0
91.0 /6.0
Qwen2.5-VL-72B
COCO
3.5/95.8
44.7/48.9
9.8/90.2
91.0 /8.5
GQA
5.0/91.0
52.0/47.0
12.0/88.0
86.0 /13.0
GLM-4.6V
COCO
0.0/99.0
37.0/62.0
5.0/95.0
87.0 /12.0
GQA
0.0/100.0
42.0/57.0
31.0/69.0
92.0 /7.0
Table 2: S1 Task 1 policy accuracy / over-answer rate (%) on COCO and GQA. Bold indicates the highest policy accuracy in each row. VS = visual sketchpad .
Figure 3: Process-prior transfer is model-sensitive across five frontier VLMs on VSR. Each line traces one model’s accuracy as the SRT condition progresses from default through generic, relation-routed, and model-specific instantiations. Cross-model points are dev100 ( n =100); Qwen’s relation-routed point is test400 ( n =400).
Figure 4: Oracle boundary-sensitive SRT versus generic SRT on three of the four standardized process families ( n =60 per family per model, five-model averages); results for the fourth family (object-use affordance) are reported in Table C4. The rightmost panel shows a single process-overload case (GLM, boundary P7) and is not a family-level result.
Condition
Phys.
Assem.
Default
29.4
44.4
Visual evidence only
51.1
45.6
Policy only ( T )
15.6
33.9
S+T
27.8
40.6
R+T
41.1
41.6
S+R , no T
35.0
40.0
Table 3: Ablation accuracy (%), averaged across three models ( n=180 per condition and family). † denotes boundary-alignment ablation; ‡ denotes routing-feasibility ablation. Wilson 95% intervals are reported in Table D1.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Dataset / Family
Items
Models
S1
COCO Task 1 (ambiguous)
457 ∗
5
S1
COCO Task 2 (unique)
330 ∗
5
S1
GQA Task 1 (ambiguous)
100
5
S1
GQA Task 2 (unique)
96
5
S2
VSR dev100
100
5
S2
VSR test400 (Qwen)
400
1
Appendix
Table 4: Nominal evaluation sample sizes per stage and dataset. S3 family-level results ( n =60 per model) are treated as diagnostic evidence; aggregate trends across families and models form the primary basis for our conclusions.
Field
Meaning
gold_step_label
Current visible process stage.
boundary_family
Process boundary being tested.
visible_actor
Whether the relevant actor is visible.
neighbor_step_label
Most confusable adjacent process stage.
why_not_neighbor
Visible evidence ruling out adjacent stage.
issue_type
Ambiguity, occlusion, or image-quality flag.
Appendix
Table 5: Main annotation fields used for process-family examples.
Family
Generic SRT
Self-router
Oracle router
Self − Generic
Physical
46.1
47.8
86.1
+1.7
Assembly
42.2
42.8
85.0
+0.6
Appendix
Table 6: Oracle boundary routing versus self-routing (accuracy %, averaged across GPT-5.4-mini, Qwen2.5-VL-72B, and Mistral-Small-3.2; parseable-only convention). Self-router gains over generic SRT are not statistically established; intervals in Table D1.
Family
Default
Label-only
Oracle SRT
CSR
Δ
Standardized (paired)
Physical
26.3 [21.7,31.6]
55.7 [50.0,61.2]
76.7 [71.6,81.1]
+29.3
+21.0
Assembly
46.2 [40.1,52.6]
55.8 [49.5,62.0]
78.8 [73.1,83.5]
0 +9.6
+22.9
Traffic
47.1 [40.9,53.4]
43.3 [37.2,49.7]
71.2 [65.2,76.6]
0 − 3.8
+27.9
Affordance
37.1 [31.2,43.4]
33.3 [27.7,39.5]
57.5 [51.2,63.6]
0 − 3.8
+24.2
Supporting (unpaired, parseable-only)
Appendix
Table 7: B1: Decomposition of the oracle boundary router gain. The four standardized families are computed on the three-way paired intersection with unparseable = incorrect (Appendix A.1); physical uses five models ( n =300) and the other three use four ( n =240). Supporting families use the parseable-only convention and are not paired here; see Table B1 ′ for the paired version of Cleaning, Craft, and Mobility. CSR = Label-only − Default; Δ = Oracle SRT − Label-only.
Family
n
b
c
c−b
χ2
p
Physical
300
29
92
63
31.8
< 0.001
Assembly
240
20
75
55
30.7
< 0.001
Traffic
240
14
81
67
45.9
< 0.001
Affordance
240
18
76
58
34.6
< 0.001
Appendix
Table 8: B2: McNemar test, label-only vs. oracle SRT, computed from item-level paired responses (unparseable = incorrect). b = label-only correct and oracle wrong; c = oracle correct and label-only wrong; c−b equals the net item-count gain in the paired set. χ2 with continuity correction.
Family
n
Label-only
Oracle SRT
Δ
Cleaning
280
46.4 [40.7,52.3]
43.2 [37.5,49.1]
− 3.2
Craft
231
79.2 [73.5,84.0]
42.9 [36.6,49.3]
− 36.3
Mobility
530
53.4 [49.1,57.6]
46.4 [42.2,50.7]
− 7.0
Appendix
Table 9: B1 ′ : Label-only versus oracle SRT on the three reversed families, paired at the item level (unparseable = incorrect, five models), the same convention as Table B1.
Family
n
b
c
χ2
p
Cleaning
280
75
66
0.5
0.50
Craft
231
88
4
74.9
< 0.001
Mobility
530
159
122
4.6
0.032
Appendix
Table 10: B2 ′ : McNemar test, label-only vs. oracle SRT, three reversed families. b = label-only correct and oracle wrong; c = oracle correct and label-only wrong; note b>c throughout, the reverse of Table B2. Exact binomial test for Craft given the extreme imbalance; χ2 with continuity correction otherwise.
Model
Default
Label-only
Oracle SRT
Δ
GPT-5.4-mini
35.5
73.8
84.6
+10.9
Qwen2.5-VL-72B
37.1
69.5
76.7
+7.2
Gemini-3.1-Pro
32.5
57.5
67.7
+10.2
GLM-4.6V
44.6
61.8
60.4
− 1.4
Mistral-Small-3.2
38.3
63.3
80.0
+16.7
Four of five models show a positive margin; GLM is the exception, consistent
Appendix
Table 11: B3: Label-only vs. oracle SRT by model, micro-averaged across the standardized families for which that model has oracle outputs (parseable-only convention). Mistral covers physical state transition and assembly.
Family
Def.
Best CoT
Generic
Oracle
Δ
Physical
26.7
44.3
38.3
80.0
+35.7
Assembly
44.7
50.0
44.0
79.7
+29.7
Traffic
52.7
52.7
48.3
74.3
+21.7
Affordance
37.0
39.3
32.7
59.7
+20.4
Aggregate
40.3 [37.6,43.1]
46.6 [43.8,49.4]
40.8 [38.1,43.6]
73.4 [70.8,75.8]
+26.9
Appendix
Table 12: B4: SRT vs. CoT-style baselines (accuracy %, five-model averages, parseable-only convention). Best CoT = best of five CoT conditions. Wilson 95% CIs given for the pooled row ( n =1,200).
Family
App. B (paired)
Main text / App. C
Physical
76.7 ( n =300)
80.0 ( n =300)
Assembly
78.8 ( n =240)
79.7 ( n =300)
Traffic
71.2 ( n =240)
74.3 ( n =300)
Affordance
57.5 ( n =240)
59.7 ( n =300)
Appendix
Table 13: C1: Correspondence between the two accuracy conventions, for the oracle SRT condition. Appendix B pairs items across three conditions and counts unparseable responses as incorrect; the main text and the remainder of Appendix C use all parseable responses from all available models.
Set
Task
n
Default
Evidence
Marked
VS
COCO
T1
1214–1219 †
2.5 [1.7,3.5]
45.3 [42.4,48.2]
10.7 [9.1,12.6]
90.4 [88.6,91.9]
COCO
T2
865–961 ‡
96.3 [94.8,97.4]
96.1 [94.7,97.2]
99.5 [98.8,99.8]
99.6 [98.9,99.8]
Appendix
Table 14: C2: COCO T1/T2 pooled accuracy under the parseable-only convention. Actual pooled n per condition, not the nominal 5×457=2,285 or 5×330=1,650 , because three models’ evaluation files are capped at 100 items; see footnote. T1 correct policy = not-answer (clarify or enumerate); T2 correct policy = answer.
Model
Split
Condition
Accuracy [95% CI]
Qwen2.5-VL-72B
dev100
default
76.0 [66.8, 83.3]
Qwen2.5-VL-72B
dev100
evidence
85.0 [76.7, 90.7]
Qwen2.5-VL-72B
dev100
verify
80.0 [71.1, 86.7]
Qwen2.5-VL-72B
test400
default
78.5 [74.2, 82.2]
Qwen2.5-VL-72B
test400
evidence
78.5 [74.2, 82.2]
Qwen2.5-VL-72B
test400
verify
77.5 [73.2, 81.3]
Appendix
Table 15: C3: S2 VSR accuracy with Wilson 95% confidence intervals. Each row is a condition that was evaluated; conditions not run for a model are omitted rather than listed as empty. All cross-model comparisons use dev100; test400 was run for Qwen only. Gemini’s default-to-verify difference is the only S2 comparison whose intervals are disjoint.
Family
Default
Generic SRT
Scenario SRT
Boundary Router
Physical
26.7 [22.0, 31.9]
38.3 [33.0, 43.9]
37.7 [32.4, 43.3]
80.0 [75.1, 84.1]
Assembly
44.7 [39.1, 50.3]
44.0 [38.5, 49.7]
39.7 [34.3, 45.3]
79.7 [74.8, 83.8]
Traffic
52.7 [47.0, 58.2]
48.3 [42.7, 54.0]
41.7 [36.2, 47.3]
74.3 [69.1, 78.9]
Affordance
37.0 [31.7, 42.6]
32.7 [27.6, 38.2]
30.7 [25.7, 36.1]
59.7 [54.0, 65.1]
Cleaning
38.2 [32.7, 44.0]
29.3 [24.3, 34.9]
28.9 [23.9, 34.5]
45.4 [39.6, 51.2]
Craft †
44.9 [40.0, 49.9]
45.5 [40.6, 50.4]
51.2 [46.2, 56.1]
49.1 [44.1, 54.1]
Appendix
Table 16: C4: S3 process-family accuracy with Wilson 95% confidence intervals, five-model averages, parseable-only convention ( n =300 per cell for the four standardized families). These are the values reported in the main text. See Table B1 ′ for a paired re-analysis of Cleaning, Craft, and Mobility.
Family
Model
Default
Targeted
Δ
Condition
Sports
GPT-5.4-mini
83.9
90.3
+6.5
Generic
Qwen2.5-VL-72B
93.5
93.5
+0.0
Generic
Gemini-3.1-Pro
71.0
74.2
+3.2
Generic
GLM-4.6V
77.4
74.2
− 3.2
Generic
Mistral-3.2
77.4
71.0
− 6.5
Generic
Tool-use
GPT-5.4-mini
63.5
73.1
+9.6
Scenario
Appendix
Table 17: C5: Supporting families, per-model results (%). Targeted is the intervention condition designated for that family before evaluation (named in the last column), not a per-model maximum; Δ is therefore negative where the designated intervention underperforms the default. Point estimates on per-model subsets.
Family
Scenario SRT v2
Interpretation
Tool-use
61.9 [55.9, 67.6]
below v1 scenario SRT
Cleaning
41.1 [35.5, 46.9]
overlaps default
Craft
48.3 [43.4, 53.3]
overlaps v1
Mobility
50.8 [46.5, 55.0]
below default
Appendix
Table 18: C6: Exploratory Scenario SRT v2 variants (accuracy %, Wilson 95% CI). Introduced after the primary conditions were frozen; excluded from all main-paper aggregate and comparative claims.
Condition
Physical
Assembly
Default
29.4 [23.3,36.5]
44.4 [37.4,51.7]
Visual evidence only
51.1 [43.9,58.3]
45.6 [38.4,52.8]
Policy only (T)
15.6 [11.0,21.6]
33.9 [27.4,41.1]
S+T
27.8 [21.8,34.7]
40.6 [33.7,47.9]
R+T
41.1 [34.2,48.4]
41.6 [34.7,49.0]
S+R, no T
35.0 [28.4,42.2]
40.0 [33.1,47.3]
Appendix
Table 19: D1: Ablation results with Wilson 95% CI (accuracy %, three-model average: GPT-5.4-mini, Qwen2.5-VL-72B, Mistral-Small-3.2; n =180 per condition per family; parseable-only convention). Point estimates match Table 3 of the main text. † = boundary alignment ablation; ‡ = routing feasibility ablation.
Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.
Rong Yu Xu, Prayag Tiwari, Shaolei Zhang
Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China
When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of ρ= 0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.
Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2
Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.
Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.
L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5