Multimodal reasoning requires models to combine multiple pieces of evidence into decisions that satisfy all relevant constraints. Evaluations centered on final-answer accuracy leave unclear whether failures arise from reading individual facts or using them together. We study this question through the multi-evidence integration gap, linking local fact recovery to global decision validity on controlled subset-selection and path-finding tasks. Our evaluation combines conditional performance given correct local answers with same-grid local queries, text-oracle inputs, retrieval controls, and constraint ablations. We further examine hidden-state probes and activation replacement across network depth, token positions, and model families. Across the evaluated models, accurate local recovery coexists with substantial global failure. For example, Gemma-3-27B recovers all evaluated isolated item facts correctly but achieves only 19.75% global validity on eight-item visual instances. Probes decode information about solution membership from global-prompt states, while corresponding clean-state replacement recovers failed decisions over model-dependent depth ranges. These findings show that recovering facts and representing solution-related information are insufficient for reliable joint decisions. They motivate evaluations and model improvements that explicitly target how available evidence is combined, alongside advances in perception and model scale.
Figures & tables
Figure 1: Overview of our study. (a) A packing example illustrates the local-to-global gap: every price is read correctly, yet the selected combination exceeds the budget. (b) Same-grid local queries, matched retrieval, constraint variants, and visual/text inputs examine possible explanations for the gap; a weighted-path task tests whether it extends beyond subset selection. (c) Probes assess solution membership and answer correctness in global-prompt hidden states.
Figure 2: Recoverable evidence does not guarantee globally valid decisions. (a) Qwen2.5-VL-3B: instance-level local recovery under single-card and full-grid queries, compared with unconditional visual global validity. (b) Global validity conditioned on every full-grid local query being correct. Cells report percentages and retained sample counts; asterisks indicate n<20 . Rows correspond to Qwen2.5-VL-3B, Gemma-3-4B, InternVL3-8B, and LLaVA-OneVision-7B. (c) Visual and text-oracle validity across eight checkpoints at k=8 . (d) Within-family visual validity across model sizes at k=8 ; the vertical axis spans 0–26%. Panels (a), (c), and (d) use 400 instances per condition. All rates are point estimates.
Figure 3: Behavioral diagnostics of the multi-evidence integration gap. Q3 and Q7 denote Qwen2.5-VL-3B and Qwen2.5-VL-7B; G4 and G12 denote Gemma-3-4B and Gemma-3-12B. V and T indicate visual-grid and text-oracle inputs, respectively. (a) Retrieval versus joint selection. At k=8 , models more reliably retrieve target items from identifying attribute descriptions than select a combination satisfying all constraints in the original task. (b) Constraint ablation. Cells report valid-answer rates (%) at k=8 ; darker green indicates higher performance. Full retains all constraint groups, whereas Num., Cat., and Rel. retain only numerical, categorical, and compatibility constraints, respectively. (c) Beyond subset selection. Visual-input exact-path accuracy decreases as the number of graph edges increases, although the target path remains three edges long. Each plotted condition contains 400 instances.
Model
Local
Visual
Text
Qwen2.5-VL-3B
100.00
39.00
27.75
Qwen2.5-VL-7B
100.00
42.50
45.50
Gemma-3-4B
100.00
13.25
43.00
Gemma-3-12B
100.00
40.50
71.25
Table 1: Correct edge facts, incorrect paths. Local: all-field accuracy over 16,000 isolated edge queries. Visual/Text: exact-path accuracy on 400 sixteen-edge instances. All values are percentages.
Figure 4: Decoding solution components and predicting answer correctness are different tests. Q3/Q7 denote Qwen2.5-VL-3B/7B; G4/G12 denote Gemma-3-4B/12B. (a,b) Correctness AUC for visual-grid and text-oracle prompts. (c,d) Candidate-membership macro-AUC for the same input conditions. Points show means over five nested-cross-validation seeds; error bars show the standard deviation across seeds. Each candidate-count group contains 400 instances. Layers are selected within the training folds. The dashed line marks AUC 0.5 .
Model
k=4
k=6
k=8
Layers
Qwen2.5-VL-3B
99
100
100
13
Qwen2.5-VL-7B
37
15
10
9
Gemma-3-4B
34
8
6
14
Gemma-3-12B
47
15
13
9
InternVL3-8B
60
16
11
13
Table 2: Activation-patching coverage. Counts report evaluated clean-correct/corrupt-wrong pairs. Layers gives the number of tested decoder blocks for full-prompt replacement. Qwen2.5-VL-3B also receives image- and text-token interventions at eight shared blocks on the same pairs.
Figure 5: Recovery depends on both depth and replaced token positions. All panels use Qwen2.5-VL-3B with visual-grid inputs. (a) Valid-answer rates after full-prompt matched or mismatched replacement at k=4 . (b,c) Matched-minus-mismatched valid-answer rates for three replacement scopes at k=4 and k=8 , restricted to their shared tested blocks. Each scope uses the same 99 and 100 recipient pairs, respectively. Depth is the zero-based decoder-block index divided by the final block index. Shading shows pointwise paired-bootstrap 95% intervals; intervals in (a) are bootstrap intervals for each recovery rate.
Figure 6: Full-prompt recovery profiles vary across model families and sizes. Both panels show matched-minus-mismatched valid-answer rates at k=4 with visual-grid inputs. (a) Q3/Q7: Qwen2.5-VL-3B/7B. (b) G4/G12: Gemma-3-4B/12B; I8: InternVL3-8B. Shading shows pointwise paired-bootstrap 95% intervals. Normalized depth aligns relative block indices.
Appendix figures & tables53 assets
Supplementary material from the paper’s appendix.
Appendix
k
Full
Numerical
Categorical
Compatibility
2
1.00
1.00
1.00
1.00
4
1.00
1.96
3.38
1.19
6
1.00
5.84
14.75
2.01
8
1.00
14.87
36.31
3.36
Appendix
Table A1: Mean number of valid subsets after constraint ablation. Enumerated on the final 400 instances per candidate count. Required subset size remains fixed; this quantifies the change in the answer acceptance criterion.
Model
Input
k=2
k=4
k=6
Q3
V
99.75
53.00
18.00
Q3
T
100.00
67.75
29.50
Q7
V
100.00
76.00
29.50
Q7
T
100.00
83.00
36.25
Q32
V
100.00
90.50
39.25
Q32
T
100.00
86.50
49.25
Appendix
Table A2: Global selection accuracy across candidate counts. Entries report the percentage of answers satisfying all selection constraints on 400 instances per candidate count. V and T denote visual-grid and text-oracle inputs, respectively. The k=8 results appear in the main cross-model figure.
Model
Input
k
c=0,d=0
0,.25
1,0
1,.25
Q3
V
4
41.00
71.00
39.00
61.00
Q3
V
6
17.00
30.00
10.00
15.00
Q3
V
8
10.00
4.00
3.00
6.00
Q3
T
4
63.00
74.00
64.00
70.00
Q3
T
6
30.00
36.00
22.00
30.00
Q3
T
8
14.00
15.00
11.00
22.00
Appendix
Table A3: Final selection predictions stratified by nominal conflict c and distractor ratio d . Each cell is validity (%) over 100 instances. These are disjoint subgroups of the main benchmark. Parameter labels describe generation settings; the realized number of distractors is capped by the number of non-solution items.
Model
Cost
Weight
Value
Color
Type
Compat.
All
Q3
100.00
100.00
100.00
100.00
100.00
100.00
100.00
Q7
100.00
100.00
100.00
100.00
100.00
100.00
100.00
Q32
100.00
100.00
100.00
100.00
100.00
100.00
100.00
G4
100.00
100.00
100.00
100.00
100.00
100.00
100.00
G12
100.00
100.00
100.00
100.00
100.00
100.00
100.00
G27
100.00
100.00
100.00
100.00
100.00
100.00
100.00
Appendix
Table A4: Single-card field recovery (%) over 8,000 item queries per model. All denotes that all six fields of an item are correct. This is an item-level metric; conditional selection uses all items in an instance.
Model
k
n
V (%)
T (%)
Q3
2
400
99.75
100.00
Q3
4
400
53.00
67.75
Q3
6
400
18.00
29.50
Q3
8
400
5.75
15.50
Q7
2
400
100.00
100.00
Q7
4
400
76.00
83.00
Appendix
Table A5: Selection validity conditioned on all single-card local questions being correct. n is the retained number of instances out of 400. V/T are separate global runs conditioned on the same local-query subset. Unlike the main same-grid control, these local questions show one card at a time.
Model
k
Cost
Weight
Value
Color
Type
Compat.
All
Q3
2
100.00
100.00
100.00
100.00
100.00
100.00
100.00
Q3
4
100.00
100.00
100.00
100.00
100.00
100.00
100.00
Q3
6
100.00
99.96
99.96
99.96
100.00
98.96
98.96
Q3
8
100.00
99.91
100.00
100.00
100.00
98.88
98.78
G4
2
100.00
100.00
100.00
100.00
100.00
100.00
100.00
G4
4
98.81
99.69
99.81
100.00
100.00
98.44
97.19
Appendix
Table A6: Same-grid local field recovery (%). All denotes all fields of one queried item. Each row contains 400k local queries. The main-text heatmap already reports instance-level retention and conditional global validity.
Model
Input
k=2
k=4
k=6
Q3
V
100.00
87.25
89.75
Q3
T
100.00
83.00
83.50
Q7
V
100.00
100.00
99.75
Q7
T
100.00
100.00
98.25
G4
V
100.00
87.75
82.75
G4
T
100.00
98.50
90.75
Appendix
Table A7: Exact target-set retrieval (%) for the remaining candidate counts. Each cell uses 400 instances; the main text reports k=8 .
Model
Input
k
Full
Num.
Cat.
Rel.
Q3
V
2
99.50
100.00
99.00
100.00
Q3
V
4
52.00
72.00
97.25
61.75
Q3
V
6
18.25
46.25
96.75
22.25
Q3
T
2
100.00
100.00
100.00
100.00
Q3
T
4
69.50
79.75
96.50
81.50
Q3
T
6
31.50
46.50
95.75
50.50
Appendix
Table A8: Constraint-ablation validity (%) at the remaining candidate counts, 400 instances per cell. Full is the within-suite prompt control; Num., Cat., and Rel. retain numerical, categorical, and compatibility requirements, respectively. Required subset size is retained throughout.
Model
4 edges
8 edges
12 edges
Q3
96.25
56.00
41.50
Q7
99.00
83.00
60.75
G4
96.00
77.00
56.75
G12
99.25
88.25
80.75
Appendix
Table A9: Text-oracle exact-path accuracy (%) at the remaining graph sizes, 400 instances per cell. The main text provides the full visual curve and the sixteen-edge text result.
Model
Input
4 edges
8 edges
12 edges
16 edges
Q3
V
100.00
100.00
100.00
100.00
Q3
T
100.00
100.00
100.00
100.00
Q7
V
100.00
100.00
100.00
100.00
Q7
T
100.00
100.00
100.00
100.00
G4
V
100.00
100.00
100.00
100.00
G4
T
100.00
100.00
100.00
100.00
Appendix
Table A10: Nonempty parsed-path rates (%).
Model
Source
Target
Cost
Q3
100.00
100.00
100.00
Q7
100.00
100.00
100.00
G4
100.00
100.00
100.00
G12
100.00
100.00
100.00
Appendix
Table A11: Local edge-field accuracy (%) over 16,000 isolated queries. The main-text all-field score is not repeated.
Figure A1: Qwen2.5-VL-3B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output.
Figure A2: Qwen2.5-VL-7B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output. Only the reconciled text condition is included.
Figure A3: Gemma-3-4B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output.
Figure A4: Gemma-3-12B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output.
Figure A5: InternVL3-8B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output.
Figure A6: LLaVA-OneVision-7B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output.
Model
Input
k
AUC
Acc.
Positive
Depth
I8
V
4
0.790
0.720
0.760
0.157
I8
V
6
0.561
0.610
0.295
0.443
I8
V
8
0.470
0.735
0.140
0.557
I8
T
4
0.873
0.850
0.848
0.457
I8
T
6
0.621
0.588
0.490
0.386
I8
T
8
0.616
0.620
0.325
0.836
Appendix
Table A12: Additional-model nested correctness probes. AUC and accuracy are means across outer folds. Positive rate describes class balance; depth is the mean selected hidden-state index normalized by the final index. These models are absent from the main repeated-probe figure.
Model
Input
k
Shuffle AUC
SD
Shuffles/fold
Q3
V
4
0.505
0.059
10
Q3
V
6
0.486
0.081
10
Q3
V
8
0.515
0.134
10
Q3
T
4
0.516
0.089
10
Q3
T
6
0.522
0.083
10
Q3
T
8
0.501
0.102
10
Appendix
Table A13: Training-label shuffle controls at the already selected layer. SD is across pooled shuffle evaluations, not a confidence interval or a nested permutation test. Each included run uses five outer folds and ten shuffles per fold. The main repeated-seed runs use zero shuffles.
Model
Input
k
Positive
n
Q3
V
2
399
400
Q3
T
2
400
400
Q7
V
2
400
400
Q7
T
2
400
400
G4
V
2
400
400
G4
T
2
400
400
Appendix
Table A14: Candidate-count groups absent from the fixed-layer five-fold probe outputs. Positive counts refer to the final behavioral predictions; all remaining instances are negatives.
Index
AUC
AUC SD
Accuracy
0
0.500
0.000
0.010
1
0.785
0.159
0.985
2
0.854
0.025
0.985
3
0.881
0.031
0.983
4
0.937
0.023
0.982
5
0.970
0.010
0.982
Appendix
Table A15: LLaVA-OneVision visual correctness at k=2 : all recorded fixed-layer scores. There are 396 positive and four negative instances. These small-minority-class results are reported separately from the five-outer-fold nested analysis. SD describes the recorded cross-validation folds.
Model
Input
k
13
23
37
51
79
Q3
V
4
0.754
0.754
0.714
0.740
0.759
Q3
V
6
0.569
0.542
0.508
0.541
0.581
Q3
V
8
0.594
0.598
0.547
0.543
0.598
Q3
T
4
0.792
0.801
0.787
0.789
0.792
Q3
T
6
0.606
0.628
0.664
0.669
0.588
Q3
T
8
0.615
0.544
0.588
0.597
0.559
Appendix
Table A16: Qwen2.5-VL-3B: per-seed held-out correctness AUC. The main text presents mean and SD; this table exposes sensitivity to the split without reprinting that summary.
Model
Input
k
13
23
37
51
79
Q7
T
4
0.821
0.815
0.851
0.829
0.810
Q7
T
6
0.599
0.607
0.588
0.525
0.603
Q7
T
8
0.565
0.557
0.635
0.576
0.548
Appendix
Table A17: Qwen2.5-VL-7B: per-seed held-out correctness AUC. The main text presents mean and SD; this table exposes sensitivity to the split without reprinting that summary.
Model
Input
k
13
23
37
51
79
G4
V
4
0.645
0.642
0.682
0.640
0.670
G4
V
6
0.585
0.575
0.644
0.613
0.585
G4
V
8
0.458
0.502
0.588
0.571
0.484
G4
T
4
0.765
0.753
0.778
0.775
0.782
G4
T
6
0.657
0.646
0.653
0.668
0.630
G4
T
8
0.583
0.593
0.612
0.623
0.592
Appendix
Table A18: Gemma-3-4B: per-seed held-out correctness AUC. The main text presents mean and SD; this table exposes sensitivity to the split without reprinting that summary.
Model
Input
k
13
23
37
51
79
G12
V
4
0.798
0.770
0.778
0.766
0.744
G12
V
6
0.524
0.531
0.587
0.564
0.545
G12
V
8
0.451
0.549
0.525
0.509
0.573
G12
T
4
0.893
0.883
0.870
0.879
0.851
G12
T
6
0.644
0.620
0.558
0.610
0.611
G12
T
8
0.537
0.587
0.502
0.561
0.556
Appendix
Table A19: Gemma-3-12B: per-seed held-out correctness AUC. The main text presents mean and SD; this table exposes sensitivity to the split without reprinting that summary.
Input
k
ID
13
23
37
51
79
Positive
V
4
A
0.993
0.998
0.995
0.998
0.998
0.792
V
4
B
0.967
0.978
0.980
0.969
0.973
0.745
V
4
C
0.967
0.970
0.960
0.967
0.979
0.755
V
4
D
0.976
0.991
0.994
0.990
0.985
0.708
V
6
A
0.949
0.959
0.945
0.961
0.955
0.500
V
6
B
0.845
0.838
0.862
0.846
0.829
0.510
Appendix
Table A20: Qwen2.5-VL-3B: candidate-specific membership AUC for each seed. A–H denote candidate IDs. Positive is the fraction of instances whose valid subset contains that candidate. The main figure averages these candidate-level results.
Input
k
ID
13
23
37
51
79
Positive
V
4
A
0.989
0.990
0.988
0.984
0.987
0.792
V
4
B
0.982
0.984
0.983
0.983
0.989
0.745
V
4
C
0.980
0.987
0.992
0.987
0.983
0.755
V
4
D
0.974
0.982
0.985
0.985
0.975
0.708
V
6
A
0.970
0.968
0.965
0.957
0.964
0.500
V
6
B
0.827
0.857
0.847
0.859
0.851
0.510
Appendix
Table A21: Qwen2.5-VL-7B: candidate-specific membership AUC for each seed. A–H denote candidate IDs. Positive is the fraction of instances whose valid subset contains that candidate. The main figure averages these candidate-level results.
Input
k
ID
13
23
37
51
79
Positive
V
4
A
0.992
0.991
0.992
0.981
0.995
0.792
V
4
B
0.976
0.972
0.966
0.978
0.971
0.745
V
4
C
0.975
0.986
0.982
0.982
0.981
0.755
V
4
D
0.973
0.982
0.986
0.980
0.978
0.708
V
6
A
0.927
0.916
0.928
0.916
0.920
0.500
V
6
B
0.859
0.870
0.862
0.845
0.872
0.510
Appendix
Table A22: Gemma-3-4B: candidate-specific membership AUC for each seed. A–H denote candidate IDs. Positive is the fraction of instances whose valid subset contains that candidate. The main figure averages these candidate-level results.
Input
k
ID
13
23
37
51
79
Positive
V
4
A
0.997
0.998
0.996
0.989
0.992
0.792
V
4
B
0.975
0.980
0.979
0.980
0.976
0.745
V
4
C
0.967
0.968
0.970
0.972
0.965
0.755
V
4
D
0.979
0.986
0.976
0.980
0.981
0.708
V
6
A
0.972
0.968
0.970
0.962
0.977
0.500
V
6
B
0.884
0.905
0.891
0.890
0.883
0.510
Appendix
Table A23: Gemma-3-12B: candidate-specific membership AUC for each seed. A–H denote candidate IDs. Positive is the fraction of instances whose valid subset contains that candidate. The main figure averages these candidate-level results.
Model
Input
k
Seed
Selected indices
Q3
V
4
13
25,28,28,27,28
Q3
V
4
23
28,26,26,29,22
Q3
V
4
37
26,29,21,28,26
Q3
V
4
51
26,28,22,21,21
Q3
V
4
79
28,25,26,28,22
Q3
V
6
13
2,28,28,27,5
Appendix
Table A24: Selected hidden-state indices for the five outer folds in each correctness run. These support interpretation of split sensitivity.
Figure A7: Q3: full-prompt recovery at k=6,8 , complementing the main k=4 comparisons. Both donor conditions use the same recipient pairs. Curves show absolute valid-answer rates. Points denote tested blocks.
Figure A8: Q7: full-prompt recovery at k=6,8 , complementing the main k=4 comparisons. Both donor conditions use the same recipient pairs. Curves show absolute valid-answer rates. Points denote tested blocks.
Figure A9: G4: full-prompt recovery at k=6,8 , complementing the main k=4 comparisons. Both donor conditions use the same recipient pairs. Curves show absolute valid-answer rates. Points denote tested blocks.
Figure A10: G12: full-prompt recovery at k=6,8 , complementing the main k=4 comparisons. Both donor conditions use the same recipient pairs. Curves show absolute valid-answer rates. Points denote tested blocks.
Figure A11: I8: full-prompt recovery at k=6,8 , complementing the main k=4 comparisons. Both donor conditions use the same recipient pairs. Curves show absolute valid-answer rates. Points denote tested blocks.
Model
k
Block
Δ
95% interval
G12
6
0
1.000
[1.000, 1.000]
G12
6
9
1.000
[1.000, 1.000]
G12
6
19
0.800
[0.600, 1.000]
G12
6
26
0.400
[0.133, 0.667]
G12
6
31
0.067
[0.000, 0.200]
G12
6
33
0.067
[0.000, 0.200]
Appendix
Table A25: Full-prompt paired donor contrasts at k=6,8 . Intervals use 2,000 recipient-level bootstrap resamples and are pointwise. Cohort sizes appear in the corresponding recovery panels and main coverage table. Boundary intervals can collapse when all observed differences are identical.
Figure A12: Qwen-3B token-scope contrasts at k=6 on 100 matched recipient pairs. Shading shows pointwise 95% paired-bootstrap intervals (2,000 resamples). Full prompt has 13 tested blocks; each restricted scope has eight. The main text reports the k=4,8 scope comparisons.
Scope
k
Matched
Mismatched
Full
4
0.879
0.000
Full
6
0.930
0.530
Full
8
0.920
0.630
Image
4
0.455
0.081
Image
6
0.790
0.500
Image
8
0.770
0.600
Appendix
Table A26: Decoy avoidance at Qwen-3B block 24. Each value is the fraction of patched outputs that omit the modified non-solution item. An output can avoid the decoy yet fail other constraints.
Model
Bank
k
CC
CW
WC
WW
Q3
base
4
296
145
78
481
Q3
base
6
87
54
46
1013
Q3
ext1
6
85
75
47
1193
Q3
base
8
55
76
41
2828
Q3
ext1
8
35
42
17
1706
Q7
base
4
27
37
6
30
Appendix
Table A27: Pair-bank outcome counts before intervention subsampling. CC/CW/WC/WW refer to clean and corrupted correctness, respectively. Only CW pairs are eligible. Qwen-3B uses a larger base bank plus independent extensions; eligible counts can exceed the evaluated counts in the main table.
Model
Input
k
Correct/ n
Validity [95% CI]
Q3
V
2
399/400
99.75 [98.60, 99.96]
Q3
V
4
212/400
53.00 [48.10, 57.84]
Q3
V
6
72/400
18.00 [14.55, 22.06]
Q3
V
8
23/400
5.75 [3.86, 8.48]
Q3
T
2
400/400
100.00 [99.05, 100.00]
Q3
T
4
271/400
67.75 [63.02, 72.14]
Appendix
Table A28: Selection validity with 95% Wilson intervals. Each row uses 400 final benchmark instances.
Model
k
Retained/400
Recovery [95% CI]
Q3
2
400/400
100.00 [99.05, 100.00]
Q3
4
400/400
100.00 [99.05, 100.00]
Q3
6
400/400
100.00 [99.05, 100.00]
Q3
8
400/400
100.00 [99.05, 100.00]
Q7
2
400/400
100.00 [99.05, 100.00]
Q7
4
400/400
100.00 [99.05, 100.00]
Appendix
Table A29: Isolated-card complete local recovery at the instance level: every field of every item must be correct. Denominator: 400 instances per row.
Model
Input
k
Correct/retained
Validity [95% CI]
Q3
V
2
399/400
99.75 [98.60, 99.96]
Q3
T
2
400/400
100.00 [99.05, 100.00]
Q3
V
4
212/400
53.00 [48.10, 57.84]
Q3
T
4
271/400
67.75 [63.02, 72.14]
Q3
V
6
72/400
18.00 [14.55, 22.06]
Q3
T
6
118/400
29.50 [25.24, 34.15]
Appendix
Table A30: Isolated-card conditional selection validity. The denominator is the retained cohort, not 400; these intervals describe that selected cohort.
Model
k
Retained/400
Recovery [95% CI]
Q3
2
400/400
100.00 [99.05, 100.00]
Q3
4
400/400
100.00 [99.05, 100.00]
Q3
6
376/400
94.00 [91.23, 95.94]
Q3
8
362/400
90.50 [87.23, 93.00]
G4
2
400/400
100.00 [99.05, 100.00]
G4
4
356/400
89.00 [85.55, 91.70]
Appendix
Table A31: Same-grid complete local recovery at the instance level: every field of every item must be correct. Denominator: 400 instances per row.
Model
Input
k
Correct/retained
Validity [95% CI]
Q3
V
2
399/400
99.75 [98.60, 99.96]
Q3
V
4
212/400
53.00 [48.10, 57.84]
Q3
V
6
66/376
17.55 [14.04, 21.72]
Q3
V
8
21/362
5.80 [3.83, 8.71]
G4
V
2
400/400
100.00 [99.05, 100.00]
G4
V
4
192/356
53.93 [48.74, 59.04]
Appendix
Table A32: Same-grid conditional selection validity. The denominator is the retained cohort, not 400; these intervals describe that selected cohort.
Model
Input
k
Correct/ n
Rate [95% CI]
Q3
V
2
400/400
100.00 [99.05, 100.00]
Q3
V
4
349/400
87.25 [83.62, 90.17]
Q3
V
6
359/400
89.75 [86.39, 92.35]
Q3
V
8
341/400
85.25 [81.44, 88.39]
Q3
T
2
400/400
100.00 [99.05, 100.00]
Q3
T
4
332/400
83.00 [79.01, 86.36]
Appendix
Table A33: Target-set retrieval with 95% Wilson intervals; 400 instances per row.
Model
Input
k
Scope
Correct/ n
Rate [95% CI]
Q3
V
2
Full
398/400
99.50 [98.20, 99.86]
Q3
V
4
Full
208/400
52.00 [47.11, 56.85]
Q3
V
6
Full
73/400
18.25 [14.77, 22.33]
Q3
V
8
Full
24/400
6.00 [4.06, 8.77]
Q3
V
2
Num.
400/400
100.00 [99.05, 100.00]
Q3
V
4
Num.
288/400
72.00 [67.41, 76.17]
Appendix
Table A34: Constraint-ablation validity with 95% Wilson intervals; 400 instances per row.
Model
Input
k
Correct/ n
Rate [95% CI]
Q3
V
4
392/400
98.00 [96.10, 98.98]
Q3
V
8
324/400
81.00 [76.87, 84.54]
Q3
V
12
212/400
53.00 [48.10, 57.84]
Q3
V
16
156/400
39.00 [34.35, 43.86]
Q3
T
4
385/400
96.25 [93.91, 97.71]
Q3
T
8
224/400
56.00 [51.10, 60.78]
Appendix
Table A35: Exact-path accuracy with 95% Wilson intervals; 400 instances per row.
Model
Edges
Correct graphs/ n
Recovery [95% CI]
Q3
4
400/400
100.00 [99.05, 100.00]
Q3
8
400/400
100.00 [99.05, 100.00]
Q3
12
400/400
100.00 [99.05, 100.00]
Q3
16
400/400
100.00 [99.05, 100.00]
Q7
4
400/400
100.00 [99.05, 100.00]
Q7
8
400/400
100.00 [99.05, 100.00]
Appendix
Table A36: Complete isolated-edge recovery per graph. A success requires every field of every edge to be correct. The graph, rather than an edge query, is the sampling unit.
Model
k
Direct
Review
ES
ESV
Qwen2.5-VL-3B
4
53.25
7.00
62.25
59.25
6
17.50
2.75
16.50
15.50
8
5.75
1.50
5.25
5.25
Qwen2.5-VL-7B
4
71.50
0.75
74.75
74.25
6
28.75
0.25
26.25
26.25
8
11.75
0.00
10.50
10.50
Appendix
Table A37: Prompting controls produce model-dependent changes without consistently resolving difficult global decisions. Entries are exact globally valid answers (%) over 400 paired visual-grid instances. ES denotes Extract–Select and ESV denotes Extract–Select–Verify.
Checkpoint
Condition
n
Valid [95% CI]
Final JSON
At cap
Qwen3-VL-8B Instruct
Answer only
195
5.64 [3.18, 9.82]
99.49
0.00
Explicit reasoning
195
48.21 [41.29, 55.19]
53.33
0.00
Qwen3-VL-8B Thinking
Answer only
195
0.00 [0.00, 1.93]
0.00
100.00
Explicit reasoning
195
0.51 [0.09, 2.85]
0.51
100.00
Appendix
Table A38: Explicit reasoning changes global validity on paired eight-candidate visual instances. Valid reports exact globally valid answers with pointwise Wilson 95% intervals. Final JSON is the share of responses containing a parseable item-set object. At cap is the share whose final generation reaches its token budget.
Figure A13: Explicit reasoning repairs some decisions and can also overturn valid answers. Each horizontal bar partitions the same 195 instances by their paired answer-only and explicit-reasoning outcomes. Repaired denotes an invalid answer that becomes globally valid; regressed denotes a valid answer that becomes invalid. Qwen3-VL-8B-Instruct and Qwen3-VL-8B-Thinking use identical instances and deterministic decoding.
Model
k
Direct
ES
Extract–Solve [95% CI]
Δ Direct
95% CI for Δ
Qwen2.5-VL-3B
4
53.25
62.25
36.50 [31.93, 41.33]
−16.75
[ −24.00,−9.50 ]
6
17.50
16.50
82.25 [78.20, 85.68]
+64.75
[ +59.25,+70.00 ]
8
5.75
5.25
92.25 [89.21, 94.49]
+86.50
[ +82.50,+90.01 ]
Qwen2.5-VL-7B
4
71.50
74.75
84.00 [80.09, 87.27]
+12.50
[ +7.00,+18.00 ]
6
28.75
26.25
94.50 [91.81, 96.34]
+65.75
[ +61.25,+70.50 ]
8
11.75
10.50
96.75 [94.52, 98.09]
+85.00
[ +81.25,+88.50 ]
Appendix
Table A39: Extract–Solve converts usable transcriptions into globally valid decisions. All entries are percentages over 400 paired instances. ES denotes Extract–Select. Brackets beside Extract–Solve report pointwise Wilson 95% intervals. The final column reports 95% intervals for the paired difference from Direct, estimated with 10,000 bootstrap draws stratified by conflict intensity and distractor ratio.
Figure A14: Outcome decomposition of the semantic Extract–Solve pipeline. Each bar contains 400 instances. Solved denotes transcriptions that yield one unique feasible solution; all 3,137 instances in this category match the benchmark target. Incomplete denotes missing item cards, Invalid covers malformed JSON or invalid fields, and Inconsistent combines zero- and multiple-solution outcomes. Q3 and Q7 denote Qwen2.5-VL-3B and Qwen2.5-VL-7B; G4 and G12 denote Gemma-3-4B and Gemma-3-12B.
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.
Rong Yu Xu, Prayag Tiwari, Shaolei Zhang
Shenzhen College of International Education · Halmstad University, Sweden · Renmin University of China
When a multimodal large language model answers a visual reasoning question correctly, is the prediction actually supported by the task-critical visual evidence? Correct answers can coexist with flawed reasoning, making accuracy alone an incomplete test of grounding. We introduce VisualFLIP, a paired benchmark with 1,374 images arranged as same-question perturbation pairs across cardinality, attribute, spatial, and logic tasks. Each pair keeps the question fixed but minimally changes the evidence so the gold answer deterministically flips. We evaluate 24 MLLMs with pair accuracy, which requires solving both sides of a pair, and Collapse Rate (CR), which measures how often a model that solves at least one side repeats the same non-empty answer for both images. Together, these metrics show that paired correctness and evidence dependence are related but distinct: capable models can still fail to update after task-critical visual changes, and collapse becomes more severe for some models when the edited image follows an earlier answer in a sequential setting. Further details are available on our project page: https://didizhu-judy.github.io/VisualFLIP/
Vision-language models (VLMs) answer visual questions by combining visual information extraction with downstream problem solving. We investigate a fundamental question: Does an incorrect answer necessarily reflect a failure in visual extraction or problem solving? A model may succeed at both abilities when tested separately yet still fail on the original multimodal question, a distinction that overall answer accuracy cannot reveal. To study this, we perform a question-level empirical analysis across multiple VLMs and visual domains. We define an exactly scorable task state (i.e., the visual information sufficient to solve a question) and use it to test whether the same model can extract the required state, solve the question from the ground-truth state, and answer the original multimodal question. We find that composition failures, where extraction and solving both succeed but direct answering fails, account for 17.7% to 75.6% of direct-answering errors across multiple VLMs and datasets. To address this failure mode, we introduce a simple yet effective method, termed State Realization Tuning (SRT). SRT fine-tunes LoRA adapters attached to the language-model layers while keeping the pretrained VLM weights frozen. It trains the model to output the ground-truth task state before the final answer in a single autoregressive response. SRT improves over standard supervised fine-tuning by 1.7 to 14.1 percentage points and repairs 92.5% to 98.1% of diagnosed composition failures. A single LoRA adapter trained with SRT also improves performance across substantially different task-state structures. Our work shows that having both visual extraction and problem-solving capabilities does not guarantee correct multimodal answering. Requiring the model to first output the visual information needed to solve the question can help bridge this gap.
Ziheng Wang, Mingxuan Xie, Yilin Liu +3
Sun Yat-sen University · Zhejiang University · The Hong Kong University of Science and Technology +2