cs.AIOct 24, 2026
SaveAll the Facts, Still the Wrong Answer: Integration Gaps in Vision-Language Models
Organizations: Florida State University · INSAIT, Sofia University “St. Kliment Ohridski” · MBZUAI
Abstract
Multimodal reasoning requires models to combine multiple pieces of evidence into decisions that satisfy all relevant constraints. Evaluations centered on final-answer accuracy leave unclear whether failures arise from reading individual facts or using them together. We study this question through the multi-evidence integration gap, linking local fact recovery to global decision validity on controlled subset-selection and path-finding tasks. Our evaluation combines conditional performance given correct local answers with same-grid local queries, text-oracle inputs, retrieval controls, and constraint ablations. We further examine hidden-state probes and activation replacement across network depth, token positions, and model families. Across the evaluated models, accurate local recovery coexists with substantial global failure. For example, Gemma-3-27B recovers all evaluated isolated item facts correctly but achieves only 19.75% global validity on eight-item visual instances. Probes decode information about solution membership from global-prompt states, while corresponding clean-state replacement recovers failed decisions over model-dependent depth ranges. These findings show that recovering facts and representing solution-related information are insufficient for reliable joint decisions. They motivate evaluations and model improvements that explicitly target how available evidence is combined, alongside advances in perception and model scale.
Figures & tables
Figure 1: Overview of our study. (a) A packing example illustrates the local-to-global gap: every price is read correctly, yet the selected combination exceeds the budget. (b) Same-grid local queries, matched retrieval, constraint variants, and visual/text inputs examine possible explanations for the gap; a weighted-path task tests whether it extends beyond subset selection. (c) Probes assess solution membership and answer correctness in global-prompt hidden states.
Figure 2: Recoverable evidence does not guarantee globally valid decisions. (a) Qwen2.5-VL-3B: instance-level local recovery under single-card and full-grid queries, compared with unconditional visual global validity. (b) Global validity conditioned on every full-grid local query being correct. Cells report percentages and retained sample counts; asterisks indicate . Rows correspond to Qwen2.5-VL-3B, Gemma-3-4B, InternVL3-8B, and LLaVA-OneVision-7B. (c) Visual and text-oracle validity across eight checkpoints at . (d) Within-family visual validity across model sizes at ; the vertical axis spans 0–26%. Panels (a), (c), and (d) use 400 instances per condition. All rates are point estimates.
Figure 3: Behavioral diagnostics of the multi-evidence integration gap. Q3 and Q7 denote Qwen2.5-VL-3B and Qwen2.5-VL-7B; G4 and G12 denote Gemma-3-4B and Gemma-3-12B. V and T indicate visual-grid and text-oracle inputs, respectively. (a) Retrieval versus joint selection. At , models more reliably retrieve target items from identifying attribute descriptions than select a combination satisfying all constraints in the original task. (b) Constraint ablation. Cells report valid-answer rates (%) at ; darker green indicates higher performance. Full retains all constraint groups, whereas Num., Cat., and Rel. retain only numerical, categorical, and compatibility constraints, respectively. (c) Beyond subset selection. Visual-input exact-path accuracy decreases as the number of graph edges increases, although the target path remains three edges long. Each plotted condition contains 400 instances.
| Model | Local | Visual | Text |
|---|---|---|---|
| Qwen2.5-VL-3B | 100.00 | 39.00 | 27.75 |
| Qwen2.5-VL-7B | 100.00 | 42.50 | 45.50 |
| Gemma-3-4B | 100.00 | 13.25 | 43.00 |
| Gemma-3-12B | 100.00 | 40.50 | 71.25 |
Table 1: Correct edge facts, incorrect paths. Local: all-field accuracy over 16,000 isolated edge queries. Visual/Text: exact-path accuracy on 400 sixteen-edge instances. All values are percentages.
Figure 4: Decoding solution components and predicting answer correctness are different tests. Q3/Q7 denote Qwen2.5-VL-3B/7B; G4/G12 denote Gemma-3-4B/12B. (a,b) Correctness AUC for visual-grid and text-oracle prompts. (c,d) Candidate-membership macro-AUC for the same input conditions. Points show means over five nested-cross-validation seeds; error bars show the standard deviation across seeds. Each candidate-count group contains 400 instances. Layers are selected within the training folds. The dashed line marks AUC .
| Model | Layers | |||
|---|---|---|---|---|
| Qwen2.5-VL-3B | 99 | 100 | 100 | 13 |
| Qwen2.5-VL-7B | 37 | 15 | 10 | 9 |
| Gemma-3-4B | 34 | 8 | 6 | 14 |
| Gemma-3-12B | 47 | 15 | 13 | 9 |
| InternVL3-8B | 60 | 16 | 11 | 13 |
Table 2: Activation-patching coverage. Counts report evaluated clean-correct/corrupt-wrong pairs. Layers gives the number of tested decoder blocks for full-prompt replacement. Qwen2.5-VL-3B also receives image- and text-token interventions at eight shared blocks on the same pairs.
Figure 5: Recovery depends on both depth and replaced token positions. All panels use Qwen2.5-VL-3B with visual-grid inputs. (a) Valid-answer rates after full-prompt matched or mismatched replacement at . (b,c) Matched-minus-mismatched valid-answer rates for three replacement scopes at and , restricted to their shared tested blocks. Each scope uses the same 99 and 100 recipient pairs, respectively. Depth is the zero-based decoder-block index divided by the final block index. Shading shows pointwise paired-bootstrap 95% intervals; intervals in (a) are bootstrap intervals for each recovery rate.
Figure 6: Full-prompt recovery profiles vary across model families and sizes. Both panels show matched-minus-mismatched valid-answer rates at with visual-grid inputs. (a) Q3/Q7: Qwen2.5-VL-3B/7B. (b) G4/G12: Gemma-3-4B/12B; I8: InternVL3-8B. Shading shows pointwise paired-bootstrap 95% intervals. Normalized depth aligns relative block indices.
Appendix figures & tables53 assets
Supplementary material from the paper’s appendix.
Appendix
| Full | Numerical | Categorical | Compatibility | |
|---|---|---|---|---|
| 2 | 1.00 | 1.00 | 1.00 | 1.00 |
| 4 | 1.00 | 1.96 | 3.38 | 1.19 |
| 6 | 1.00 | 5.84 | 14.75 | 2.01 |
| 8 | 1.00 | 14.87 | 36.31 | 3.36 |
Table A1: Mean number of valid subsets after constraint ablation. Enumerated on the final 400 instances per candidate count. Required subset size remains fixed; this quantifies the change in the answer acceptance criterion.
| Model | Input | |||
|---|---|---|---|---|
| Q3 | V | 99.75 | 53.00 | 18.00 |
| Q3 | T | 100.00 | 67.75 | 29.50 |
| Q7 | V | 100.00 | 76.00 | 29.50 |
| Q7 | T | 100.00 | 83.00 | 36.25 |
| Q32 | V | 100.00 | 90.50 | 39.25 |
| Q32 | T | 100.00 | 86.50 | 49.25 |
Table A2: Global selection accuracy across candidate counts. Entries report the percentage of answers satisfying all selection constraints on 400 instances per candidate count. V and T denote visual-grid and text-oracle inputs, respectively. The results appear in the main cross-model figure.
| Model | Input | |||||
|---|---|---|---|---|---|---|
| Q3 | V | 4 | 41.00 | 71.00 | 39.00 | 61.00 |
| Q3 | V | 6 | 17.00 | 30.00 | 10.00 | 15.00 |
| Q3 | V | 8 | 10.00 | 4.00 | 3.00 | 6.00 |
| Q3 | T | 4 | 63.00 | 74.00 | 64.00 | 70.00 |
| Q3 | T | 6 | 30.00 | 36.00 | 22.00 | 30.00 |
| Q3 | T | 8 | 14.00 | 15.00 | 11.00 | 22.00 |
Table A3: Final selection predictions stratified by nominal conflict and distractor ratio . Each cell is validity (%) over 100 instances. These are disjoint subgroups of the main benchmark. Parameter labels describe generation settings; the realized number of distractors is capped by the number of non-solution items.
| Model | Cost | Weight | Value | Color | Type | Compat. | All |
|---|---|---|---|---|---|---|---|
| Q3 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| Q7 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| Q32 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| G4 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| G12 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| G27 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
Table A4: Single-card field recovery (%) over 8,000 item queries per model. All denotes that all six fields of an item are correct. This is an item-level metric; conditional selection uses all items in an instance.
| Model | V (%) | T (%) | ||
|---|---|---|---|---|
| Q3 | 2 | 400 | 99.75 | 100.00 |
| Q3 | 4 | 400 | 53.00 | 67.75 |
| Q3 | 6 | 400 | 18.00 | 29.50 |
| Q3 | 8 | 400 | 5.75 | 15.50 |
| Q7 | 2 | 400 | 100.00 | 100.00 |
| Q7 | 4 | 400 | 76.00 | 83.00 |
Table A5: Selection validity conditioned on all single-card local questions being correct. is the retained number of instances out of 400. V/T are separate global runs conditioned on the same local-query subset. Unlike the main same-grid control, these local questions show one card at a time.
| Model | Cost | Weight | Value | Color | Type | Compat. | All | |
|---|---|---|---|---|---|---|---|---|
| Q3 | 2 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| Q3 | 4 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| Q3 | 6 | 100.00 | 99.96 | 99.96 | 99.96 | 100.00 | 98.96 | 98.96 |
| Q3 | 8 | 100.00 | 99.91 | 100.00 | 100.00 | 100.00 | 98.88 | 98.78 |
| G4 | 2 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 |
| G4 | 4 | 98.81 | 99.69 | 99.81 | 100.00 | 100.00 | 98.44 | 97.19 |
Table A6: Same-grid local field recovery (%). All denotes all fields of one queried item. Each row contains local queries. The main-text heatmap already reports instance-level retention and conditional global validity.
| Model | Input | |||
|---|---|---|---|---|
| Q3 | V | 100.00 | 87.25 | 89.75 |
| Q3 | T | 100.00 | 83.00 | 83.50 |
| Q7 | V | 100.00 | 100.00 | 99.75 |
| Q7 | T | 100.00 | 100.00 | 98.25 |
| G4 | V | 100.00 | 87.75 | 82.75 |
| G4 | T | 100.00 | 98.50 | 90.75 |
Table A7: Exact target-set retrieval (%) for the remaining candidate counts. Each cell uses 400 instances; the main text reports .
| Model | Input | Full | Num. | Cat. | Rel. | |
|---|---|---|---|---|---|---|
| Q3 | V | 2 | 99.50 | 100.00 | 99.00 | 100.00 |
| Q3 | V | 4 | 52.00 | 72.00 | 97.25 | 61.75 |
| Q3 | V | 6 | 18.25 | 46.25 | 96.75 | 22.25 |
| Q3 | T | 2 | 100.00 | 100.00 | 100.00 | 100.00 |
| Q3 | T | 4 | 69.50 | 79.75 | 96.50 | 81.50 |
| Q3 | T | 6 | 31.50 | 46.50 | 95.75 | 50.50 |
Table A8: Constraint-ablation validity (%) at the remaining candidate counts, 400 instances per cell. Full is the within-suite prompt control; Num., Cat., and Rel. retain numerical, categorical, and compatibility requirements, respectively. Required subset size is retained throughout.
| Model | 4 edges | 8 edges | 12 edges |
|---|---|---|---|
| Q3 | 96.25 | 56.00 | 41.50 |
| Q7 | 99.00 | 83.00 | 60.75 |
| G4 | 96.00 | 77.00 | 56.75 |
| G12 | 99.25 | 88.25 | 80.75 |
Table A9: Text-oracle exact-path accuracy (%) at the remaining graph sizes, 400 instances per cell. The main text provides the full visual curve and the sixteen-edge text result.
| Model | Input | 4 edges | 8 edges | 12 edges | 16 edges |
|---|---|---|---|---|---|
| Q3 | V | 100.00 | 100.00 | 100.00 | 100.00 |
| Q3 | T | 100.00 | 100.00 | 100.00 | 100.00 |
| Q7 | V | 100.00 | 100.00 | 100.00 | 100.00 |
| Q7 | T | 100.00 | 100.00 | 100.00 | 100.00 |
| G4 | V | 100.00 | 100.00 | 100.00 | 100.00 |
| G4 | T | 100.00 | 100.00 | 100.00 | 100.00 |
Table A10: Nonempty parsed-path rates (%).
| Model | Source | Target | Cost |
|---|---|---|---|
| Q3 | 100.00 | 100.00 | 100.00 |
| Q7 | 100.00 | 100.00 | 100.00 |
| G4 | 100.00 | 100.00 | 100.00 |
| G12 | 100.00 | 100.00 | 100.00 |
Table A11: Local edge-field accuracy (%) over 16,000 isolated queries. The main-text all-field score is not repeated.
Figure A1: Qwen2.5-VL-3B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output.
Figure A2: Qwen2.5-VL-7B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output. Only the reconciled text condition is included.
Figure A3: Gemma-3-4B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output.
Figure A4: Gemma-3-12B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output.
Figure A5: InternVL3-8B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output.
Figure A6: LLaVA-OneVision-7B: fixed-layer correctness AUC, pooled and grouped by candidate count. V/T denote visual/text input. Each layer is independently evaluated by cross-validation; choosing a peak from these curves is exploratory, unlike nested layer selection. Hidden-state index 0 denotes the embedding output.
| Model | Input | AUC | Acc. | Positive | Depth | |
|---|---|---|---|---|---|---|
| I8 | V | 4 | 0.790 | 0.720 | 0.760 | 0.157 |
| I8 | V | 6 | 0.561 | 0.610 | 0.295 | 0.443 |
| I8 | V | 8 | 0.470 | 0.735 | 0.140 | 0.557 |
| I8 | T | 4 | 0.873 | 0.850 | 0.848 | 0.457 |
| I8 | T | 6 | 0.621 | 0.588 | 0.490 | 0.386 |
| I8 | T | 8 | 0.616 | 0.620 | 0.325 | 0.836 |
Table A12: Additional-model nested correctness probes. AUC and accuracy are means across outer folds. Positive rate describes class balance; depth is the mean selected hidden-state index normalized by the final index. These models are absent from the main repeated-probe figure.
| Model | Input | Shuffle AUC | SD | Shuffles/fold | |
|---|---|---|---|---|---|
| Q3 | V | 4 | 0.505 | 0.059 | 10 |
| Q3 | V | 6 | 0.486 | 0.081 | 10 |
| Q3 | V | 8 | 0.515 | 0.134 | 10 |
| Q3 | T | 4 | 0.516 | 0.089 | 10 |
| Q3 | T | 6 | 0.522 | 0.083 | 10 |
| Q3 | T | 8 | 0.501 | 0.102 | 10 |
Table A13: Training-label shuffle controls at the already selected layer. SD is across pooled shuffle evaluations, not a confidence interval or a nested permutation test. Each included run uses five outer folds and ten shuffles per fold. The main repeated-seed runs use zero shuffles.
| Model | Input | Positive | ||
|---|---|---|---|---|
| Q3 | V | 2 | 399 | 400 |
| Q3 | T | 2 | 400 | 400 |
| Q7 | V | 2 | 400 | 400 |
| Q7 | T | 2 | 400 | 400 |
| G4 | V | 2 | 400 | 400 |
| G4 | T | 2 | 400 | 400 |
Table A14: Candidate-count groups absent from the fixed-layer five-fold probe outputs. Positive counts refer to the final behavioral predictions; all remaining instances are negatives.
| Index | AUC | AUC SD | Accuracy |
|---|---|---|---|
| 0 | 0.500 | 0.000 | 0.010 |
| 1 | 0.785 | 0.159 | 0.985 |
| 2 | 0.854 | 0.025 | 0.985 |
| 3 | 0.881 | 0.031 | 0.983 |
| 4 | 0.937 | 0.023 | 0.982 |
| 5 | 0.970 | 0.010 | 0.982 |
Table A15: LLaVA-OneVision visual correctness at : all recorded fixed-layer scores. There are 396 positive and four negative instances. These small-minority-class results are reported separately from the five-outer-fold nested analysis. SD describes the recorded cross-validation folds.
| Model | Input | 13 | 23 | 37 | 51 | 79 | |
|---|---|---|---|---|---|---|---|
| Q3 | V | 4 | 0.754 | 0.754 | 0.714 | 0.740 | 0.759 |
| Q3 | V | 6 | 0.569 | 0.542 | 0.508 | 0.541 | 0.581 |
| Q3 | V | 8 | 0.594 | 0.598 | 0.547 | 0.543 | 0.598 |
| Q3 | T | 4 | 0.792 | 0.801 | 0.787 | 0.789 | 0.792 |
| Q3 | T | 6 | 0.606 | 0.628 | 0.664 | 0.669 | 0.588 |
| Q3 | T | 8 | 0.615 | 0.544 | 0.588 | 0.597 | 0.559 |
Table A16: Qwen2.5-VL-3B: per-seed held-out correctness AUC. The main text presents mean and SD; this table exposes sensitivity to the split without reprinting that summary.
| Model | Input | 13 | 23 | 37 | 51 | 79 | |
|---|---|---|---|---|---|---|---|
| Q7 | T | 4 | 0.821 | 0.815 | 0.851 | 0.829 | 0.810 |
| Q7 | T | 6 | 0.599 | 0.607 | 0.588 | 0.525 | 0.603 |
| Q7 | T | 8 | 0.565 | 0.557 | 0.635 | 0.576 | 0.548 |
Table A17: Qwen2.5-VL-7B: per-seed held-out correctness AUC. The main text presents mean and SD; this table exposes sensitivity to the split without reprinting that summary.
| Model | Input | 13 | 23 | 37 | 51 | 79 | |
|---|---|---|---|---|---|---|---|
| G4 | V | 4 | 0.645 | 0.642 | 0.682 | 0.640 | 0.670 |
| G4 | V | 6 | 0.585 | 0.575 | 0.644 | 0.613 | 0.585 |
| G4 | V | 8 | 0.458 | 0.502 | 0.588 | 0.571 | 0.484 |
| G4 | T | 4 | 0.765 | 0.753 | 0.778 | 0.775 | 0.782 |
| G4 | T | 6 | 0.657 | 0.646 | 0.653 | 0.668 | 0.630 |
| G4 | T | 8 | 0.583 | 0.593 | 0.612 | 0.623 | 0.592 |
Table A18: Gemma-3-4B: per-seed held-out correctness AUC. The main text presents mean and SD; this table exposes sensitivity to the split without reprinting that summary.
| Model | Input | 13 | 23 | 37 | 51 | 79 | |
|---|---|---|---|---|---|---|---|
| G12 | V | 4 | 0.798 | 0.770 | 0.778 | 0.766 | 0.744 |
| G12 | V | 6 | 0.524 | 0.531 | 0.587 | 0.564 | 0.545 |
| G12 | V | 8 | 0.451 | 0.549 | 0.525 | 0.509 | 0.573 |
| G12 | T | 4 | 0.893 | 0.883 | 0.870 | 0.879 | 0.851 |
| G12 | T | 6 | 0.644 | 0.620 | 0.558 | 0.610 | 0.611 |
| G12 | T | 8 | 0.537 | 0.587 | 0.502 | 0.561 | 0.556 |
Table A19: Gemma-3-12B: per-seed held-out correctness AUC. The main text presents mean and SD; this table exposes sensitivity to the split without reprinting that summary.
| Input | ID | 13 | 23 | 37 | 51 | 79 | Positive | |
|---|---|---|---|---|---|---|---|---|
| V | 4 | A | 0.993 | 0.998 | 0.995 | 0.998 | 0.998 | 0.792 |
| V | 4 | B | 0.967 | 0.978 | 0.980 | 0.969 | 0.973 | 0.745 |
| V | 4 | C | 0.967 | 0.970 | 0.960 | 0.967 | 0.979 | 0.755 |
| V | 4 | D | 0.976 | 0.991 | 0.994 | 0.990 | 0.985 | 0.708 |
| V | 6 | A | 0.949 | 0.959 | 0.945 | 0.961 | 0.955 | 0.500 |
| V | 6 | B | 0.845 | 0.838 | 0.862 | 0.846 | 0.829 | 0.510 |
Table A20: Qwen2.5-VL-3B: candidate-specific membership AUC for each seed. A–H denote candidate IDs. Positive is the fraction of instances whose valid subset contains that candidate. The main figure averages these candidate-level results.
| Input | ID | 13 | 23 | 37 | 51 | 79 | Positive | |
|---|---|---|---|---|---|---|---|---|
| V | 4 | A | 0.989 | 0.990 | 0.988 | 0.984 | 0.987 | 0.792 |
| V | 4 | B | 0.982 | 0.984 | 0.983 | 0.983 | 0.989 | 0.745 |
| V | 4 | C | 0.980 | 0.987 | 0.992 | 0.987 | 0.983 | 0.755 |
| V | 4 | D | 0.974 | 0.982 | 0.985 | 0.985 | 0.975 | 0.708 |
| V | 6 | A | 0.970 | 0.968 | 0.965 | 0.957 | 0.964 | 0.500 |
| V | 6 | B | 0.827 | 0.857 | 0.847 | 0.859 | 0.851 | 0.510 |
Table A21: Qwen2.5-VL-7B: candidate-specific membership AUC for each seed. A–H denote candidate IDs. Positive is the fraction of instances whose valid subset contains that candidate. The main figure averages these candidate-level results.
| Input | ID | 13 | 23 | 37 | 51 | 79 | Positive | |
|---|---|---|---|---|---|---|---|---|
| V | 4 | A | 0.992 | 0.991 | 0.992 | 0.981 | 0.995 | 0.792 |
| V | 4 | B | 0.976 | 0.972 | 0.966 | 0.978 | 0.971 | 0.745 |
| V | 4 | C | 0.975 | 0.986 | 0.982 | 0.982 | 0.981 | 0.755 |
| V | 4 | D | 0.973 | 0.982 | 0.986 | 0.980 | 0.978 | 0.708 |
| V | 6 | A | 0.927 | 0.916 | 0.928 | 0.916 | 0.920 | 0.500 |
| V | 6 | B | 0.859 | 0.870 | 0.862 | 0.845 | 0.872 | 0.510 |
Table A22: Gemma-3-4B: candidate-specific membership AUC for each seed. A–H denote candidate IDs. Positive is the fraction of instances whose valid subset contains that candidate. The main figure averages these candidate-level results.
| Input | ID | 13 | 23 | 37 | 51 | 79 | Positive | |
|---|---|---|---|---|---|---|---|---|
| V | 4 | A | 0.997 | 0.998 | 0.996 | 0.989 | 0.992 | 0.792 |
| V | 4 | B | 0.975 | 0.980 | 0.979 | 0.980 | 0.976 | 0.745 |
| V | 4 | C | 0.967 | 0.968 | 0.970 | 0.972 | 0.965 | 0.755 |
| V | 4 | D | 0.979 | 0.986 | 0.976 | 0.980 | 0.981 | 0.708 |
| V | 6 | A | 0.972 | 0.968 | 0.970 | 0.962 | 0.977 | 0.500 |
| V | 6 | B | 0.884 | 0.905 | 0.891 | 0.890 | 0.883 | 0.510 |
Table A23: Gemma-3-12B: candidate-specific membership AUC for each seed. A–H denote candidate IDs. Positive is the fraction of instances whose valid subset contains that candidate. The main figure averages these candidate-level results.
| Model | Input | Seed | Selected indices | |
|---|---|---|---|---|
| Q3 | V | 4 | 13 | 25,28,28,27,28 |
| Q3 | V | 4 | 23 | 28,26,26,29,22 |
| Q3 | V | 4 | 37 | 26,29,21,28,26 |
| Q3 | V | 4 | 51 | 26,28,22,21,21 |
| Q3 | V | 4 | 79 | 28,25,26,28,22 |
| Q3 | V | 6 | 13 | 2,28,28,27,5 |
Table A24: Selected hidden-state indices for the five outer folds in each correctness run. These support interpretation of split sensitivity.
Figure A7: Q3: full-prompt recovery at , complementing the main comparisons. Both donor conditions use the same recipient pairs. Curves show absolute valid-answer rates. Points denote tested blocks.
Figure A8: Q7: full-prompt recovery at , complementing the main comparisons. Both donor conditions use the same recipient pairs. Curves show absolute valid-answer rates. Points denote tested blocks.
Figure A9: G4: full-prompt recovery at , complementing the main comparisons. Both donor conditions use the same recipient pairs. Curves show absolute valid-answer rates. Points denote tested blocks.
Figure A10: G12: full-prompt recovery at , complementing the main comparisons. Both donor conditions use the same recipient pairs. Curves show absolute valid-answer rates. Points denote tested blocks.
Figure A11: I8: full-prompt recovery at , complementing the main comparisons. Both donor conditions use the same recipient pairs. Curves show absolute valid-answer rates. Points denote tested blocks.
| Model | Block | 95% interval | ||
|---|---|---|---|---|
| G12 | 6 | 0 | 1.000 | [1.000, 1.000] |
| G12 | 6 | 9 | 1.000 | [1.000, 1.000] |
| G12 | 6 | 19 | 0.800 | [0.600, 1.000] |
| G12 | 6 | 26 | 0.400 | [0.133, 0.667] |
| G12 | 6 | 31 | 0.067 | [0.000, 0.200] |
| G12 | 6 | 33 | 0.067 | [0.000, 0.200] |
Table A25: Full-prompt paired donor contrasts at . Intervals use 2,000 recipient-level bootstrap resamples and are pointwise. Cohort sizes appear in the corresponding recovery panels and main coverage table. Boundary intervals can collapse when all observed differences are identical.
Figure A12: Qwen-3B token-scope contrasts at on 100 matched recipient pairs. Shading shows pointwise 95% paired-bootstrap intervals (2,000 resamples). Full prompt has 13 tested blocks; each restricted scope has eight. The main text reports the scope comparisons.
| Scope | Matched | Mismatched | |
|---|---|---|---|
| Full | 4 | 0.879 | 0.000 |
| Full | 6 | 0.930 | 0.530 |
| Full | 8 | 0.920 | 0.630 |
| Image | 4 | 0.455 | 0.081 |
| Image | 6 | 0.790 | 0.500 |
| Image | 8 | 0.770 | 0.600 |
Table A26: Decoy avoidance at Qwen-3B block 24. Each value is the fraction of patched outputs that omit the modified non-solution item. An output can avoid the decoy yet fail other constraints.
| Model | Bank | CC | CW | WC | WW | |
|---|---|---|---|---|---|---|
| Q3 | base | 4 | 296 | 145 | 78 | 481 |
| Q3 | base | 6 | 87 | 54 | 46 | 1013 |
| Q3 | ext1 | 6 | 85 | 75 | 47 | 1193 |
| Q3 | base | 8 | 55 | 76 | 41 | 2828 |
| Q3 | ext1 | 8 | 35 | 42 | 17 | 1706 |
| Q7 | base | 4 | 27 | 37 | 6 | 30 |
Table A27: Pair-bank outcome counts before intervention subsampling. CC/CW/WC/WW refer to clean and corrupted correctness, respectively. Only CW pairs are eligible. Qwen-3B uses a larger base bank plus independent extensions; eligible counts can exceed the evaluated counts in the main table.
| Model | Input | Correct/ | Validity [95% CI] | |
|---|---|---|---|---|
| Q3 | V | 2 | 399/400 | 99.75 [98.60, 99.96] |
| Q3 | V | 4 | 212/400 | 53.00 [48.10, 57.84] |
| Q3 | V | 6 | 72/400 | 18.00 [14.55, 22.06] |
| Q3 | V | 8 | 23/400 | 5.75 [3.86, 8.48] |
| Q3 | T | 2 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | T | 4 | 271/400 | 67.75 [63.02, 72.14] |
Table A28: Selection validity with 95% Wilson intervals. Each row uses 400 final benchmark instances.
| Model | Retained/400 | Recovery [95% CI] | |
|---|---|---|---|
| Q3 | 2 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | 4 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | 6 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | 8 | 400/400 | 100.00 [99.05, 100.00] |
| Q7 | 2 | 400/400 | 100.00 [99.05, 100.00] |
| Q7 | 4 | 400/400 | 100.00 [99.05, 100.00] |
Table A29: Isolated-card complete local recovery at the instance level: every field of every item must be correct. Denominator: 400 instances per row.
| Model | Input | Correct/retained | Validity [95% CI] | |
|---|---|---|---|---|
| Q3 | V | 2 | 399/400 | 99.75 [98.60, 99.96] |
| Q3 | T | 2 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | V | 4 | 212/400 | 53.00 [48.10, 57.84] |
| Q3 | T | 4 | 271/400 | 67.75 [63.02, 72.14] |
| Q3 | V | 6 | 72/400 | 18.00 [14.55, 22.06] |
| Q3 | T | 6 | 118/400 | 29.50 [25.24, 34.15] |
Table A30: Isolated-card conditional selection validity. The denominator is the retained cohort, not 400; these intervals describe that selected cohort.
| Model | Retained/400 | Recovery [95% CI] | |
|---|---|---|---|
| Q3 | 2 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | 4 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | 6 | 376/400 | 94.00 [91.23, 95.94] |
| Q3 | 8 | 362/400 | 90.50 [87.23, 93.00] |
| G4 | 2 | 400/400 | 100.00 [99.05, 100.00] |
| G4 | 4 | 356/400 | 89.00 [85.55, 91.70] |
Table A31: Same-grid complete local recovery at the instance level: every field of every item must be correct. Denominator: 400 instances per row.
| Model | Input | Correct/retained | Validity [95% CI] | |
|---|---|---|---|---|
| Q3 | V | 2 | 399/400 | 99.75 [98.60, 99.96] |
| Q3 | V | 4 | 212/400 | 53.00 [48.10, 57.84] |
| Q3 | V | 6 | 66/376 | 17.55 [14.04, 21.72] |
| Q3 | V | 8 | 21/362 | 5.80 [3.83, 8.71] |
| G4 | V | 2 | 400/400 | 100.00 [99.05, 100.00] |
| G4 | V | 4 | 192/356 | 53.93 [48.74, 59.04] |
Table A32: Same-grid conditional selection validity. The denominator is the retained cohort, not 400; these intervals describe that selected cohort.
| Model | Input | Correct/ | Rate [95% CI] | |
|---|---|---|---|---|
| Q3 | V | 2 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | V | 4 | 349/400 | 87.25 [83.62, 90.17] |
| Q3 | V | 6 | 359/400 | 89.75 [86.39, 92.35] |
| Q3 | V | 8 | 341/400 | 85.25 [81.44, 88.39] |
| Q3 | T | 2 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | T | 4 | 332/400 | 83.00 [79.01, 86.36] |
Table A33: Target-set retrieval with 95% Wilson intervals; 400 instances per row.
| Model | Input | Scope | Correct/ | Rate [95% CI] | |
|---|---|---|---|---|---|
| Q3 | V | 2 | Full | 398/400 | 99.50 [98.20, 99.86] |
| Q3 | V | 4 | Full | 208/400 | 52.00 [47.11, 56.85] |
| Q3 | V | 6 | Full | 73/400 | 18.25 [14.77, 22.33] |
| Q3 | V | 8 | Full | 24/400 | 6.00 [4.06, 8.77] |
| Q3 | V | 2 | Num. | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | V | 4 | Num. | 288/400 | 72.00 [67.41, 76.17] |
Table A34: Constraint-ablation validity with 95% Wilson intervals; 400 instances per row.
| Model | Input | Correct/ | Rate [95% CI] | |
|---|---|---|---|---|
| Q3 | V | 4 | 392/400 | 98.00 [96.10, 98.98] |
| Q3 | V | 8 | 324/400 | 81.00 [76.87, 84.54] |
| Q3 | V | 12 | 212/400 | 53.00 [48.10, 57.84] |
| Q3 | V | 16 | 156/400 | 39.00 [34.35, 43.86] |
| Q3 | T | 4 | 385/400 | 96.25 [93.91, 97.71] |
| Q3 | T | 8 | 224/400 | 56.00 [51.10, 60.78] |
Table A35: Exact-path accuracy with 95% Wilson intervals; 400 instances per row.
| Model | Edges | Correct graphs/ | Recovery [95% CI] |
|---|---|---|---|
| Q3 | 4 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | 8 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | 12 | 400/400 | 100.00 [99.05, 100.00] |
| Q3 | 16 | 400/400 | 100.00 [99.05, 100.00] |
| Q7 | 4 | 400/400 | 100.00 [99.05, 100.00] |
| Q7 | 8 | 400/400 | 100.00 [99.05, 100.00] |
Table A36: Complete isolated-edge recovery per graph. A success requires every field of every edge to be correct. The graph, rather than an edge query, is the sampling unit.
| Model | Direct | Review | ES | ESV | |
|---|---|---|---|---|---|
| Qwen2.5-VL-3B | 4 | 53.25 | 7.00 | 62.25 | 59.25 |
| 6 | 17.50 | 2.75 | 16.50 | 15.50 | |
| 8 | 5.75 | 1.50 | 5.25 | 5.25 | |
| Qwen2.5-VL-7B | 4 | 71.50 | 0.75 | 74.75 | 74.25 |
| 6 | 28.75 | 0.25 | 26.25 | 26.25 | |
| 8 | 11.75 | 0.00 | 10.50 | 10.50 |
Table A37: Prompting controls produce model-dependent changes without consistently resolving difficult global decisions. Entries are exact globally valid answers (%) over 400 paired visual-grid instances. ES denotes Extract–Select and ESV denotes Extract–Select–Verify.
| Checkpoint | Condition | Valid [95% CI] | Final JSON | At cap | |
|---|---|---|---|---|---|
| Qwen3-VL-8B Instruct | Answer only | 195 | 5.64 [3.18, 9.82] | 99.49 | 0.00 |
| Explicit reasoning | 195 | 48.21 [41.29, 55.19] | 53.33 | 0.00 | |
| Qwen3-VL-8B Thinking | Answer only | 195 | 0.00 [0.00, 1.93] | 0.00 | 100.00 |
| Explicit reasoning | 195 | 0.51 [0.09, 2.85] | 0.51 | 100.00 |
Table A38: Explicit reasoning changes global validity on paired eight-candidate visual instances. Valid reports exact globally valid answers with pointwise Wilson 95% intervals. Final JSON is the share of responses containing a parseable item-set object. At cap is the share whose final generation reaches its token budget.
Figure A13: Explicit reasoning repairs some decisions and can also overturn valid answers. Each horizontal bar partitions the same 195 instances by their paired answer-only and explicit-reasoning outcomes. Repaired denotes an invalid answer that becomes globally valid; regressed denotes a valid answer that becomes invalid. Qwen3-VL-8B-Instruct and Qwen3-VL-8B-Thinking use identical instances and deterministic decoding.
| Model | Direct | ES | Extract–Solve [95% CI] | Direct | 95% CI for | |
|---|---|---|---|---|---|---|
| Qwen2.5-VL-3B | 4 | 53.25 | 62.25 | 36.50 [31.93, 41.33] | [ ] | |
| 6 | 17.50 | 16.50 | 82.25 [78.20, 85.68] | [ ] | ||
| 8 | 5.75 | 5.25 | 92.25 [89.21, 94.49] | [ ] | ||
| Qwen2.5-VL-7B | 4 | 71.50 | 74.75 | 84.00 [80.09, 87.27] | [ ] | |
| 6 | 28.75 | 26.25 | 94.50 [91.81, 96.34] | [ ] | ||
| 8 | 11.75 | 10.50 | 96.75 [94.52, 98.09] | [ ] |
Table A39: Extract–Solve converts usable transcriptions into globally valid decisions. All entries are percentages over 400 paired instances. ES denotes Extract–Select. Brackets beside Extract–Solve report pointwise Wilson 95% intervals. The final column reports 95% intervals for the paired difference from Direct, estimated with 10,000 bootstrap draws stratified by conflict intensity and distractor ratio.
Figure A14: Outcome decomposition of the semantic Extract–Solve pipeline. Each bar contains 400 instances. Solved denotes transcriptions that yield one unique feasible solution; all 3,137 instances in this category match the benchmark target. Incomplete denotes missing item cards, Invalid covers malformed JSON or invalid fields, and Inconsistent combines zero- and multiple-solution outcomes. Q3 and Q7 denote Qwen2.5-VL-3B and Qwen2.5-VL-7B; G4 and G12 denote Gemma-3-4B and Gemma-3-12B.
More in this topic
VLM Evaluation
Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.
See all 989 papers