Seen but Unsaid: Locating and Repairing Cross-Image Comparison in Vision-Language Models
Organizations: University of Toronto
Abstract
Vision-language models (VLMs) often fail to match a point or an object across two images, even when the answer can be decoded from their own hidden states. We trace this failure from the visual tokens to the answer letter, in Qwen3-VL (2B-32B), InternVL3.5-8B, and Gemma-3-12B, on synthetic shape matching, same-instance point correspondence (HPatches), and semantic correspondence (SPair-71k), and find two distinct failure points, a small set of attention heads that carries the comparison, and a repair that touches 0.02% of the parameters. Two failure points: exact copies are compared but not read out. An answer-position probe exceeds the calibrated output by 15 to 25 points in every family, for nameable and unnameable shapes alike, and steering along the probe's direction moves no more answers than random directions. Semantic correspondences are present in the visual tokens but never reach the answer. One pathway: redistributing the cross-image attention of 16-32 heads, with its mass preserved, removes 17-22 points of synthetic and 15-32 points of HPatches accuracy, against at most 7 for random or attention-matched heads. A lever that training finds on its own: training only these heads' query and key projections raises SPair-71k accuracy by 28.3 points (Qwen3-VL-8B) and 25.1 points (InternVL3.5-8B), beats both layer-matched and attention-matched heads, and transfers to HPatches and BLINK. An unconstrained all-layer adapter routes 21-51% of its gain through the attention of the same heads, against at most 7% for random heads, and does so by changing what they read (83-86% of the effect) rather than their weights. Every decision rule was frozen before its run, and every reported number is recomputed by independent verification scripts.
Figures & tables
| Model | Condition | (pp) |
|---|---|---|
| Qwen3-VL-2B | squiggle | [ , ] |
| Qwen3-VL-2B | known | [ , ] |
| Qwen3-VL-8B | squiggle | [ , ] |
| Qwen3-VL-8B | known | [ , ] |
| Model | Task | [CI] | |||
|---|---|---|---|---|---|
| Qwen-8B | HP | .770 | .778 | .804 | [ , ] |
| SP | .438 | .460 | .433 | [ , ] | |
| InternVL-8B | HP | .543 | .536 | .542 | [ , ] |
| SP | .355 | .369 | .310 | [ , ] | |
| Gemma-12B | HP | .590 | .591 | .601 | [ , ] |
| SP | .407 | .404 | .351 | [ , ] |
| Model | Syn -squiggle | HP | SP |
|---|---|---|---|
| Qwen3-VL-8B | |||
| InternVL3.5-8B | |||
| Gemma-3-12B |
| Task | Base | Uniform | Shuffle | Block | Ctrl | |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B | ||||||
| Syn | 41.2 | 17.2 (18.0) | 18.2 (19.3) | 17.4 ( 0 3.1) | 3.6 | |
| HP | 76.2 | 32.2 (31.8) | 31.8 (31.4) | 21.2 (20.9) | 2.4 | |
| SP | 46.6 | 0 2.2 ( 0 2.4) | 0 3.2 ( 0 3.7) | 0 3.6 ( 0 3.8) | 2.0 | |
| InternVL3.5-8B | ||||||
| Syn | 48.6 | 20.2 (19.9) | 22.0 (22.0) | 22.4 (22.2) | 6.8 | |
| Model | Task | [95% CI] | AUC | |
|---|---|---|---|---|
| Qwen-8B | Syn | [ , ] | 10.36 | .48 |
| HP | [ , ] | 1.88 | .70 | |
| SP | [ , ] | 1.30 | .66 | |
| InternVL-8B | Syn | [ , ] | 2.88 | .69 |
| HP | [ , ] | 1.32 | .62 | |
| SP | [ , ] | 1.13 | .56 |
| Arm (trainable parameters) | Params | SP | HP | BLINK-SC | BLINK-VC | Syn -squiggle | Syn -known |
|---|---|---|---|---|---|---|---|
| Qwen3-VL-8B | |||||||
| base (none) | – | .430 | .782 | .367 | .669 | .414 | .686 |
| A: QK of | 1.68M | .712 | .934 | .549 | .802 | .551 | .833 |
| A Q : Q of | 0.85M | .674 | .922 | .504 | .791 | .480 | .842 |
| B: QK of random heads | 1.69M | .677 | .899 | .499 | .808 | .451 | .701 |
| C: V of | 1.65M | .747 | .925 | .561 | .876 | .488 | .785 |
| Claim | Evidence | Controls | Where |
|---|---|---|---|
| Exact copies are compared but not read out | ( Qwen3-VL-8B , Syn -squiggle); to in every family; answer-position edits move by at most ; probe-direction steering at the level of random directions | Output calibration; identical probe protocol on HP / SP (gap ); 20 magnitude-matched random subspaces; letter-direction positive control (switches – ) | § 4.1 – 4.2 |
| The gap is not about nameability | Gap difference squiggle vs. known: ( Qwen3-VL-8B ), ( Qwen3-VL-2B ) | Size- and stroke-matched nameable shapes | § 4.1 |
| Semantic correspondence is present in visual tokens | on SP : / , for Gemma-3-12B | Marker-free images; layers chosen on development data | § 4.3 |
| A small set of cross-image heads is causal | Mass-preserving knockout of : to on Syn , to on HP | 10 random and 3 attention-matched head sets ( ); in-band items; within-image-2 knockout | § 5.1 |
| Their attention is salience-gated and content-weighted | : HP / , SP / ; decoy markers raise to ; moved marker makes the new target selective ( ) | Fresh items; head sets and rules frozen in advance | § 5.2 |
| Training these heads repairs semantic correspondence | SP / ; transfer to HP / and BLINK | Layer-matched random heads (A B / ); attention-matched heads (A M / ); query-only arms (A B Q / , A M Q / ) | § 6.1 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Run | Question | Frozen outcome |
|---|---|---|
| Readout ladder | Where does the Syn gap fall (readout vs. retention/computation)? | Qwen3-VL-8B : readout-failure candidate; Qwen3-VL-2B : retention/computation candidate (within 0.2 pp of the rule boundary). |
| Gap diagnostics | Is the gap specific to unnameable shapes? Letter or position coding? Probe saturation? Composition? | Gap not nameability-specific in either model; mixed letter/position coding; Qwen3-VL-8B probe not saturated; Qwen3-VL-8B composition gap. |
| Calibration | Does output calibration close the gap? | Qwen3-VL-8B : gap persists on both conditions; Qwen3-VL-2B : partial (squiggle), closed (known). |
| Breadth and scale | Is the gap general across tasks and families? | Not general across tasks: present on Syn -squiggle in every family, absent on HP and SP . |
| Privileged readout, real images | Is correspondence information present in visual tokens? | SP : present in two of three models (third below threshold); no model without information. |
| Cross-image heads | Do causal heads exist; does per-option retrieval dissociate across tasks? | Heads causal in both models; per-option retrieval criteria not met (mixed). |
| Symbol | Meaning |
|---|---|
| Behavioural accuracy: argmax over the four letter log-probabilities at the scoring position. | |
| Accuracy after multinomial logistic calibration of the four letter log-probabilities, fitted on development data. | |
| Input-matched probe on the answer-position residual stream (PCA to 256 dimensions, multinomial logistic regression). | |
| , | Privileged readouts over annotated regions: training-free MaxSim score ( ) and paired probe ( ). |
| , with counted only if its selectivity over shuffled labels is at least . | |
| Cross-image head set: (32 heads, layers 1–24) for Qwen3-VL-8B , (16 heads, layers 3–23) for InternVL3.5-8B . |
| Comparison | Qwen3-VL-8B | InternVL3.5-8B |
|---|---|---|
| A base ( ) | [ , ] | [ , ] |
| A mean(B) | [ , ] | [ , ] |
| A mean(B), hierarchical | [ , ] | [ , ] |
| A Q mean(B Q ), hierarchical | [ , ] | [ , ] |
| A D | ||
| A C | – |
| Model | Arm | Seed | Gain (pp) | Removed by Restore | Recovered by Transplant |
|---|---|---|---|---|---|
| Qwen3-VL-8B | A | 0 | [ , ] | ||
| A | 1 | ||||
| A | 2 | ||||
| C | 0 | ||||
| D | 0 | [ , ] | [ , ] | ||
| D | 1 |
| Model | Task | Base | Ablate | Amplify | Amplify | Random range |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B | Syn -squiggle | .413 | to | |||
| Qwen3-VL-8B | HP | .767 | to | |||
| InternVL3.5-8B | Syn -squiggle | .514 | to | |||
| InternVL3.5-8B | HP | .543 | to |
| Model | Arm | Gain (pp) | Restore-all | Random Restore-all | Restore-group |
|---|---|---|---|---|---|
| Qwen3-VL-8B | A | [ , ] | to | ||
| Qwen3-VL-8B | D | [ , ] | to | [ , ] | |
| InternVL3.5-8B | A | [ , ] | to | ||
| InternVL3.5-8B | D | [ , ] | to | [ , ] |
| Model | Set | ratio | ratio | ratio |
|---|---|---|---|---|
| Qwen3-VL-8B | M 0 / M 1 / M 2 | .985 / .950 / .803 | .107 / .043 / .081 | 1.670 / 1.575 / 1.319 |
| B 0 / B 1 / B 2 | .350 / .430 / .368 | .131 / .152 / .127 | .448 / .554 / .487 | |
| InternVL3.5-8B | M 0 / M 1 / M 2 | .969 / .875 / .827 | .101 / .125 / .105 | 1.709 / 1.390 / 1.401 |
| B 0 / B 1 / B 2 | .316 / .467 / .464 | .017 / .115 / .112 | .522 / .701 / .717 |
| Model | Comparison | arm | Matched (per set) | B mean | Difference | Paired CI | Hierarchical CI |
|---|---|---|---|---|---|---|---|
| Qwen3-VL-8B | A M | 71.24 | 67.90 (68.28, 67.40, 68.03) | 67.66 | [ , ] | [ , ] | |
| A Q M Q | 67.35 | 65.23 (65.03, 66.03, 64.65) | 63.82 | [ , ] | [ , ] | ||
| InternVL3.5-8B | A M | 60.51 | 58.03 (57.38, 59.18, 57.53) | 58.38 | [ , ] | [ , ] | |
| A Q M Q | 58.05 | 53.73 (52.78, 54.87, 53.52) | 53.52 | [ , ] | [ , ] |
| Model | Arms | HP | BLINK-SC | BLINK-VC | Syn -squiggle | Syn -known |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B | A / M | 93.4 / 91.7 | 54.9 / 48.9 | 80.2 / 79.8 | 55.1 / 51.1 | 83.3 / 79.1 |
| A Q / M Q | 92.2 / 89.7 | 50.4 / 45.6 | 79.1 / 74.6 | 48.0 / 49.5 | 84.2 / 79.0 | |
| InternVL3.5-8B | A / M | 83.7 / 82.5 | 36.7 / 33.6 | 56.2 / 57.2 | 36.6 / 47.3 | 77.6 / 91.2 |
| A Q / M Q | 83.7 / 79.0 | 41.0 / 34.8 | 55.8 / 52.9 | 31.5 / 47.8 | 78.4 / 92.9 |
| Model | Task | Letter switch, | Random max | Probe moved | Random moved |
|---|---|---|---|---|---|
| Qwen3-VL-8B | Syn -squiggle | 0.4 / 0.4 / 0.8 / 2.8 | 0.4 | 0.4 | 0.2–0.4 |
| HP | 0.8 / 1.2 / 3.0 / 9.6 | 0.8 | 0.4 | 0.0–0.6 | |
| InternVL3.5-8B | Syn -squiggle | 0.6 / 0.6 / 1.8 / 3.8 | 0.4 | 0.9 | 0.0–1.5 |
| HP | 3.2 / 4.6 / 9.4 / 18.6 | 1.8 | 1.6 | 0.9–1.6 |
| Model | Arm | Split | Gain (pp) | Full | Upstream | Local | Interaction | Rank |
|---|---|---|---|---|---|---|---|---|
| Qwen3-VL-8B | D | SP | 34.8 | .339 [.268, .404] | .282 [.207, .348] | .043 [ .013, .095] | .014 [ .059, .088] | 1 / 1 / 1 |
| D | HP | 16.2 | .585 [.417, .759] | .692 [.550, .857] | .077 [ .062, .207] | .185 [ .339, .034] | 1 / 1 / 1 | |
| E | SP | 35.3 | .411 [.340, .484] | .360 [.291, .435] | .028 [ .012, .066] | .023 [ .031, .078] | 1 / 1 / 2 | |
| E | HP | 16.8 | .806 [.672, .953] | .791 [.646, .962] | .090 [.000, .190] | .075 [ .213, .061] | 1 / 1 / 2 | |
| InternVL3.5-8B | D | SP | 37.4 | .270 [.201, .339] | .233 [.166, .298] | .035 [.000, .068] | .003 [ .041, .054] | 1 / 1 / 1 |
| D | HP | 39.0 | .462 [.366, .558] | .442 [.344, .539] | .026 [ .029, .077] | .006 [ .078, .069] | 1 / 1 / 1 |
| Task | Operation | drop | drop | Random | Matched | Out |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B (base / : Syn 41.2 / 62.2, HP 76.2 / 81.2, SP 46.6 / 42.0) | ||||||
| Syn | Uniform | 17.2 [12.6, 22.0] | 38.2 [32.6, 43.8] | 2.8 | 0.6 | 4.6 |
| Shuffle | 18.2 [13.8, 22.8] | 37.6 [31.6, 43.2] | 3.0 | 0.2 | 5.6 | |
| Block | 17.4 [12.2, 22.8] | 34.8 [28.8, 40.4] | 3.6 | 0.8 | 49.0 | |
| HP | Uniform | 32.2 [27.6, 37.0] | 35.0 [30.0, 39.8] | 1.2 | 1.2 | 5.6 |
| Shuffle | 31.8 [27.2, 36.4] | 32.4 [27.8, 37.6] | 1.0 | 1.8 | 5.8 | |