cs.CVNov 16, 2026

Seen but Unsaid: Locating and Repairing Cross-Image Comparison in Vision-Language Models

Authors: Zhaoning Zhang

Organizations: University of Toronto

Abstract

Vision-language models (VLMs) often fail to match a point or an object across two images, even when the answer can be decoded from their own hidden states. We trace this failure from the visual tokens to the answer letter, in Qwen3-VL (2B-32B), InternVL3.5-8B, and Gemma-3-12B, on synthetic shape matching, same-instance point correspondence (HPatches), and semantic correspondence (SPair-71k), and find two distinct failure points, a small set of attention heads that carries the comparison, and a repair that touches 0.02% of the parameters. Two failure points: exact copies are compared but not read out. An answer-position probe exceeds the calibrated output by 15 to 25 points in every family, for nameable and unnameable shapes alike, and steering along the probe's direction moves no more answers than random directions. Semantic correspondences are present in the visual tokens but never reach the answer. One pathway: redistributing the cross-image attention of 16-32 heads, with its mass preserved, removes 17-22 points of synthetic and 15-32 points of HPatches accuracy, against at most 7 for random or attention-matched heads. A lever that training finds on its own: training only these heads' query and key projections raises SPair-71k accuracy by 28.3 points (Qwen3-VL-8B) and 25.1 points (InternVL3.5-8B), beats both layer-matched and attention-matched heads, and transfers to HPatches and BLINK. An unconstrained all-layer adapter routes 21-51% of its gain through the attention of the same heads, against at most 7% for random heads, and does so by changing what they read (83-86% of the effect) rather than their weights. Every decision rule was frozen before its run, and every reported number is recomputed by independent verification scripts.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models

    Sep 17, 2026Farooq Ahmad Wani, Maria Sofia Bucarelli, Mujtaba Hussain Mirza +5VLM RobustnessPrompt Sensitivity

  2. Composition, Not Conversation: VLMs Lose the Scene, Not the Thread

    Sep 29, 2026L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5VLM EvaluationVisual Question Answering

  3. Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models

    Jul 6, 2026Khang Nhat Hoang Vo, Artem Vazhentsev, Artem Shelmanov +2VLM EvaluationVision-Language Models