cs.CVNov 11, 2026

Back to the Question: Tracing Visual Distraction Through Text Tokens

Authors: Marie Amale Huynh, Regev Cohen, Serena Yeung-Levy, Sivan Doveh

Organizations: Stanford University · Reichman University

Abstract

The visual world is cluttered: robust vision-language reasoning requires ignoring irrelevant information. Yet vision-language models (VLMs) can suffer substantial accuracy losses when answering questions about a target image with distractors present. Existing mitigations focus on selecting relevant images or limiting information exchange between images. Through a systematic analysis of three open 7-8B VLMs on five perception benchmarks adapted to include distractor images, we show that distractors reduce accuracy even when models correctly identify the relevant image, suggesting that selection alone is not enough. To understand the mechanism behind this interference, we combine attention analysis with targeted knockouts in Qwen2.5-VL and identify middle-layer distractor-to-text connections as a pathway through which irrelevant visual information affects reasoning. We introduce answer consistency regularization, which encourages consistent answer distributions with and without distractors. Without directly constraining attention, this objective reduces middle-layer attention from question tokens to distractor images, consistent with the pathway our analysis implicates. With five distractors, our approach improves circular accuracy by an average of 16 percentage points over the Qwen2.5-VL baseline across the five benchmarks while preserving single-image performance.

Figures & tables

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Understanding the Effects of Distractors on Reasoning Vision-Language Models

    Nov 26, 2025Jiyun Bae, Hyunjong Ok, Sangwoo Mo +1VLM ReasoningVisual Question Answering

  2. Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?

    Jun 8, 2026Yizheng Sun, Mochuan Zhan, Yanan Ma +10VLM EvaluationVLM Robustness