Back to the Question: Tracing Visual Distraction Through Text Tokens
Organizations: Stanford University · Reichman University
Abstract
The visual world is cluttered: robust vision-language reasoning requires ignoring irrelevant information. Yet vision-language models (VLMs) can suffer substantial accuracy losses when answering questions about a target image with distractors present. Existing mitigations focus on selecting relevant images or limiting information exchange between images. Through a systematic analysis of three open 7-8B VLMs on five perception benchmarks adapted to include distractor images, we show that distractors reduce accuracy even when models correctly identify the relevant image, suggesting that selection alone is not enough. To understand the mechanism behind this interference, we combine attention analysis with targeted knockouts in Qwen2.5-VL and identify middle-layer distractor-to-text connections as a pathway through which irrelevant visual information affects reasoning. We introduce answer consistency regularization, which encourages consistent answer distributions with and without distractors. Without directly constraining attention, this objective reduces middle-layer attention from question tokens to distractor images, consistent with the pathway our analysis implicates. With five distractors, our approach improves circular accuracy by an average of 16 percentage points over the Qwen2.5-VL baseline across the five benchmarks while preserving single-image performance.
Figures & tables
| Benchmark | Competency | Subsets | |
| CLEVR ( Johnson et al., 2017 ) | Size reasoning | size | 3,554 |
| CV-Bench ( Tong et al., 2024 ) | Vision-centric spatial | count, relation, depth, distance | 2,560 |
| What’sUp ( Kamath et al., 2023 ) | Spatial relations | controlled_a, controlled_b | 820 |
| BLINK ( Fu et al., 2024 ) | Low-level perception | spatial relation, relative reflectance, relative depth, localization, counting | 618 |
| MIMIC ( Das et al., 2026 ) | Counting | count | 1,500 |
| Total | 9,052 |
| Benchmark | Qwen2.5-VL | InternVL3 | LLaVA-OV | |||||||||
| CLEVR (size) | 99.9 | 99.7 | 99.7 | 99.4 | 99.9 | 98.6 | 96.4 | 84.1 | 93.4 | 66.0 | 49.5 | 29.8 |
| CV-Bench | 96.3 | 94.5 | 92.9 | 88.0 | 94.1 | 91.0 | 84.5 | 61.6 | 92.0 | 57.0 | 46.2 | 25.9 |
| MIMIC | 99.7 | 99.5 | 99.2 | 99.1 | 99.9 | 99.1 | 96.3 | 80.5 | 99.0 | 76.3 | 67.1 | 34.3 |
| BLINK | 89.5 | 85.6 | 81.7 | 81.7 | 88.2 | 83.8 | 75.4 | 51.9 | 72.8 | 53.9 | 43.7 | 26.5 |
| What’sUp | 94.8 | 92.2 | 90.9 | 85.4 | 94.8 | 91.7 | 76.7 | 57.7 | 93.9 | 73.7 | 64.9 | 34.3 |
| cond. | route | recov./broken | range | macro | broken/right |
| P | IMG D everything | 0.56 [0.54, 0.58] | 0.46–0.66 | 0.56 | 0.05 [0.04, 0.05] |
| MM | all preceding IMG (MIMIC) | 0.10 [0.09, 0.12] | 0.04–0.53 | 0.20 | 0.57 [0.55, 0.58] |
| D | IMG D all text | 0.47 [0.45, 0.49] | 0.44–0.54 | 0.49 | 0.05 [0.04, 0.06] |
| R | IMG D readout | 0.03 [0.03, 0.04] | 0.02–0.07 | 0.04 | 0.02 [0.01, 0.02] |
| Per benchmark ( mean) | Pooled | ||||||||
| Model | CLEVR | CV-B | MIMIC | WhatsUp | BLINK | mean | |||
| Qwen2.5-VL-7B (base) | 49.9 | 65.1 | 64.1 | 70.6 | 47.0 | 70.7 | 51.4 | 59.3 | 0.0 |
| Ours | |||||||||
| CE only | 77.9 | 69.1 | 56.4 | 77.0 | 52.6 | 72.0 | 64.1 | 66.6 | +7.3 |
| CE + KL | 76.1 | 68.5 | 60.6 | 77.5 | 53.0 | 73.7 | 63.8 | 67.1 | +7.8 |
| CE + KL | 75.0 | 69.5 | 64.8 | 79.8 | 55.0 | 74.2 | 66.0 | 68.8 | +9.5 |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Pipeline Steps |
| CLEVR | 1. Load questions filtered by category (size/color/count/material) 2. Build attribute index from scene graphs 3. Per-question vetting: exclude renders containing queried attribute 4. Sample distractors, shuffle with relevant image |
| What’sUp | 1. Parse caption to extract subject + spatial relation 2. Reframe as MCQ: “Where is/are the X?” 3. Domain-stratified pool selection (tabletop vs. natural) 4. Structural exclusion (own image, other subsets’ images) 5. VLM answerability gate (per-candidate check) 6. Sample distractors, shuffle with relevant image |
| BLINK | 1. Load COCO test2017 as candidate pool for distractors 2. Remove duplicate images 3. VLM answerability gate 4. Sample distractors, shuffle with relevant image |
| CV-Bench | 1. Load COCO test2017 as candidate pool for distractors 2. Route: COCO test2017 pool (if queried classes are COCO categories) or CV-Bench pool (if not) 3. COCO route: VLM answerability gate 4. Own pool route: VLM answerability gate 5. Draw colored marker boxes on distractors where the question names one (1,619 rows) 6. Sample distractors, shuffle with relevant image |
| MIMIC | 1. Load the item’s native MIMIC distractors 2. Subsample of them (no external pool) 3. Shuffle with relevant image |
| Benchmark | Pool Source | Pool Size | Domain Type |
| CLEVR | CLEVR val renders | 15,000 | In-domain |
| What’sUp (controlled_a/b) | controlled_images + controlled_clevr | 829 | In-domain |
| What’sUp (coco_qa) | COCO val2017 | 5,000 | In-domain |
| What’sUp (vg_qa) | Visual Genome | 4,797 | In-domain |
| BLINK | COCO test2017 | 40,641 | Matched-domain |
| CV-Bench | COCO test2017 CV-Bench box-free | 40,641 940 | Hybrid |
| Benchmark | Gate Type | Description |
| CLEVR | Ground truth | Scene graphs provide exact attribute vetting; no VLM needed |
| What’sUp | Answerability | Per-candidate check: “Does this image contain any of the objects the question asks about, or could this image on its own support answering the question?” |
| BLINK | Answerability | Same prompt as What’sUp; applied per candidate image |
| CV-Bench | Answerability | Same prompt as What’sUp; applied per candidate image |
| MIMIC | Native (inherited) | Uses MIMIC’s own per-item distractors; no additional gate |
| Benchmark | Transformation | Details |
| CV-Bench | Marker drawing | Extract marker colors from question text Draw matching colored bounding boxes on distractors Match marker pixel mass to relevant image |
| BLINK | Duplicate filtering | Duplicate detection Remove duplicate images |
| All | Resize to match | Distractors resized to relevant image dimensions |
| All | Alternative variants | Randomly sampled distractor images (rnd) Black images (black) Used as control conditions |
| Benchmark | Qwen2.5-VL | InternVL3 | LLaVA-OV | ||||||||||||
| Vanilla accuracy | |||||||||||||||
| CLEVR (size) | 88.0 | 83.3 | 79.1 | 75.1 | 67.6 | 74.6 | 71.7 | 69.4 | 65.4 | 63.3 | 79.0 | 74.9 | 73.5 | 72.9 | 70.6 |
| CV-Bench | 79.0 | 74.4 | 72.4 | 71.1 | 67.3 | 83.1 | 80.4 | 78.0 | 76.7 | 72.5 | 78.3 | 73.2 | 71.8 | 68.2 | 64.1 |
| MIMIC | 73.6 | 72.3 | 71.8 | 71.1 | 68.9 | 76.3 | 73.5 | 70.6 | 69.2 | 65.7 | 70.9 | 69.5 | 66.3 | 66.4 | 65.0 |
| BLINK | 66.7 | 65.0 | 63.8 | 61.8 | 59.7 | 66.3 | 66.8 | 63.4 | 62.3 | 58.6 | 66.0 | 61.0 | 60.5 | 59.4 | 57.6 |
| Benchmark | Ground Truth | Two-Step | Hard Isolation | |||||||||
| Vanilla accuracy | ||||||||||||
| CLEVR (size) | 86.4 | 84.6 | 83.3 | 78.6 | 86.4 | 84.6 | 83.2 | 78.3 | 87.4 | 87.0 | 87.3 | 87.3 |
| CV-Bench | 70.2 | 68.9 | 69.4 | 66.1 | 70.2 | 68.7 | 69.2 | 65.4 | 78.0 | 77.4 | 77.1 | 75.3 |
| MIMIC | 68.4 | 67.0 | 66.8 | 65.7 | 68.3 | 66.9 | 66.9 | 65.8 | 74.5 | 74.3 | 73.8 | 73.4 |
| BLINK | 61.2 | 59.4 | 58.9 | 56.3 | 61.3 | 59.4 | 58.3 | 56.3 | 65.0 | 65.4 | 63.3 | 63.6 |
| Qwen2.5-VL | InternVL3 | LLaVA-OV | |||||||
| Benchmark | Content | Content | Content | ||||||
| CLEVR (size) | 20.0 | 1.7 | 91% | 11.6 | 4.3 | 63% | 7.3 | 1.4 | 119% |
| CV-Bench | 11.5 | 2.9 | 75% | 11.1 | 2.2 | 81% | 14.0 | 3.0 | 79% |
| MIMIC | 5.2 | 4.0 | 24% | 10.9 | 6.1 | 44% | 6.2 | 2.2 | 63% |
| BLINK | 6.1 | 1.1 | 82% | 7.9 | 1.5 | 81% | 7.3 | 2.0 | 72% |
| What’sUp | 23.0 | 2.6 | 89% | 26.3 | 6.8 | 74% | 31.7 | 8.8 | 72% |
| group | contents |
| system / header | attention sink (position ) + system prompt + user header |
| target image | visual tokens of the relevant image |
| distractor images | visual tokens of each distractor, in positional order |
| question | question stem + answer choices (content words only) |
| readout (answer) | the last position, where the answer is produced |
| separator | vision delimiters (between the image blocks), answer-format |
| Cond. | Source | Target | Definition of the target span |
| P | everything | every position that is not itself a distractor token | |
| MM | all preceding | image | visual tokens of each image, including the relevant image (MIMIC) |
| D | all text | every position after the final image: question stem, choices, answer-format instruction, generation scaffold, readout | |
| R | readout | the last prompt position, |
| benchmark | cond. | route | recov./broken | broken/right | ||
| CLEVR | P | 1331 | 1181 | IMG D everything | 0.54 [0.51, 0.56] | 0.07 [0.06, 0.09] |
| MM | 1331 | 1181 | all preceding IMG (MIMIC) | 0.04 [0.03, 0.05] | 0.86 [0.83, 0.87] | |
| D | 1331 | 1181 | IMG D all text | 0.44 [0.41, 0.47] | 0.08 [0.07, 0.10] | |
| R | 1331 | 1181 | IMG D readout | 0.02 [0.02, 0.03] | 0.04 [0.03, 0.05] | |
| CV-Bench | P | 410 | 1430 | IMG D everything | 0.58 [0.53, 0.63] | 0.03 [0.02, 0.04] |
| MM | 410 | 1430 | all preceding IMG (MIMIC) | 0.19 [0.15, 0.22] | 0.34 [0.31, 0.36] |
| Benchmark | P | D | both | D-only | P D / D | |
| CLEVR | 1331 | 54% | 44% | 39% | 4% | 90% |
| CV-Bench | 410 | 58% | 53% | 49% | 4% | 93% |
| BLINK | 89 | 55% | 49% | 47% | 2% | 95% |
| What’sUp (A+B) | 339 | 66% | 54% | 52% | 1% | 97% |
| MIMIC | 154 | 46% | 45% | 38% | 6% | 86% |
| Pooled | 2323 | 56% | 47% | 43% | 4% | 92% |
| cond. | route | early | middle | late | all |
| 0–8 | 9–17 | 18–27 | 0–27 | ||
| P | IMG D everything | 0.34 [0.32, 0.36] | 0.56 [0.54, 0.58] | 0.07 [0.06, 0.08] | 0.76 [0.74, 0.77] |
| MM | all preceding IMG (MIMIC) | 0.29 [0.27, 0.31] | 0.10 [0.09, 0.12] | 0.03 [0.02, 0.04] | 0.12 [0.11, 0.13] |
| D | IMG D all text | 0.19 [0.18, 0.21] | 0.47 [0.45, 0.49] | 0.07 [0.06, 0.08] | 0.57 [0.55, 0.59] |
| R | IMG D readout | 0.03 [0.03, 0.04] | 0.03 [0.03, 0.04] | 0.04 [0.03, 0.04] | 0.03 [0.03, 0.04] |
| Distractor images | Summary | ||||||
| Model | 0 | 1 | 2 | 3 | 5 | Mean | |
| CLEVR-size | |||||||
| Qwen2.5-VL-7B (base) | 70.7 | 60.1 | 54.9 | 48.3 | 36.4 | 49.9 | 0.0 |
| CE only | 82.5 | 79.9 | 78.5 | 77.9 | 75.4 | 77.9 | +28.0 |
| CE + KL | 82.8 | 79.4 | 77.8 | 75.4 | 71.9 | 76.1 | +26.2 |
| CE + KL | 79.8 | 77.9 | 75.9 | 74.2 | 72.0 | 75.0 | +25.1 |
| Prompt | Model | |||||
| Regular | base | |||||
| KL-clean | ||||||
| Ground Truth | base | — | ||||
| KL-clean | — | |||||
| — |
VLM Robustness
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.
See all 468 papers