The visual world is cluttered: robust vision-language reasoning requires ignoring irrelevant information. Yet vision-language models (VLMs) can suffer substantial accuracy losses when answering questions about a target image with distractors present. Existing mitigations focus on selecting relevant images or limiting information exchange between images. Through a systematic analysis of three open 7-8B VLMs on five perception benchmarks adapted to include distractor images, we show that distractors reduce accuracy even when models correctly identify the relevant image, suggesting that selection alone is not enough. To understand the mechanism behind this interference, we combine attention analysis with targeted knockouts in Qwen2.5-VL and identify middle-layer distractor-to-text connections as a pathway through which irrelevant visual information affects reasoning. We introduce answer consistency regularization, which encourages consistent answer distributions with and without distractors. Without directly constraining attention, this objective reduces middle-layer attention from question tokens to distractor images, consistent with the pathway our analysis implicates. With five distractors, our approach improves circular accuracy by an average of 16 percentage points over the Qwen2.5-VL baseline across the five benchmarks while preserving single-image performance.
Figures & tables
Figure 1: How does distractor information reach the answer? Cross-image knockout leaves distractor-to-text connections open. In this example ( Where is the wineglass? ), standard attention (top) and cross-image knockout (middle) yield incorrect answers, while distractor-to-text knockout (bottom) restores the correct answer. Knockouts target middle layers. Outlines identify distractor (red) and relevant (green) images. Arrows show allowed (solid red) and blocked (dotted gray) information flow from distractors; other connections are omitted.
Table 1: Composition of the distraction test sets. Each benchmark is evaluated at 0, 1, 2, 3 and 5 distractors; n is the number of questions, identical at every distraction level.
Figure 2: Distractor augmentation , illustrated on a What’sUp example. (1) A target image (green) provides the evidence for the question. (2) Candidate distractors are sampled from the same source, excluding the target image. (3) Candidates judged capable of independently supporting an answer are discarded. (4) Accepted distractors are shuffled with the target, keeping the question and answer choices unchanged.
Figure 3: Accuracy vs. number of distractor images. All three 7-8B VLMs show degradation, with benchmark-dependent severity.
Benchmark
Qwen2.5-VL
InternVL3
LLaVA-OV
k=1
k=2
k=3
k=5
k=1
k=2
k=3
k=5
k=1
k=2
k=3
k=5
CLEVR (size)
99.9
99.7
99.7
99.4
99.9
98.6
96.4
84.1
93.4
66.0
49.5
29.8
CV-Bench
96.3
94.5
92.9
88.0
94.1
91.0
84.5
61.6
92.0
57.0
46.2
25.9
MIMIC
99.7
99.5
99.2
99.1
99.9
99.1
96.3
80.5
99.0
76.3
67.1
34.3
BLINK
89.5
85.6
81.7
81.7
88.2
83.8
75.4
51.9
72.8
53.9
43.7
26.5
What’sUp
94.8
92.2
90.9
85.4
94.8
91.7
76.7
57.7
93.9
73.7
64.9
34.3
Table 2: Image-selection (phase-1) accuracy under the Two-Step protocol. k is the number of random distractors. Chance is 1/(k+1) .
Figure 4: Target guidance does not eliminate distraction in Qwen2.5-VL. Mean-rotation accuracy across distractor counts under Regular , Ground-Truth , and Two-Step prompting. We report Two-Step both end-to-end, including phase-1 selection errors, and conditioned on correct phase-1 selection. Dashed lines indicate clean accuracy.
Figure 5: Readout-token attention across decoder layers in Qwen2.5-VL-7B-Instruct. Curves show the fraction of attention allocated to image tokens (black), question tokens (blue), and remaining tokens (gray; including template tokens, instructions, and image separators). Solid lines denote clean inputs ( k=0 ), and dashed lines denote inputs with five distractors ( k=5 ). Results use n=150 examples per benchmark, except What’sUp ( n=300 ).
Figure 6: Share of the question tokens’ image attention on the relevant image , rel/(rel+∑distractors) , by decoder layer in Qwen2.5-VL-7B-Instruct, at k=1,2,3,5 distractors; dotted lines are uniform allocation 1/(k+1) . ( n=150 per benchmark; 300 for What’sUp)
cond.
route
recov./broken
range
macro
broken/right
P
IMG D → everything
0.56 [0.54, 0.58]
0.46–0.66
0.56
0.05 [0.04, 0.05]
MM
all preceding → IMG (MIMIC)
0.10 [0.09, 0.12]
0.04–0.53
0.20
0.57 [0.55, 0.58]
D
IMG D → all text
0.47 [0.45, 0.49]
0.44–0.54
0.49
0.05 [0.04, 0.06]
R
IMG D → readout
0.03 [0.03, 0.04]
0.02–0.07
0.04
0.02 [0.01, 0.02]
Table 3: Knocking out distractor → question attention repairs items that distraction broke. Qwen2.5-VL, middle layers 9–17, pooled over 5 benchmarks at k=5 distractors. Recovery is the fraction of broken items knockout fixes; damage is the fraction of preserved items it damages. range and macro are the min–max and unweighted mean over individual benchmarks. Intervals are 95% bootstrap CIs over items (10,000 resamples).
Per benchmark ( k>0 mean)
Pooled
Model
CLEVR
CV-B
MIMIC
WhatsUp
BLINK
k=0
k=5
mean
Δbase
Qwen2.5-VL-7B (base)
49.9
65.1
64.1
70.6
47.0
70.7
51.4
59.3
0.0
Ours
CE only
77.9
69.1
56.4
77.0
52.6
72.0
64.1
66.6
+7.3
CE + KL λ=0.1
76.1
68.5
60.6
77.5
53.0
73.7
63.8
67.1
+7.8
CE + KL λ=0.5
75.0
69.5
64.8
79.8
55.0
74.2
66.0
68.8
+9.5
Table 4: Circular accuracy under distraction. Qwen2.5-VL-7B with LoRA arms. Each benchmark column reports mean accuracy over k∈{1,2,3,5} distractor images. Pooled columns average across the five benchmarks and report accuracy at k=0 , at k=5 , and the mean over k>0 . Δbase is the pooled mean over k>0 minus the base model’s corresponding mean. Per- k results are shown in Table 17 ; boldface marks the best model in each column.
Figure 7: Question-token attention to images by decoder layer. Attention is averaged over heads and question-token query positions, then summed over each image’s visual-token span. Rows show distractor attention mass, and total image attention mass. Columns show the five benchmarks at k=5 ( n=150 per benchmark; n=300 for What’sUp). Curves compare the frozen base model, the CE-only control, and KL-clean model λ=1 .
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Pipeline Steps
CLEVR
1. Load questions filtered by category (size/color/count/material) 2. Build attribute index from scene graphs 3. Per-question vetting: exclude renders containing queried attribute 4. Sample k distractors, shuffle with relevant image
What’sUp
1. Parse caption to extract subject + spatial relation 2. Reframe as MCQ: “Where is/are the X?” 3. Domain-stratified pool selection (tabletop vs. natural) 4. Structural exclusion (own image, other subsets’ images) 5. VLM answerability gate (per-candidate check) 6. Sample k distractors, shuffle with relevant image
BLINK
1. Load COCO test2017 as candidate pool for distractors 2. Remove duplicate images 3. VLM answerability gate 4. Sample k distractors, shuffle with relevant image
CV-Bench
1. Load COCO test2017 as candidate pool for distractors 2. Route: COCO test2017 pool (if queried classes are COCO categories) or CV-Bench pool (if not) 3. COCO route: VLM answerability gate 4. Own pool route: VLM answerability gate 5. Draw colored marker boxes on distractors where the question names one (1,619 rows) 6. Sample k distractors, shuffle with relevant image
MIMIC
1. Load the item’s native MIMIC distractors 2. Subsample k of them (no external pool) 3. Shuffle with relevant image
Appendix
Table 5: Pipeline steps by benchmark. All benchmarks except MIMIC, which reuses its native distractors, share the core structure: pool construction, exclusion, vetting, and placement. Key differences are the vetting method and any special handling required.
Benchmark
Pool Source
Pool Size
Domain Type
CLEVR
CLEVR val renders
15,000
In-domain
What’sUp (controlled_a/b)
controlled_images + controlled_clevr
829
In-domain
What’sUp (coco_qa)
COCO val2017
5,000
In-domain
What’sUp (vg_qa)
Visual Genome
4,797
In-domain
BLINK
COCO test2017
40,641
Matched-domain
CV-Bench
COCO test2017 CV-Bench box-free
40,641 940
Hybrid
Appendix
Table 6: Distractor pool sources by benchmark. Pool size indicates the number of candidate images available for distractor selection. What’sUp uses domain-stratified pools to avoid cross-domain artifacts.
Benchmark
Gate Type
Description
CLEVR
Ground truth
Scene graphs provide exact attribute vetting; no VLM needed
What’sUp
Answerability
Per-candidate check: “Does this image contain any of the objects the question asks about, or could this image on its own support answering the question?”
BLINK
Answerability
Same prompt as What’sUp; applied per candidate image
CV-Bench
Answerability
Same prompt as What’sUp; applied per candidate image
MIMIC
Native (inherited)
Uses MIMIC’s own per-item distractors; no additional gate
Appendix
Table 7: VLM gate configurations. Each benchmark uses the strongest vetting guarantee its metadata supports. CLEVR uses exact scene-graph vetting, MIMIC reuses its native distractors, and other benchmarks use VLM-based answerability checks with identical prompts and model.
Benchmark
Transformation
Details
CV-Bench
Marker drawing
Extract marker colors from question text Draw matching colored bounding boxes on distractors Match marker pixel mass to relevant image
BLINK
Duplicate filtering
Duplicate detection Remove duplicate images
All
Resize to match
Distractors resized to relevant image dimensions
All
Alternative variants
Randomly sampled distractor images (rnd) Black images (black) Used as control conditions
Appendix
Table 8: Special transformations applied during distractor construction. Most benchmarks only require size normalization; CV-Bench additionally requires marker drawing to match visual cues in annotated questions.
Figure 9: Example items at the maximum distraction level ( k=5 ). Each row shows a question, its choices (correct answer in green), and the six images shown to the model: one relevant image (green outline) among five distractors from the same image distribution, in shuffled order.
Figure 10: The relevant image is placed uniformly across positions at every distractor level. For each benchmark, we plot the fraction of questions in which the question-relevant image occupies each slot, split by the number of distractors (1, 2, 3, 5). Distributions track the uniform reference (dashed line) across all positions and benchmarks, confirming that a model cannot exploit a fixed-position prior to locate the relevant image.
Figure 11: Token budget selection for multi-image evaluation. Accuracy at k=0 (single image) across token budgets; 1280 tokens matches native resolution.
Benchmark
Qwen2.5-VL
InternVL3
LLaVA-OV
k=0
k=1
k=2
k=3
k=5
k=0
k=1
k=2
k=3
k=5
k=0
k=1
k=2
k=3
k=5
Vanilla accuracy
CLEVR (size)
88.0
83.3
79.1
75.1
67.6
74.6
71.7
69.4
65.4
63.3
79.0
74.9
73.5
72.9
70.6
CV-Bench
79.0
74.4
72.4
71.1
67.3
83.1
80.4
78.0
76.7
72.5
78.3
73.2
71.8
68.2
64.1
MIMIC
73.6
72.3
71.8
71.1
68.9
76.3
73.5
70.6
69.2
65.7
70.9
69.5
66.3
66.4
65.0
BLINK
66.7
65.0
63.8
61.8
59.7
66.3
66.8
63.4
62.3
58.6
66.0
61.0
60.5
59.4
57.6
Appendix
Table 9: Baseline accuracy under random-image distraction. All three metrics are computed on the same runs: vanilla is the rotation-0 pass, circular requires every rotation to be correct, and mean-rotation is the per-item fraction of rotations passed, so circular ≤ mean-rotation holds cell by cell. Vanilla is one member of the rotation average rather than a bound on it, so it can sit marginally below mean-rotation. k is the number of distractor images; k=0 is the clean single-image baseline and is shared by both distractor kinds. Pooled is the mean of the five benchmark rows.
Benchmark
Ground Truth
Two-Step
Hard Isolation
k=1
k=2
k=3
k=5
k=1
k=2
k=3
k=5
k=1
k=2
k=3
k=5
Vanilla accuracy
CLEVR (size)
86.4
84.6
83.3
78.6
86.4
84.6
83.2
78.3
87.4
87.0
87.3
87.3
CV-Bench
70.2
68.9
69.4
66.1
70.2
68.7
69.2
65.4
78.0
77.4
77.1
75.3
MIMIC
68.4
67.0
66.8
65.7
68.3
66.9
66.9
65.8
74.5
74.3
73.8
73.4
BLINK
61.2
59.4
58.9
56.3
61.3
59.4
58.3
56.3
65.0
65.4
63.3
63.6
Appendix
Table 10: Qwen2.5-VL under the relevance prompts, random-image distraction. Ground Truth names the relevant image; Two-Step selects it then answers; Hard Isolation re-asks with only the selected image in context. k is the number of distractor images. Pooled is the mean of the five benchmark rows.
Figure 12: Prompting strategies under distraction (InternVL3-8B) . Mean-rotation accuracy as a function of the number of distractor images for four inference strategies (Regular, Ground-truth, Two-step, and Hard isolation) on each benchmark.
Figure 13: Prompting strategies under distraction (LLaVA-OV) . Mean-rotation accuracy as a function of the number of distractor images for four inference strategies (Regular, Ground-truth, Two-step, and Hard isolation) on each benchmark.
Figure 14: Correctly locating the relevant image does not undo distractor-induced degradation. (InternVL3-8B)
Figure 15: Correctly locating the relevant image does not undo distractor-induced degradation. (LLaVA-OneVision-7B)
Figure 16: Phase-1 image selection is positionally biased toward option B (second image) in LLaVA-OV. We zoom on the case where there are 3 distractors. For each benchmark, green bars give the distribution over the image slot selected in phase 1 (A = first image, …). Error bars are 95% Wilson score confidence intervals. Although the relevant image is near-uniformly distributed across positions, the model selects option B far above chance ( 25% ) in every benchmark.
Figure 17: Degradation under black-image distractors (sequence-length control). Same layout as Figure 3 , but each distractor is replaced by a solid black image of identical dimensions. The curves are flatter, demonstrating that sequence length alone does not explain the distractor penalty.
Qwen2.5-VL
InternVL3
LLaVA-OV
Benchmark
Δrnd
Δblack
Content
Δrnd
Δblack
Content
Δrnd
Δblack
Content
CLEVR (size)
20.0
1.7
91%
11.6
4.3
63%
7.3
− 1.4
119%
CV-Bench
11.5
2.9
75%
11.1
2.2
81%
14.0
3.0
79%
MIMIC
5.2
4.0
24%
10.9
6.1
44%
6.2
2.2
63%
BLINK
6.1
1.1
82%
7.9
1.5
81%
7.3
2.0
72%
What’sUp
23.0
2.6
89%
26.3
6.8
74%
31.7
8.8
72%
Appendix
Table 11: Content vs. sequence-length contribution to distractor penalty. Δrnd and Δblack are the drops in mean-rotation accuracy from k=0 to k=5 distractors. The content fraction measures how much of the penalty is attributable to distractor content rather than sequence length: (Δrnd−Δblack)/Δrnd .
group
contents
system / header
attention sink (position 0 ) + system prompt + user header
target image
visual tokens of the relevant image
distractor images
visual tokens of each distractor, in positional order
question
question stem + answer choices (content words only)
readout (answer)
the last position, where the answer is produced
separator
vision delimiters (between the image blocks), answer-format
Appendix
Table 12: Token groups .
Figure 18: Readout-token attention across decoder layers in Qwen2.5-VL-7B-Instruct. Curves show the fraction of attention allocated to all image tokens (black), question tokens (blue), separators and prompt-scaffold tokens (gold), system and header tokens (gray), and the readout position itself (purple). Solid lines denote clean inputs ( k=0 ), and dashed lines denote inputs with five distractor images ( k=5 ). Results use n=150 examples per benchmark, except What’sUp ( n=300 ).
Cond.
Source
Target
Definition of the target span
P
\textscimgD
everything
every position that is not itself a distractor token
MM
all preceding
image
visual tokens of each image, including the relevant image (MIMIC)
D
\textscimgD
all text
every position after the final image: question stem, choices, answer-format instruction, generation scaffold, readout
R
\textscimgD
readout
the last prompt position, L−1
Appendix
Table 13: Knockout conditions: source and target spans. Entries of the additive attention mask at (query = target, key = source) are set to −∞ during prefill, within the designated layer band only. Because the text block follows every image, R⊂D⊂P .
benchmark
cond.
nbroken
nright
route
recov./broken
broken/right
CLEVR
P
1331
1181
IMG D → everything
0.54 [0.51, 0.56]
0.07 [0.06, 0.09]
MM
1331
1181
all preceding → IMG (MIMIC)
0.04 [0.03, 0.05]
0.86 [0.83, 0.87]
D
1331
1181
IMG D → all text
0.44 [0.41, 0.47]
0.08 [0.07, 0.10]
R
1331
1181
IMG D → readout
0.02 [0.02, 0.03]
0.04 [0.03, 0.05]
CV-Bench
P
410
1430
IMG D → everything
0.58 [0.53, 0.63]
0.03 [0.02, 0.04]
MM
410
1430
all preceding → IMG (MIMIC)
0.19 [0.15, 0.22]
0.34 [0.31, 0.36]
Appendix
Table 14: Attention knockouts with 95% bootstrap intervals (10,000 resamples over items, band middle ). recov./broken is the fraction of broken items the knockout fixes. broken/right is the fraction of the already-correct population it damages.
Benchmark
n
P
D
both
D-only
P ∩ D / D
CLEVR
1331
54%
44%
39%
4%
90%
CV-Bench
410
58%
53%
49%
4%
93%
BLINK
89
55%
49%
47%
2%
95%
What’sUp (A+B)
339
66%
54%
52%
1%
97%
MIMIC
154
46%
45%
38%
6%
86%
Pooled
2323
56%
47%
43%
4%
92%
Appendix
Table 15: Item-level overlap between knockout P and knockout D in the middle layers ( 9 – 17 ). Percentages are of the broken population. P blocks distractor attention to every token group; D blocks it to text only. The final column is the fraction of D’s repairs that P also repairs. Computed from the per-item caches of the same sweep as Table 3 , so the P and D columns reproduce its recovery rates.
cond.
route
early
middle
late
all
0–8
9–17
18–27
0–27
P
IMG D → everything
0.34 [0.32, 0.36]
0.56 [0.54, 0.58]
0.07 [0.06, 0.08]
0.76 [0.74, 0.77]
MM
all preceding → IMG (MIMIC)
0.29 [0.27, 0.31]
0.10 [0.09, 0.12]
0.03 [0.02, 0.04]
0.12 [0.11, 0.13]
D
IMG D → all text
0.19 [0.18, 0.21]
0.47 [0.45, 0.49]
0.07 [0.06, 0.08]
0.57 [0.55, 0.59]
R
IMG D → readout
0.03 [0.03, 0.04]
0.03 [0.03, 0.04]
0.04 [0.03, 0.04]
0.03 [0.03, 0.04]
Appendix
Table 16: Knockout recovery by layer band. Qwen2.5-VL at k=5 distractors, pooled over five benchmarks at the item level. recovery (fraction of broken items repaired). Bands partition the 28 decoder layers; all masks every layer. The middle column reproduces Table 3 . Intervals are 95% bootstrap CIs over items (10,000 resamples).
Distractor images k
Summary
Model
0
1
2
3
5
Mean k>0
Δbase
CLEVR-size
Qwen2.5-VL-7B (base)
70.7
60.1
54.9
48.3
36.4
49.9
0.0
CE only
82.5
79.9
78.5
77.9
75.4
77.9
+28.0
CE + KL λ=0.1
82.8
79.4
77.8
75.4
71.9
76.1
+26.2
CE + KL λ=0.5
79.8
77.9
75.9
74.2
72.0
75.0
+25.1
Appendix
Table 17: Circular accuracy under distraction, per distractor level. Qwen2.5-VL-7B with LoRA arms. Circular accuracy: correct under every cyclic rotation of the answer options. k is the number of distractor images. One block per benchmark, plus a pooled block averaging the five. The last two columns report the mean over k>0 and Δbase , that mean minus the base model’s within the same block. Boldface marks the best value in each column. Summarised in Table 4 .
Prompt
Model
k=0
k=1
k=2
k=3
k=5
Regular
base
70.7
64.9
61.9
59.1
51.4
KL-clean
74.8
71.7
70.3
69.3
67.0
Δ
+4.1
+6.8
+8.4
+10.1
+15.7
Ground Truth
base
—
52.5
50.5
49.8
45.6
KL-clean
—
68.6
66.2
65.3
62.3
Δ
—
+16.1
+15.7
+15.4
+16.7
Appendix
Table 18: Circular accuracy (%) of the KL-clean model ( λ=1.0 ) against the untrained base model , under the regular, ground truth and two-step prompts. Accuracy is macro-averaged over five benchmark groups weighted equally (What’sUp A and B counted as one). Each cell covers n=9,052 items. Random distractors, max_pixels =1,003,520 . The relevance prompts have no k=0 point: selecting among zero distractors is not a task.
Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.
How does irrelevant information (i.e., distractors) affect test-time scaling in vision-language models (VLMs)? Prior work on text-only language models has shown that textual distractors can intensify inverse scaling, causing models to reason longer but less effective reasoning traces. In this work, we investigate whether similar phenomena arise in multimodal settings. We introduce Idis (Images with distractors), a visual question-answering dataset that systematically varies distractors along semantic and numerical dimensions. Our analyses reveal that visual distractors affect reasoning VLMs in a fundamentally different way from textual distractors: although inverse scaling still emerges, visual distractors reduce accuracy without increasing reasoning length. We further show that attribute counts extracted from reasoning traces provide key insights into how distractors interact with reasoning length and accuracy. As a sanity check, we propose a simple prompting strategy that mitigates distractor-driven predictions in reasoning vision-language models.
Jiyun Bae, Hyunjong Ok, Sangwoo Mo +1
Pohang University of Science and Technology (POSTECH)
Reasoning Vision-Language Models (VLMs) achieve strong performance on complex multimodal tasks, but reliable real-world application requires handling visual inputs that are messier than clean, curated benchmarks. Existing works mainly evaluate such reliability of VLMs through input corruptions, such as noise, blur and weather effects, which make visual evidence harder to perceive. This leaves a critical reliability failure mode underexplored: a model may perceive the evidence correctly, yet reason from plausible but irrelevant and distracting evidence and propagate this mistake to its final answer. To address this gap, we introduce Distract-Bench, a benchmark for evaluating VLM robustness to semantic visual distractions, defined as meaningful but task-irrelevant visual cues added to inputs while preserving the ground-truth answer. We comprehensively evaluate eight leading open-source and two closed-source VLMs across conventional vision corruptions and Distract-Bench. Our results show that Distract-Bench exposes a robustness failure distinct from vision corruptions: reasoning VLMs largely track their non-reasoning base models under perceptual degradation, but show consistently lower robustness to semantic distractions. Further analysis shows that these distractions often enter the reasoning process of VLMs, are treated as evidence, and lead to incorrect answers. Together, these findings reframe robustness evaluation for reasoning VLMs, shifting the focus from degraded perception to distractions for reliable real-world visual reasoning. Our data and code are available at https://github.com/Yizheng-Sun/Distract-Bench.
Yizheng Sun, Mochuan Zhan, Yanan Ma +10
University of Manchester · Marex · Imperial College London
Vision-Language Models (VLMs) have demonstrated strong capability in a wide range of tasks such as visual recognition, document parsing, and visual grounding. Nevertheless, recent work shows that while VLMs often manage to capture the correct image region corresponding to the question, they do not necessarily produce the correct answers. In this work, we demonstrate that this misalignment could be attributed to suboptimal information flow within VLMs, where text tokens distribute too much attention to irrelevant visual tokens, leading to incorrect answers. Based on the observation, we show that modulating the information flow during inference can improve the perception capability of VLMs. The idea is that text tokens should only be associated with important visual tokens during decoding, eliminating the interference of irrelevant regions. To achieve this, we propose a token dynamics-based method to determine the importance of visual tokens, where visual tokens that exhibit distinct activation patterns during different decoding stages are viewed as important. We apply our approach to representative open-source VLMs and evaluate on various datasets, including visual question answering, visual grounding and counting, optical character recognition, and object hallucination. The results show that our approach significantly improves the performance of baselines. Project page: https://cxliu0.github.io/AIF/.