Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to properly respond to visual evidence when input images violate those regularities. It is challenging to disambiguate these prior-induced errors from errors caused by the model missing the underlying visual capabilities. In this work, we use controlled visual in-context learning to encourage the model to focus on the visual evidence, and demonstrate that this alone is frequently sufficient to correct the errors -- suggesting that the models in fact do possess the relevant visual capabilities. Specifically, we show that across five vision-centric tasks, demonstrations yield large gains on prior-conflicting counterfactual examples (+27.2% for Qwen3-VL 235B and +15.1% for Qwen3-VL 32B) while leaving accuracy on prior-aligned examples nearly unchanged. These findings generalize across five VLM families with gains from +9.7% to +30.3%. We analyze visual representations and find that demonstrations indeed redirect attention toward grounding cues while reducing reliance on cues that support the learned prior. Finally, we find that in-context learning is limited when the underlying visual capability is weak or absent. Together, these results show that current VLMs possess visual grounding capabilities that standard evaluations may overlook, underscoring the need for controlled evaluation design in diagnosing and improving model capability.
Figures & tables
Figure 1: Visual ICL overcomes training prior dominance. (A) A VLM answers “2” for a three-legged peacock under zero-shot prompting; label-balanced demonstrations shift the answer to the visually grounded “3.” (B) Scaling ICL shots yields a +27.2 pp accuracy gain on counterfactual images while real images remain near ceiling. (C) Mechanistically, demonstrations redirect attention from the prior cue (“peacock”) to the grounding cue (“leg”).
Figure 2: Data design. (A) Our benchmark covers five vision-centric tasks, each contrasting a prior answer with the grounded visual answer. (B) Each task contains complex, simple, and no-prior settings to separate prior override from ordinary visual task learning. (C) We probe the training prior by comparing counterfactual examples, where the prior and visual answers conflict, with real examples, where they agree.
Figure 3: Qwen3-VL scales with in-context demonstrations. Accuracy vs. number of in-context demonstrations for Qwen3-VL model family on complex, simple, and no-prior settings across tasks.
Model
0-shot
16-shot
Δ16−0
Qwen3-VL 32B
69.9%
79.6%
+9.7%
Qwen2.5-VL 32B
49.1%
64.7%
+15.5%
Qwen3.5 27B
44.1%
74.5%
+30.3%
InternVL3.5 38B
52.4%
70.6%
+18.3%
Gemma-4 31B
75.2%
86.6%
+11.5%
Table 1: In-context scaling of ∼ 30B model families on the complex setting. Accuracies and deltas are reported with % and micro-averaged over query entries.
Figure 4: Scaling and diagnostic analysis of Qwen3-VL across model sizes on the complex setting. (a) Aggregate accuracy on the real and counterfactual subsets with varying numbers of in-context learning (ICL) shots, for the 4B, 8B, 32B, and 235B Qwen3-VL models. (b) Per-task 0-to-16-shot accuracy change ( Δ Acc) on the overall, counterfactual, and real subsets, broken down by task (counting, color, shape, direction, spatial relation) and model size. Positive values (blue) indicate tasks where ICL improves accuracy; negative values (red) indicate degradation.
Figure 5: Two complementary routes to visual grounding. (A) Masking visual tokens that encode prior cues weakens the learned-training prior pathway. (B) In-context demonstrations strengthen attention to image-grounded evidence. Both interventions shift the prediction from a prior-driven error toward the visually grounded answer.
Figure 6: Demonstrations reduce reliance on prior cues. (a) From 0-shot to 16-shot, VLMs shift answer-token attention away from prior cues and toward grounding cues. (b) On the complex setting with greedy decoding, prior-cue masking improves accuracy over both zero-shot and matched random-token masking for Qwen3-VL 32B and Gemma-4 31B on all five tasks. Table 13 reports all attention shifts and full masking accuracies.
Figure 7: ICL gains weaken for harder counting cases. The plot shows the accuracy change from 0-shot to 16-shot prompting.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Prompts used for querying models across tasks and settings.
Task
Setting
Source
Labels
Count
Real
CF
Edited
HV
Counting
Simple
Procedural (Pillow)
{1,…,8}
320
Counting
No-prior
Procedural (Pillow)
{1,…,8}
320
Counting
Complex
VLMs are Biased (regenerated) [ 2 ]
{1,…,5}
200
90
110
✓
✓
Color ID
Simple
Procedural (Pillow)
{G, O, Br, Bk, Pk, R}
240
Color ID
No-prior
Procedural (Pillow)
{G, O, Br, Bk, Pk, R}
240
Color ID
Complex
Generated + Edited; task design [ 1 ]
{G, O, Br, Bk, Pk, R}
240
120
120
✓
✓
Appendix
Table 2: Dataset statistics. (a) Original benchmark sources, labels, and counts, including real and counterfactual (CF) subsets. The real/CF split applies only to complex examples. Citations identify image sources or, for color and shape, task designs. Procedural images use Pillow or SVG; generated, regenerated, and edited images use Gemini. Regeneration reproduces a source image; HV denotes human validation. (b) Counting datasets used in Section 4.3 ; high-count shapes have 50 images per label.
Task
Input
Real
Counterfactual
Overall
Object Direction
Image
93.3
45.9
77.5
Caption Only
42.1
2.7
29.0
Spatial Relation
Image
83.2
41.2
69.2
Appendix
Table 3: Configuration-prior diagnostics. Zero-shot accuracy (%) for Qwen3-VL 32B on the original complex setting. Caption-only input replaces the direction image with object-identity text from its source caption. Overall accuracy weights the 160 real and 80 counterfactual examples in each task.
Figure 9: Dataset construction. For each task, images are rendered, generated, regenerated, or edited across the simple, no-prior, and complex settings, then paired with closed answer labels and validation controls for label-balanced ICL evaluation.
Task
Accepted
Rejected
Candidates
Acceptance (%)
Counting
110
192
302
36.4
Color ID
120
15
135
88.9
Shape ID
120
13
133
90.2
Object Direction
80
78
158
50.6
Spatial Relation
80
128
208
38.5
All Tasks
510
426
936
54.5
Appendix
Table 4: Human validation statistics for the original counterfactual portion of the complex setting. Acceptance requires both annotators to select the intended label independently and confirm that the image meets the quality criteria.
Task
Stage
Prompt
Counting
Edit
{new_total_legs} leg(s) total. Edit this image so the {animal_name} has exactly {new_total_legs} leg(s). {arrangement hint}. Every leg must match the size and texture of the original legs. The result must be photo-realistic with all {new_total_legs} leg(s) clearly visible and distinct. Preserve the same background, facial features, lighting, and camera angle.
Direction
Edit
Edit this image so the {object_name} faces {target_direction}ward. {front_part} should point toward the {top|bottom} of the frame, with the {object_name} {pose_desc}. The {object_name} must remain clearly visible and recognizable. The result must be photo-realistic. Preserve the same background, lighting, object size, and camera angle.
Spatial
Caption
You generate counterfactual captions by changing ONLY the spatial relation between the same two objects. Caption: {original_caption}. Choose a preposition from {allowed_preps} that makes the resulting caption feel the most unnatural. Produce {num_variants} candidate captions.
Spatial
Edit
Edit this image. From the viewer’s perspective, the {obj1} is {relation} the {obj2}. Both objects must remain fully visible. Maintain photorealistic appearance. Preserve the same background, lighting, and object appearance.
Color ID
Generate
Generate a photorealistic image of a {animal_name}. The {animal_name} should be the central object in the image. Include realistic natural scenery around the animal. The animal must be clearly visible.
Color ID
Edit
Edit this image: change the color of the {animal_name} from {original_color} to {target_color}. The {animal_name} must remain clearly recognizable. Preserve the same background, lighting, and camera angle.
Appendix
Table 5: Generation and editing prompts used for image construction in the complex setting.
Mode
Model family
Max tokens
Temp.
Top- p
Top- k
Rep. penalty
Presence penalty
Standard non-thinking
Qwen3-VL, Qwen2.5-VL, Qwen3.5, InternVL3.5
512
0.7
0.8
20
1.0
vLLM: 1.5 ; HF: omitted
Standard non-thinking
Gemma-4
512
1.0
0.95
64
1.0
vLLM: 0.0 ; HF: omitted
Thinking mode
Qwen3-VL, Qwen2.5-VL, Qwen3.5, InternVL3.5
1024
1.0
0.95
20
N/A
N/A
Thinking mode
Gemma-4
1024
1.0
0.95
64
N/A
N/A
Appendix
Table 6: Sampling parameters for VLM benchmark evaluation. Values follow each model card’s recommended decoding settings and are used for the vLLM behavioral runs. Mechanistic experiments (Hugging Face Transformers) instead use greedy decoding and therefore do not use the temperature, top- p , top- k , repetition, or presence parameters listed here.
Figure 10: ICL scaling across model families and settings. Rows: model families (aggregated, then Qwen3-VL, Qwen2.5-VL, Qwen3.5, InternVL3.5, Gemma-4). Columns: settings. Qwen3-VL scales consistently; Qwen2.5-VL less so. Qwen3.5 and InternVL3.5 degrade beyond ∼ 8–16 shots. Gemma-4 achieves the strongest and most stable scaling.
InternVL3.5
Qwen2.5-VL
Qwen3-VL
Qwen3.5
Gemma-4
Setting
Shots
4B
38B
3B
32B
4B
32B
4B
27B
E4B
31B
Counting
No-prior
0
43.1
28.8
35.6
41.9
32.5
43.8
58.8
49.7
35.0
23.8
16
49.8 ( ±29.4 )
76.5 ( ±26.3 )
19.9 ( ±25.9 )
70.8 ( ±20.0 )
35.6 ( ±18.1 )
71.9 ( ±18.6 )
16.8 ( ±30.0 )
80.4 ( ±22.6 )
28.6 ( ±22.5 )
72.4 ( ±25.8 )
Δ16−0
+6.6
+47.8
-15.7
+28.9
+3.1
+28.2
-41.9
+30.8
-6.4
+48.5
Simple
0
92.5
63.8
87.5
84.4
88.1
93.8
95.0
95.6
63.8
54.4
Appendix
Table 7: Cross-family accuracies by task and setting. Accuracies are percentages; standard deviations and Δ16−0 are in percentage points. For 16-shot rows, parentheses report standard deviations over sampled demonstration contexts. Best 0- and 16-shot entries per row are bold; for Δ16−0 the largest per-row improvement is bold. A dash indicates that a valid run was unavailable, and Δ16−0 is omitted unless both endpoints are available.
Subset
Shots
Both correct (%)
Gemma-4 only (%)
Qwen3-VL only (%)
Both incorrect (%)
Agreement (%)
Δ Agreement (pts)
Real
0
82.4
5.9
4.5
7.2
89.6
—
16
80.9
6.6
4.5
8.1
88.9
− 0.7
Counterfactual
0
51.7
15.4
9.2
23.7
75.4
—
16
72.6
13.5
3.4
10.4
83.1
+7.7
Appendix
Table 8: Paired correctness of Gemma-4 31B and Qwen3-VL 32B. Both models answer the same complex queries, split into real and counterfactual examples, at 0 and 16 shots. Agreement is the share of cases where both models are correct or both are incorrect. Percentages use the same aggregation as Figure 4 (a), and entries are rounded independently.
Table 9: Expanded complex-setting dataset and partial results. Exact expanded counterfactual and total counts are unavailable. The partial evaluation covers Qwen3-VL 32B at two shot counts; 16-shot entries are mean accuracy ± standard deviation across ten fixed demonstration contexts, while zero-shot has no context variance. Accuracies are percentages; standard deviations and Δ are in percentage points (pp).
Table 10: Masking and length-matched control ablations. Panels (a), (b), and (c) report accuracy (%) for Qwen3-VL 32B on the complex setting with greedy decoding. Ground mask removes grounding-cue tokens from the query image; Blank replaces each demonstration image with a same-size blank image. Unrelated uses demonstrations from another split of the same task, while Vision only removes question text from both the demonstrations and query. Panel (d) reports simple -setting endpoints from the shuffled-label sweep in Figure 11 ; Δ is the change from zero-shot accuracy in percentage points. Compute limits restrict panel (c) to counting.
Figure 11: Qwen3-VL unlearns under label shuffling. In the simple setting, shuffled demonstration labels reduce performance below the zero-shot baseline, with larger drops for larger models.
Demonstration Task
Images
Shots
Accuracy (%)
None
—
0
32.1
Counting
Complex (standard)
16
54.4
Counting
2D shapes
16
43.5
Color ID
Animals
16
34.6
Color ID
2D shapes
16
20.2
Color ID
Mixed splits
16
27.1
Appendix
Table 11: Cross-task demonstrations. Accuracy on complex counting queries for Qwen3-VL 32B. Mixed color demonstrations are sampled uniformly across splits.
Counting
Original
Explicit Language Prior
0-shot
33.0
28.9
16-shot
53.7
51.0
Δ
+20.7
+22.1
Appendix
Table 12: Counting under an explicit language prior. Accuracy (%) for standard counting prompts and prompts prepended with an explicit prior statement. The explicit prior lowers zero-shot accuracy, but the 0-to-16-shot gain remains comparable to the original prompt.
Figure 12: Reasoning (thinking) vs. non-reasoning ICL accuracy. Overall accuracy for Qwen3-VL-4B (Instruct vs. Reasoning), Qwen3-VL-32B (Instruct vs. Reasoning), Gemma-4-E4B-it, and Gemma-4-31B-it. Solid lines denote non-thinking mode; dotted lines denote thinking mode. Enabling thinking does not consistently improve ICL scaling.
Figure 13: Answer-label distribution shifts under ICL in the complex setting. For Qwen3-VL-32B-Instruct and Gemma4-31B, we plot the probability assigned to the correct label at 0-shot and 16-shot for color identification (top row) and counting (bottom row), separated into counterfactual and real entries. Probability mass on counterfactual answers increases substantially with demonstrations, while probabilities on real entries remain largely intact.
Table 13: Mechanistic results across Qwen3-VL and Gemma-4. (a) Changes in answer-token attention to prior cues, grounding cues, and other query-image tokens. Attention is normalized over query-image visual tokens and averaged per token within each group. (b) Zero-shot accuracy, zero-shot masking interventions, and 16-shot accuracy on all five tasks. The random mask matches the prior-cue mask’s token budget. Object direction and spatial relation use object identity as a proxy prior cue.
Figure 14: Prior-cue masking examples. Two queries from the complex setting with their identified prior-cue tokens (red overlay on the query image) and the candidate-normalized answer distributions at 0-shot, after prior-cue token ablation, and at 16-shot. Left: a one-legged swan, where the 0-shot model places nearly all mass on the prior-default answer “2”; ablating the prior-cue tokens shifts mass onto the ground-truth “1”; and 16-shot demonstrations produce a similar shift. Right: a pink raven, where 0-shot mass concentrates on the training prior default “black”, and both prior-cue ablation and 16-shot ICL move probability mass onto the ground-truth “pink”. Prior-cue ablation reproduces the qualitative effect of in-context demonstrations.
Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by learned language and world priors. Counting provides a precise testbed: when visual evidence conflicts with canonical object knowledge, a model must rely on the image rather than a prototypical count. We introduce CounterCount, a diagnostic framework for counterfactual counting in VLMs, consisting of paired factual and counterfactual images with edited count-relevant attributes, verified answers, and localized evidence annotations. Evaluating recent VLMs, we find strong performance on factual images but consistent degradation under counterfactual attribute changes, indicating reliance on object-level priors even when contradictory visual evidence is present. Using localized annotations, we show that these failures are not solely due to missing or ambiguous visual evidence, but to models underweighting attention to count-relevant visual tokens. We introduce a unified inference-time attention modulation strategy that reweights selected visual tokens, improving counterfactual counting accuracy by up to 8% across multiple VLMs. Overall, CounterCount exposes prior-driven counting failures and provides diagnostic insights for designing future VLMs.
Reem Alzahrani, Hassan Alshanqiti, Bushra Bin Hemid +3
King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · University of Edinburgh · KAUST
Visual inputs are often assumed to improve language understanding in multimodal models. We examine this assumption by asking whether vision-language models (VLMs) can distinguish useful visual evidence from incidental image context in lexical judgments. We use human concreteness and imagery ratings because they span words with varying expected visual relevance, from abstract and low-imagery words to concrete and high-imagery words. We find that real-image contexts do not yield consistent gains and often hurt alignment with human ratings, most sharply when visual evidence is least relevant. Through probing and canonical correlation analysis, complemented by an attribution case study, we find that real-image contexts are associated with representational shifts and greater sensitivity to spurious visual cues, coinciding with weaker recoverability of the targeted lexical properties. We further show that instructing models to focus solely on textual content at inference time can reduce this degradation, with the clearest gains on these vulnerable subsets. Our findings suggest that current instruction-tuned VLMs need better calibration of when visual context should inform lexical judgments.
Visual in-context learning (ICL) enables vision-language models (VLMs) to adapt at test time via demonstration examples, but its effectiveness depends critically on the examples selected. Recent counterfactual selection methods improve ICL by constructing demonstrations that expose how individual attribute changes affect the answer. However, they require multi-stage pipelines involving attribute extraction, caption engineering, and composed image matching. We propose a simpler alternative grounded in a key observation: a model's own prediction errors are natural counterfactual demonstrations that directly encode its failure modes. Our method, SMILE (Self-Mined Visual In-Context Learning from Errors), builds a model-specific pool of hard negatives offline by recording the target VLM's own mistakes. At test time, the prediction serves as a direction signal that surfaces, in the semantic embedding space, the past mistakes most likely to recur for the current query. This prediction-conditioned identification provides a unified mechanism for both closed-set classification and open-ended visual question answering without task-specific design. Across four benchmarks and four VLMs, SMILE delivers an average task-score gain of +3.09 points over the SOTA baseline at 5x lower latency. Project page: https://tree-shu-zhao.github.io/SMILE/