Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts

Nimrod Shabtay; Asaf Yehudai; Avihu Dekel; Eli Schwartz; Raja Giryes

cs.CVOct 22, 2026

Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts

Authors: Nimrod Shabtay, Asaf Yehudai, Avihu Dekel, Eli Schwartz, Raja Giryes

Organizations: IBM Research · Tel-Aviv University

Abstract

Vision-language models (VLMs) struggle with in context learning compared to their textual counterparts. Our preliminary study shows that VLMs rely on textual shortcuts and underutilize the visual demonstrations. We therefore hypothesize that VLMs are insufficiently trained to perform visual in-context learning (vICL). To this end, we construct MIME, a task-agnostic vICL training set without textual shortcuts. We ensure the instruction does not state the rule linking the demonstration images to their answers, so the model must infer it from the visual demonstrations. Fine-tuning on MIME improves vICL accuracy by 8.1 to 18.0 points across three VLMs, preserves single-image performance, and improves multi-image and text only in-context learning. In contrast, stating the rule in the training data sharply reduces this gain. We further propose DTAD (Demonstration Text Attention Dropout), which randomly withholds demonstration labels during training so that the VLM must infer the rule from the demonstration images. While achieving comparable vICL accuracy to fine-tuning on MIME, DTAD yields more consistent attention to the demonstration images. On TrueMICL, Qwen3-VL-8B trained on MIME matches frontier models such as Claude Opus 5, and surpasses them with DTAD. Finally, adding MIME to the instruction-tuning mix of LLaVA 1.5 13B more than doubles its vICL accuracy without hurting single-image abilities, whereas a same-size multi-image set improves it only slightly. These results suggest that vICL data should be a standard component of VLM training.

Figures & tables

Figure 1: We fine-tune on MIME, synthetic in-context instances whose task is a hidden rule that must be inferred from the demonstrations. Despite being synthetic, MIME improves results on unseen real-world benchmarks. DTAD, which masks demonstration answers, produces more stable attention patterns that focus on the demonstration images.

Model	Clock	Op.Ind	Palin	Shapes	Classif	T23	Op.Ind	Interl	TextOCR	Open MI	Match MI	Avg	$\Delta$
	TrueMICL					SB-2	VL-ICL-Bench
Qwen3-VL-8B	12.8	76.7	63.9	95.8	45.7	35.8	78.5	59.4	59.2	72.1	85.8	62.3 ±0.9	–
+ DARA ( Chen et al., 2025c )	12.6	76.7	64.2	95.8	45.3	35.8	78.3	59.8	59.5	71.8	85.8	62.3	+0.0
+ MGI ( Wang and Li, 2026 )	12.9	76.6	64.3	96.0	45.4	35.8	78.3	59.4	59.6	72.2	85.2	62.3	+0.0
+ MIME	62.8	88.9	68.8	97.3	63.4	50.0	91.1	86.9	58.7	98.2	85.3	77.4 ±1.2	+15.1
+ MIME + DTAD	70.1	88.1	79.3	99.2	53.9	45.8	90.0	87.8	60.7	94.9	81.5	77.4 ±1.5	+15.1

Table 1: Visual ICL results (generic instruction protocol). Accuracy (%) of each base model before and after fine-tuning on MIME, averaged over 1/2/4 shots and three seeds.

Model	Size	Clock	Op.Ind	Palin	Shapes	Classif	Overall
Qwen3-VL-235B-A22B	235B	31.0	99.3	86.0	97.8	64.2	75.7
InternVL3.5-241B-A28B	241B	15.3	99.7	79.0	83.9	48.4	65.3
Claude Opus 4.6	-	43.3	90.2	76.7	99.1	39.6	69.8
Claude Opus 5	-	36.1	88.6	95.1	99.6	61.0	76.1
Qwen3-VL-8B	8B	12.8	76.7	63.9	95.8	45.7	59.0
+ MIME	8B	62.8	88.9	68.8	97.3	63.4	76.2

Table 2: Comparison to frontier and Large models on TrueMICL. Per-task and overall accuracy on the five TrueMICL tasks. Fine-tuned on MIME, Qwen3-VL-8B outperforms models roughly 30

\times

larger and Claude Opus 4.6, and matches Claude Opus 5; with DTAD, it outperforms all of them.

Training mix	Avg	POPE	MME-P	MME-C
	TrueMICL	Regression check
LLaVA-1.5-13B (official)	18.9	84.2	1561	118
+ Multi-Image data	21.5	54.1	1509	116
+ MIME	39.7	85.2	1575	127

Table 3: Full retraining of LLaVA-1.5-13B. MIME and a same-size multi-image set (Mantis) are each added to the original instruction-tuning mix. TrueMICL accuracy is averaged over 1/2/4 shots.

	Regression			Text-only ICL
Model	SI	MI	MARVEL	MMLU	GSM8K	GPQA
Qwen3-VL-8B	76.1	54.9	27.0	74.6	89.8	36.4
+ MIME	76.4	57.5	29.1	75.6	91.4	37.1
+ MIME + DTAD	76.9	58.1	31.6	75.8	92.7	37.1

Table 4: Beyond visual ICL. SI (single-image): average of MMStar, RealWorldQA, POPE, and InfoVQA. MI (multi-image): average of Mantis and LLaVA-Interleave. MARVEL is evaluated as multi-turn visual ICL (chance 25%). Text-only ICL uses multi-turn demonstrations.

2 shots
Model	Correct	Answers	Images	Foreign	No
Model	Correct	shuffled	shuffled	rule	demos
Qwen3-VL-8B	57.5	$-$ 19.3	$-$ 19.8	$-$ 43.5	$-$ 25.3
+ MIME	74.5	$-$ 35.7	$-$ 35.1	$-$ 63.8	$-$ 48.7
+ DTAD (AM)	74.3	$-$ 32.5	$-$ 33.4	$-$ 70.7	$-$ 42.3
+ DTAD (TR)	77.8	$-$ 34.6	$-$ 33.6	$-$ 71.4	$-$ 47.6

Table 5: Trained models depend on the demonstrations. Accuracy with correct demonstrations, and change in points under each corruption, on the five TrueMICL tasks (generic instruction protocol, three seeds).

Training data	TrueMICL	Answers shuffled
Untrained	57.5 / 66.6	$-$ 19.3 / $-$ 30.9
Rule stated	69.6 / 78.9	$-$ 26.3 / $-$ 42.8
Rule hidden (ours)	74.5 / 88.6	$-$ 35.7 / $-$ 52.3

Table 6: Stating the rule substantially reduces the transfer gain. Accuracy (2 / 4 shots) on the TrueMICL tasks of Table 5 , and change under shuffled demonstration answers.

Figure 2: Mean visual share per layer for 1-shot (left), 2-shot (center), and 4-shot (right); lines show the mean over the 5 TrueMICL tasks and shaded bands show

\pm 1

SD across tasks. DTAD raises the visual share in the late layers above the base model at every shot count, with a narrow spread across tasks. Fine-tuning on the same data increases it mainly in the middle layers; in the late layers it varies widely across tasks. Training with the rule stated in the prompt lowers the visual share below the base model, indicating that the model relies mostly on the textual pathway.

Model	1 shot	2 shots	4 shots
Qwen3-VL-8B	19.9 $\pm$ 2.0	20.6 $\pm$ 1.7	24.6 $\pm$ 1.8
+ MIME (rule stated)	20.2 $\pm$ 3.2	22.3 $\pm$ 2.5	26.9 $\pm$ 2.9
+ MIME	22.1 $\pm$ 3.4	26.1 $\pm$ 4.8	29.7 $\pm$ 3.5
+ MIME + DTAD (TR)	22.9 $\pm$ 2.6	26.3 $\pm$ 2.8	31.3 $\pm$ 3.3

Table 7: Attention redistribution under demonstration-text blocking. Absorption: share (%) of the attention freed by blocking demonstration text that moves to demonstration images, all layers, mean

\pm

SD over 5 TrueMICL tasks.

	Qwen3-VL-8B	+ MIME	+ DTAD (TR)
Acc@4	66.6	88.6	90.0
Acc@16	84.6	96.3	96.9
Error reduction	53.9%	67.5%	69.0%

Table 8: Error reduction from 4 to 16 shots (TrueMICL). Error reduction is the fraction of the 4-shot error removed at 16 shots. DTAD and fine-tuning keep improving with more demonstrations. The higher ERR demonstrates they improve more relative to their stronger starting point.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Shots	Clock	Operator induction	Mean
Pure text-ICL	1	100.0	99.0	99.5
	2	100.0	100.0	100.0
	4	100.0	100.0	100.0
	Avg	100.0	99.7	99.8
Transcribe & answer	1	100.0	99.0	99.5
	2	100.0	100.0	100.0

Appendix

Table 9: Preliminary study: what limits vICL? Accuracy (%) of Qwen3-VL-8B on two TrueMICL tasks (Clock, Operator induction). Pure text-ICL : the task is posed as text-only ICL using the task metadata. Transcribe & answer : the model first transcribes the images, then answers. Task instruction (baseline) : standard vICL with the official TrueMICL prompt. Generic instruction : vICL with an instruction that gives no hint about the task. Avg : mean over shots; Mean: mean over the two tasks. Near-perfect text-ICL and transcribe & answer scores show that perception and textual ICL are not the bottleneck, while the drop under a generic instruction indicates reliance on textual cues.

Source	Episodes	Shots	Mean shots
PuzzleVQA polygon sides	5000	$\{2,3,4\}$	2.94
Bongard-LOGO shape concept	5000	$\{2,4\}$	3.00
RE-ARC transform induction	5000	$\{3,4,5\}$	3.97
PuzzleVQA triangle arithmetic	5000	$\{2,3,4\}$	3.00
CLEVR	3578	$\{1,2,4\}$	2.64
Total	23578	$\{1,\dots,5\}$	3.14

Appendix

Table 13: Composition of the training set by source, showing the number of episodes, the range of shots used, and the mean number of shots per episode.

Model	Clock	Op.Ind	Palin	Shapes	Classif	Avg
Qwen3-VL-8B	12.8	76.7	63.9	95.8	45.7	59.0
Single-image control	11.0	88.7	80.3	94.3	33.3	61.5
+ MIME	62.8	88.9	68.8	97.3	63.4	76.2

Appendix

Table 14: Single-image control. TrueMICL per-task accuracy (%, Qwen3-VL-8B, averaged over 1/2/4 shots). The control trains on the same query images with each rule stated in text and no demonstrations. It improves over the untrained model but falls far short of training on our vICL data. The gap is largest on Clock, whose rule exists only in the visual demonstrations.

	Polygon	Bongard	RE-ARC	Triangle	CLEVR
Chance	7.1	50.0	14.3	4.5	4.5
Qwen3-8B text-only	23.0	27.5	4.5	11.5	4.5
Query-demo violations (count)	0	0	0	0	0

Appendix

Table 15: Per-source audit (%). Text-only predictability and query-demonstration instance violations.

Removed source	Naïve FT	DTAD (AM)	DTAD (TR)
Polygon	$-$ 2.6	$-$ 3.0	$-$ 9.7
Bongard-LOGO	$+$ 2.6	$-$ 3.9	$-$ 6.5
RE-ARC	$-$ 2.4	$-$ 4.1	$-$ 4.6
Triangle	$-$ 10.2	$-$ 8.3	$-$ 7.4
CLEVR	$-$ 1.0	$-$ 16.7	$-$ 6.7

Appendix

Table 16: Leave-one-out. Change in TrueMICL average accuracy when one source is removed from the full mix.

Multimodal ICL

Multimodal in-context learning (ICL) studies how models use examples and other context to solve tasks involving images, text, audio, and video.

See all 49 papers

Oct 9, 2026

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

Vision-language models (VLMs) learn strong statistical regularities during training, which can make them fail to properly respond to visual evidence when input images violate those regularities. It is challenging to disambiguate these prior-induced errors from errors caused by the model missing the underlying visual capabilities. In this work, we use controlled visual in-context learning to encourage the model to focus on the visual evidence, and demonstrate that this alone is frequently sufficient to correct the errors -- suggesting that the models in fact do possess the relevant visual capabilities. Specifically, we show that across five vision-centric tasks, demonstrations yield large gains on prior-conflicting counterfactual examples (

+27.2\%

for Qwen3-VL 235B and

+15.1\%

for Qwen3-VL 32B) while leaving accuracy on prior-aligned examples nearly unchanged. These findings generalize across five VLM families with gains from

+9.7\%

+30.3\%

. We analyze visual representations and find that demonstrations indeed redirect attention toward grounding cues while reducing reliance on cues that support the learned prior. Finally, we find that in-context learning is limited when the underlying visual capability is weak or absent. Together, these results show that current VLMs possess visual grounding capabilities that standard evaluations may overlook, underscoring the need for controlled evaluation design in diagnosing and improving model capability.

Kun Wang, Xindi Wu, Sanghyuk Chun +2

Princeton University

Visual Evidence Sensitivity in VLMs In-Context Learning Vision-Language Grounding Counterfactual Evaluation of VLMs

Ask PDF

Aug 3, 2026

In-Context Collapse in Vision-Language Models and How to Mitigate it?

Many-shot in-context learning (ICL) lets vision-language models (VLMs) adapt from image--label demonstrations without weight updates, and is widely assumed to improve as more demonstrations are supplied. We show the opposite: as demonstrations accumulate, a subset of VLMs undergo an in-context collapse, a sharp, sometimes catastrophic accuracy drop spanning synthetic classification, natural-image classification, and VQA benchmarks, in some models falling below chance while outputs remain well-formed. Across an open VLM panel (

0.5

B--

11

B) and a frontier model (Claude Sonnet 4.5), the collapse is graded. Two capabilities turn out to be dissociable: robustness to accumulating demonstrations and the ability to learn a novel rule in context, their combinations yield three reproducible regimes. A parameter-matched lesion-and-rescue causally localizes the collapse to the vision-language integration pathway: an adapter on the connector and early/mid layers restores genuine learning (remap accuracy

0.39!\rightarrow!0.91

at 16 shots), while an equal-capacity adapter on the late readout does not. We propose CircA, whose core is a one-time integration vaccine: trained once on one synthetic task, it transfers collapse-resistance to unseen task families (chance

\rightarrow$$0.71

0.60

on CIFAR/Fashion). The layers best for in-context integration are not the layers best for weight-based consolidation, the late readout achieves higher accuracy and less forgetting at fewer parameters. The collapse is an integration failure at the vision--language interface, correctable by a lightweight, transferable intervention.

Mohammad Rostami

Amazon Generative AI Innovation Center∗, USA.

VLM Robustness Vision-Language Models Multimodal ICL VLM Adaptation

Ask PDF

Oct 12, 2026

The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errors

Visual in-context learning (ICL) enables vision-language models (VLMs) to adapt at test time via demonstration examples, but its effectiveness depends critically on the examples selected. Recent counterfactual selection methods improve ICL by constructing demonstrations that expose how individual attribute changes affect the answer. However, they require multi-stage pipelines involving attribute extraction, caption engineering, and composed image matching. We propose a simpler alternative grounded in a key observation: a model's own prediction errors are natural counterfactual demonstrations that directly encode its failure modes. Our method, SMILE (Self-Mined Visual In-Context Learning from Errors), builds a model-specific pool of hard negatives offline by recording the target VLM's own mistakes. At test time, the prediction serves as a direction signal that surfaces, in the semantic embedding space, the past mistakes most likely to recur for the current query. This prediction-conditioned identification provides a unified mechanism for both closed-set classification and open-ended visual question answering without task-specific design. Across four benchmarks and four VLMs, SMILE delivers an average task-score gain of +3.09 points over the SOTA baseline at 5x lower latency. Project page: https://tree-shu-zhao.github.io/SMILE/

Shu Zhao, Tan Yu

NVIDIA

In-Context Learning Test-Time Adaptation of VLMs Hard Negative Mining

Ask PDF

Abstract

Multimodal ICL

Explore similar work

In-Context Learning Can Help Vision-Language Models Overcome Training Priors

In-Context Collapse in Vision-Language Models and How to Mitigate it?

The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errors