Vision-language models (VLMs) struggle with in context learning compared to their textual counterparts. Our preliminary study shows that VLMs rely on textual shortcuts and underutilize the visual demonstrations. We therefore hypothesize that VLMs are insufficiently trained to perform visual in-context learning (vICL). To this end, we construct MIME, a task-agnostic vICL training set without textual shortcuts. We ensure the instruction does not state the rule linking the demonstration images to their answers, so the model must infer it from the visual demonstrations. Fine-tuning on MIME improves vICL accuracy by 8.1 to 18.0 points across three VLMs, preserves single-image performance, and improves multi-image and text only in-context learning. In contrast, stating the rule in the training data sharply reduces this gain. We further propose DTAD (Demonstration Text Attention Dropout), which randomly withholds demonstration labels during training so that the VLM must infer the rule from the demonstration images. While achieving comparable vICL accuracy to fine-tuning on MIME, DTAD yields more consistent attention to the demonstration images. On TrueMICL, Qwen3-VL-8B trained on MIME matches frontier models such as Claude Opus 5, and surpasses them with DTAD. Finally, adding MIME to the instruction-tuning mix of LLaVA 1.5 13B more than doubles its vICL accuracy without hurting single-image abilities, whereas a same-size multi-image set improves it only slightly. These results suggest that vICL data should be a standard component of VLM training.
Figures & tables
Figure 1: We fine-tune on MIME, synthetic in-context instances whose task is a hidden rule that must be inferred from the demonstrations. Despite being synthetic, MIME improves results on unseen real-world benchmarks. DTAD, which masks demonstration answers, produces more stable attention patterns that focus on the demonstration images.
TrueMICL
SB-2
VL-ICL-Bench
Model
Clock
Op.Ind
Palin
Shapes
Classif
T23
Op.Ind
Interl
TextOCR
Open MI
Match MI
Avg
Δ
Qwen3-VL-8B
12.8
76.7
63.9
95.8
45.7
35.8
78.5
59.4
59.2
72.1
85.8
62.3 ±0.9
–
+ DARA ( Chen et al., 2025c )
12.6
76.7
64.2
95.8
45.3
35.8
78.3
59.8
59.5
71.8
85.8
62.3
+0.0
+ MGI ( Wang and Li, 2026 )
12.9
76.6
64.3
96.0
45.4
35.8
78.3
59.4
59.6
72.2
85.2
62.3
+0.0
+ MIME
62.8
88.9
68.8
97.3
63.4
50.0
91.1
86.9
58.7
98.2
85.3
77.4 ±1.2
+15.1
+ MIME + DTAD
70.1
88.1
79.3
99.2
53.9
45.8
90.0
87.8
60.7
94.9
81.5
77.4 ±1.5
+15.1
Table 1: Visual ICL results (generic instruction protocol). Accuracy (%) of each base model before and after fine-tuning on MIME, averaged over 1/2/4 shots and three seeds.
Model
Size
Clock
Op.Ind
Palin
Shapes
Classif
Overall
Qwen3-VL-235B-A22B
235B
31.0
99.3
86.0
97.8
64.2
75.7
InternVL3.5-241B-A28B
241B
15.3
99.7
79.0
83.9
48.4
65.3
Claude Opus 4.6
-
43.3
90.2
76.7
99.1
39.6
69.8
Claude Opus 5
-
36.1
88.6
95.1
99.6
61.0
76.1
Qwen3-VL-8B
8B
12.8
76.7
63.9
95.8
45.7
59.0
+ MIME
8B
62.8
88.9
68.8
97.3
63.4
76.2
Table 2: Comparison to frontier and Large models on TrueMICL. Per-task and overall accuracy on the five TrueMICL tasks. Fine-tuned on MIME, Qwen3-VL-8B outperforms models roughly 30 × larger and Claude Opus 4.6, and matches Claude Opus 5; with DTAD, it outperforms all of them.
TrueMICL
Regression check
Training mix
Avg
POPE
MME-P
MME-C
LLaVA-1.5-13B (official)
18.9
84.2
1561
118
+ Multi-Image data
21.5
54.1
1509
116
+ MIME
39.7
85.2
1575
127
Table 3: Full retraining of LLaVA-1.5-13B. MIME and a same-size multi-image set (Mantis) are each added to the original instruction-tuning mix. TrueMICL accuracy is averaged over 1/2/4 shots.
Regression
Text-only ICL
Model
SI
MI
MARVEL
MMLU
GSM8K
GPQA
Qwen3-VL-8B
76.1
54.9
27.0
74.6
89.8
36.4
+ MIME
76.4
57.5
29.1
75.6
91.4
37.1
+ MIME + DTAD
76.9
58.1
31.6
75.8
92.7
37.1
Table 4: Beyond visual ICL. SI (single-image): average of MMStar, RealWorldQA, POPE, and InfoVQA. MI (multi-image): average of Mantis and LLaVA-Interleave. MARVEL is evaluated as multi-turn visual ICL (chance 25%). Text-only ICL uses multi-turn demonstrations.
Model
Correct
Answers
Images
Foreign
No
shuffled
shuffled
rule
demos
2 shots
Qwen3-VL-8B
57.5
− 19.3
− 19.8
− 43.5
− 25.3
+ MIME
74.5
− 35.7
− 35.1
− 63.8
− 48.7
+ DTAD (AM)
74.3
− 32.5
− 33.4
− 70.7
− 42.3
+ DTAD (TR)
77.8
− 34.6
− 33.6
− 71.4
− 47.6
Table 5: Trained models depend on the demonstrations. Accuracy with correct demonstrations, and change in points under each corruption, on the five TrueMICL tasks (generic instruction protocol, three seeds).
Training data
TrueMICL
Answers shuffled
Untrained
57.5 / 66.6
− 19.3 / − 30.9
Rule stated
69.6 / 78.9
− 26.3 / − 42.8
Rule hidden (ours)
74.5 / 88.6
− 35.7 / − 52.3
Table 6: Stating the rule substantially reduces the transfer gain. Accuracy (2 / 4 shots) on the TrueMICL tasks of Table 5 , and change under shuffled demonstration answers.
Figure 2: Mean visual share per layer for 1-shot (left), 2-shot (center), and 4-shot (right); lines show the mean over the 5 TrueMICL tasks and shaded bands show ±1 SD across tasks. DTAD raises the visual share in the late layers above the base model at every shot count, with a narrow spread across tasks. Fine-tuning on the same data increases it mainly in the middle layers; in the late layers it varies widely across tasks. Training with the rule stated in the prompt lowers the visual share below the base model, indicating that the model relies mostly on the textual pathway.
Model
1 shot
2 shots
4 shots
Qwen3-VL-8B
19.9 ± 2.0
20.6 ± 1.7
24.6 ± 1.8
+ MIME (rule stated)
20.2 ± 3.2
22.3 ± 2.5
26.9 ± 2.9
+ MIME
22.1 ± 3.4
26.1 ± 4.8
29.7 ± 3.5
+ MIME + DTAD (TR)
22.9 ± 2.6
26.3 ± 2.8
31.3 ± 3.3
Table 7: Attention redistribution under demonstration-text blocking. Absorption: share (%) of the attention freed by blocking demonstration text that moves to demonstration images, all layers, mean ± SD over 5 TrueMICL tasks.
Qwen3-VL-8B
+ MIME
+ DTAD (TR)
Acc@4
66.6
88.6
90.0
Acc@16
84.6
96.3
96.9
Error reduction
53.9%
67.5%
69.0%
Table 8: Error reduction from 4 to 16 shots (TrueMICL). Error reduction is the fraction of the 4-shot error removed at 16 shots. DTAD and fine-tuning keep improving with more demonstrations. The higher ERR demonstrates they improve more relative to their stronger starting point.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Shots
Clock
Operator induction
Mean
Pure text-ICL
1
100.0
99.0
99.5
2
100.0
100.0
100.0
4
100.0
100.0
100.0
Avg
100.0
99.7
99.8
Transcribe & answer
1
100.0
99.0
99.5
2
100.0
100.0
100.0
Appendix
Table 9: Preliminary study: what limits vICL? Accuracy (%) of Qwen3-VL-8B on two TrueMICL tasks (Clock, Operator induction). Pure text-ICL : the task is posed as text-only ICL using the task metadata. Transcribe & answer : the model first transcribes the images, then answers. Task instruction (baseline) : standard vICL with the official TrueMICL prompt. Generic instruction : vICL with an instruction that gives no hint about the task. Avg : mean over shots; Mean: mean over the two tasks. Near-perfect text-ICL and transcribe & answer scores show that perception and textual ICL are not the bottleneck, while the drop under a generic instruction indicates reliance on textual cues.
Source
Episodes
Shots
Mean shots
PuzzleVQA polygon sides
5000
{2,3,4}
2.94
Bongard-LOGO shape concept
5000
{2,4}
3.00
RE-ARC transform induction
5000
{3,4,5}
3.97
PuzzleVQA triangle arithmetic
5000
{2,3,4}
3.00
CLEVR
3578
{1,2,4}
2.64
Total
23578
{1,…,5}
3.14
Appendix
Table 13: Composition of the training set by source, showing the number of episodes, the range of shots used, and the mean number of shots per episode.
Model
Clock
Op.Ind
Palin
Shapes
Classif
Avg
Qwen3-VL-8B
12.8
76.7
63.9
95.8
45.7
59.0
Single-image control
11.0
88.7
80.3
94.3
33.3
61.5
+ MIME
62.8
88.9
68.8
97.3
63.4
76.2
Appendix
Table 14: Single-image control. TrueMICL per-task accuracy (%, Qwen3-VL-8B, averaged over 1/2/4 shots). The control trains on the same query images with each rule stated in text and no demonstrations. It improves over the untrained model but falls far short of training on our vICL data. The gap is largest on Clock, whose rule exists only in the visual demonstrations.