Visual in-context learning (ICL) enables vision-language models (VLMs) to adapt at test time via demonstration examples, but its effectiveness depends critically on the examples selected. Recent counterfactual selection methods improve ICL by constructing demonstrations that expose how individual attribute changes affect the answer. However, they require multi-stage pipelines involving attribute extraction, caption engineering, and composed image matching. We propose a simpler alternative grounded in a key observation: a model's own prediction errors are natural counterfactual demonstrations that directly encode its failure modes. Our method, SMILE (Self-Mined Visual In-Context Learning from Errors), builds a model-specific pool of hard negatives offline by recording the target VLM's own mistakes. At test time, the prediction serves as a direction signal that surfaces, in the semantic embedding space, the past mistakes most likely to recur for the current query. This prediction-conditioned identification provides a unified mechanism for both closed-set classification and open-ended visual question answering without task-specific design. Across four benchmarks and four VLMs, SMILE delivers an average task-score gain of +3.09 points over the SOTA baseline at 5x lower latency. Project page: https://tree-shu-zhao.github.io/SMILE/
Figures & tables
Figure 1: A VLM’s own past mistakes can steer attention away from visual distractors and onto the target region. Direct answering and positive-only retrieval focus on visually similar but misleading evidence, while attribute-based counterfactual retrieval fails to isolate the relevant material distinction. SMILE instead retrieves a hard negative with the same error pattern from the mined pool, making the distractor evidence explicit and steering the VLM toward the correct visual region.
Figure 2: Exact confusion-pair reuse is concentrated for classification but sparse for VQA. Green : source-corpus exact error-pair distribution; Red : validation errors covered by the top- K most frequent source pairs; Grey : permuted-prediction null.
Figure 3: Wrong predictions remain semantically closer to the ground truth than random answers. Rows are task regimes; columns span model size/generation/family. Every cell shows the wrong-prediction distribution shifted to the right of the random-GT baseline, with a positive gap Δ .
Figure 4: SMILE mines failures offline and reuses them online as counterfactual demonstrations. Left: the target VLM is run on the labeled corpus; examples whose voted prediction disagrees with the ground truth are stored with both the gold answer and the mistaken answer. Right: for a new query, image similarity retrieves positive demonstrations, while the query’s zero-shot prediction is matched in text-embedding space to stored mistaken predictions; the resulting hard negatives are appended to the context before final inference.
Model
Method
CUB
Flowers
OK-VQA
VizWiz
Average
Acc
F1
Acc
F1
EM
F1
EM
F1
Score
F1
Gemma3-4B
None
10.56
8.19
46.76
46.50
19.12
26.70
39.85
57.26
29.07
34.66
Random
9.98
8.10
37.01
39.51
24.81
30.74
51.84
67.08
30.91
36.36
RICES
65.40
67.62
86.70
87.43
26.65
32.72
56.08
70.40
58.71
64.54
MUIER
65.21
67.42
86.39
87.28
26.87
32.82
56.59
70.75
58.77
64.57
MMICES
13.95
10.98
38.09
38.20
26.24
32.41
52.72
68.21
32.75
37.45
Table 1: Main comparison with both primary metric and F1 reported per dataset. The best and second best scores per model and metric are highlighted.
Figure 5: Hard negatives concentrate attention on the discriminating region and flip the prediction. Attention from the final prompt token to query-image patches, with vs. without the prediction-conditioned hard-negative block.
Full
Uncovered
Fair
Full corpus
82.15
76.27
89.77
Hard only
80.34
80.00
83.27
Clean only
46.12
11.66
92.42
Table 2: Pool decomposition on CUB. Uncovered : the query’s true class has no clean examples. Fair : both pools contain ≥4 examples of the true class.
HN-pool source
CUB
OK-VQA
Q2.5-3B
Q2.5-7B
Q3-8B
G-4B
G-12B
Q2.5-3B
Q2.5-7B
Q3-8B
G-4B
G-12B
Qwen2.5-VL-3B
+0.00
+0.77
-0.23
+1.69
-1.20
+0.00
+0.24
-1.47
-1.98
-0.71
Qwen2.5-VL-7B
+0.27
+0.00
-0.12
-0.05
-0.65
-0.38
+0.00
-1.19
-1.89
-0.79
Qwen3-VL-8B
+0.26
+0.38
+0.00
+0.67
-0.58
-0.12
+0.29
+0.00
-0.50
+0.52
Gemma-3-4B
+0.30
+1.18
-0.09
+0.00
-0.02
-0.43
-0.36
-1.20
+0.00
-0.67
Gemma-3-12B
+0.53
+0.01
+0.32
+0.31
+0.00
-0.30
+0.04
-0.71
-1.55
+0.00
Table 3: Cross-model hard-negative pool transfer heatmap. Each cell reports the change in CUB accuracy or OK-VQA EM relative to the self-mined pool for the same inference backbone and dataset.
Pos.
Rand. HN
Pred. HN
CUB
OK-VQA
✗
✗
✗
14.83
33.31
✓
✗
✗
81.18
43.16
✓
✓
✗
81.23
42.86
✓
✗
✓
82.55
47.80
Table 4: Ablation on component .
CUB
OK-VQA
Avg.
EM
82.34
40.46
61.40
F1
82.39
46.65
64.52
Semantic
82.55
47.80
65.18
Table 5: Ablation on hard selection .
Figure 6: Hyperparameter sweep over positive ( K ) and negative ( N ) demonstration counts.
Figure 7: Per-query inference latency across four VLMs. SMILE is 3.6 – 6.0× faster than CIRCLES on H100 (top) and 4.2 – 6.0× faster on H200 (bottom).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
# samples
Gemma-3-12B
Gemma-3-4B
Qwen2.5-VL-7B
Qwen2.5-VL-3B
CUB-200
5,994
3.3
1.9
2.4
1.6
Flowers-102
2,040
0.9
0.7
0.8
0.6
OK-VQA
9,009
5.1
3.2
3.5
2.5
VizWiz
20,523
13.3
6.7
11.3
6.8
Appendix
Table 6: Offline mining time on 8 H100 GPUs. Wall-clock minutes for one full R=3 -vote zero-shot pass over each dataset’s labeled corpus.
Encoder
EM
F1
CLIP ViT-B/32
70.85
70.50
CLIP ViT-B/16
74.63
74.53
CLIP ViT-L/14
79.79
79.72
CLIP ViT-H/14
83.88
83.54
CLIP ViT-G/14
82.55
82.30
SigLIP 2 base/16-224
76.68
76.36
Appendix
Table 7: Effect of the similarity encoder on SMILE performance. CUB-200 with Qwen2.5-VL-7B, swapping the encoder used for both positive retrieval and hard-negative selection while keeping all other settings fixed. CLIP ViT-G/14 is the paper baseline used in all other experiments.
Model
CUB
Flowers
OK-VQA
VizWiz
Acc
F1
Acc
F1
EM
F1
EM
F1
Qwen3-VL-4B
84.08
89.42
99.45
99.46
44.15
49.03
81.94
87.95
Qwen3-VL-8B
83.87
89.41
99.61
99.62
47.50
51.70
82.39
87.51
Appendix
Table 8: Additional SMILE results on Qwen3-VL backbones. CUB and Flowers report accuracy / weighted F1; OK-VQA and VizWiz report exact match / token-level F1.
k
queries
natural
hard-only
clean-only
clean − hard
2
2,161
90.19
83.80
86.49
+2.69
4
1,584
89.77
83.27
92.42
+9.15
8
1, 844
89.22
79.74
94.55
+14.81
12
1, 385
87.79
78.44
96.10
+17.66
Appendix
Table 9: Fair-subset analysis: queries restricted to classes with ≥k examples in both pools. Natural wins at small k (best individual visual matches); clean-only dominates at k≥4 (consistent class signal beats individual visual quality); hard-only never wins on a fair subset.