Instruction Distillation: Text Instructions as Visual Examples

Hardik Jindal; Soumyabrata Pal; Sayak Ray Chowdhury

cs.CVAug 27, 2026

Instruction Distillation: Text Instructions as Visual Examples

Authors: Hardik Jindal, Soumyabrata Pal, Sayak Ray Chowdhury

Organizations: IIT Kanpur · Adobe Research

Abstract

Visual in-context learning (ICL) with multimodal large language models (MLLMs) is effective for fine-grained visual classification, but each retrieved image example consumes several hundred context tokens, making large- $K$ settings prohibitively expensive at inference scale. We propose Instruction Distillation: an offline procedure in which the MLLM itself generates, for each individual training image, a structured identification instruction encoding general appearance cues, features that differentiate the class from visually similar ones, and a common confusion point. Unlike prior work that produces a single description per class, our instructions are generated per training image, preserving the intra-class visual diversity that per-class descriptions collapse. At inference time, we study five configurations sharing a single CLIP retrieval index: zero-shot, image ICL, instruction-only ICL, and two hybrid variants in which retrieved neighbors are split between images and instructions. Across seven fine-grained benchmarks and two MLLM backbones, instruction based pipelines match, or exceeds image ICL at $K{=}1$ and reduces per-query tokens by $2.9\times$ and inference latency by $3.3\times$ at $K{=}5$ . Hybrid configurations further show that visual and textual ICL signals are complementary, images give visual patterns to learn and see, while instructions give explicit rules and logic. When both of these are provided, the quality of context improves, which is noticeable in the performance.

Figures & tables

Figure 1: Instruction Distillation. Images are replaced by text-based instructions and passed as context during inference. It results in an improved accuracy score with reduced token consumption.

Figure 2: Inference pipeline overview. The four context-construction strategies. P1 (blue) places $K$ retrieved images in context. P2 (orange) replaces them with $K$ text instructions, the most token-efficient configuration. P3 (green) uses $K$ images as primary context and $K^{\prime}$ instructions as supplementary. P4 (purple) reverses the role assignment: $K$ instructions are primary and $K^{\prime}$ images are supplementary.

Configuration	Tokens
P0: Zero-Shot	$\approx$ 574
P1(a): Image ICL ( $K{=}5$ )	$\approx$ 3,400
P1(b): Image-compressed ICL ( $K{=}5$ )	$\approx$ 1940
P2: Instruction ICL ( $K{=}5$ )	$\approx$ 1,165
P3: Hybrid ( $K{=}3$ , $K^{\prime}{=}2$ )	$\approx$ 2,106
P4: Flip Hybrid ( $K{=}3$ , $K^{\prime}{=}2$ )	$\approx$ 1,950

Table 1: Token Cost. Approximate input token cost for each inference configuration at

K{=}5

Figure 3: Accuracy versus token budget. Mean per-class accuracy (%) plotted against input tokens for P1(a) (solid blue), P1(b) (dashed blue), P2 (orange), P3 with $K^{\prime}=3$ (green), and P4 with $K^{\prime}=3$ (purple). Top row: Gemma-3-4B-IT. Bottom row: Qwen2.5-VL-7B-Instruct. Each marker corresponds to one $(K)$ or $(K,K^{\prime})$ configuration; markers are connected in order of increasing token cost.

K	Per-class	Caption	Single rule	Paragraph	2-rule	3-rule
1	0.6609	0.6178	0.6140	0.6361	0.7286	0.7224
5	0.6972	0.7893	0.7856	0.7830	0.8304	0.8276
10	0.6565	0.7860	0.7920	0.8010	0.8215	0.8545
25	0.6672	0.7544	0.7678	0.7711	0.8124	0.8390

Table 2: Ablation study with PETS and FGVC. Comparison of three basic instruction formats: caption, paragraph, and a single rule with our 3-rule format, which outperforms all alternatives across

K

, indicating that both structured decomposition and the inclusion of general and error-avoidance cues contribute to performance.

Gemma-3-4B-IT
Pipeline	Tokens	PETS	FOOD	DTD	ESAT	FGVC	FLWR	S397	Avg.
Low budget ( $\sim$ 1.5k tokens)
P1(a) ( $K{=}1$ )	1,152	67.3	76.3	71.6	27.8	25.7	69.0	56.2	56.3
P1(b) ( $K{=}3$ )	1,402	83.7	85.4	78.2	86.9	32.8	84.5	66.9	74.0
P2 ( $K{=}10$ )	1,700	85.5	88.1	72.4	89.2	37.4	85.1	73.4	75.9
P3 ( $K{=}1,K^{\prime}{=}3$ )	1,493	83.9	85.4	79.7	88.5	38.9	83.4	63.9	74.8

Table 3: Pipeline comparison across token budgets and seven fine-grained benchmarks. Mean per-class accuracy (%) for P1(a), P1(b), P2, P3 and P4 on Gemma-3-4B-IT (top) and Qwen2.5-VL-7B-Instruct (bottom), evaluated at three token budgets: low (

\sim

1.5k), mid (

\sim

3.3k), and high (

\sim

6.3k). The Tokens column reports the actual input cost of the chosen

(K,K^{\prime})

configuration. Best per dataset within each budget block is shown in bold .

Figure 4: Average accuracy by pipeline and token budget. Mean per-class accuracy averaged across seven fine-grained benchmarks under three token budgets, for P1(a), P1(b), P2, P3, and P4. Instruction-based pipelines (P1(b), P2, P3, P4) substantially outperform full-resolution image ICL (P1(a)) at the Low budget.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Abbr.	Dataset	Images	Classes
FOOD	Food-101	25,250	101
DTD	DTD	1,692	47
FGVC	FGVC Aircraft	3,333	100
PETS	Oxford-IIIT Pets	3,699	37
ESAT	EuroSAT	8,100	10
FLWR	Flowers102	2,463	102

Appendix

Table 4: Dataset details. Summary details of the datasets used in our experiments.

Figure 5: Pipeline accuracy across token budgets and datasets. Mean per-class accuracy for the five inference pipelines (P1(a): full-resolution image ICL, P1(b): compressed-image ICL, P2: instruction ICL, P3: hybrid, P4: flip hybrid) on seven fine-grained benchmarks, evaluated under three per-query token budgets: Low ( $\sim$ 1.5k), Mid ( $\sim$ 3.3k), and High ( $\sim$ 6.3k). P1(b) is reported at the Low and Mid budgets only. Top: Gemma-3-4B-IT. Bottom: Qwen2.5-VL-7B-Instruct. At matched token budgets, the hybrid P3 is competitive with, or stronger than, P1(a) on most datasets, while P2 at low budgets approaches mid-budget P3 on PETS, FOOD, ESAT, and FLWR. This is a per-dataset bar-chart view of the same results plotted as curves in Figure 3 in the main text.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.5286	–	–	–	–	–	–	–	–
1	0.6728	0.663	0.7224	0.8016	0.8356	0.8388	0.8259	0.8508	0.8714
2	0.7827	0.7725	0.8077	0.8555	0.8626	0.8713	0.8441	0.8619	0.8762
3	0.8375	0.8366	0.8175	0.87	0.8697	0.8779	0.8541	0.8688	0.8772
5	0.8742	0.8624	0.8276	0.8754	0.8858	0.8903	0.8521	0.8647	0.8778

Appendix

Table 5: Accuracy metrics for Gemma-3-4b-it across various shot counts (

K

) for PETS dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.1103	–	–	–	–	–	–	–	–
1	0.7627	0.752	0.7850	0.8154	0.8410	0.8537	0.8238	0.8464	0.8628
2	0.8332	0.8292	0.8236	0.8520	0.8618	0.8693	0.8500	0.8626	0.8689
3	0.8565	0.8536	0.8413	0.8650	0.8727	0.8788	0.8615	0.8687	0.8730
5	0.8726	0.8694	0.8650	0.8671	0.8718	0.874	0.8712	0.8722	0.8760

Appendix

Table 6: Accuracy metrics for Gemma-3-4b-it across various shot counts (

K

) for FOOD dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.1335	–	–	–	–	–	–	–	–
1	0.7159	0.7028	0.7664	0.7452	0.7893	0.7968	0.7765	0.8015	0.7994
2	0.7755	0.7601	0.7686	0.8175	0.8271	0.8313	0.7803	0.7920	0.7893
3	0.7978	0.7815	0.7510	0.8180	0.8340	0.8335	0.7702	0.7787	0.7770
5	0.8132	0.8027	0.7579	0.8218	0.8335	0.8404	0.7537	0.7824	0.7813

Appendix

Table 7: Accuracy metrics for Gemma-3-4b-it across various shot counts (

K

) for DTD dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.0437	–	–	–	–	–	–	–	–
1	0.2781	0.2781	0.8571	0.8529	0.8759	0.8847	0.8811	0.8924	0.9000
2	0.7900	0.7900	0.8715	0.8892	0.9066	0.9054	0.8906	0.9006	0.9049
3	0.8690	0.8690	0.8892	0.8846	0.9074	0.9053	0.8995	0.9043	0.9094
5	0.8996	0.8996	0.8894	0.8859	0.9035	0.9109	0.8901	0.8971	0.9078

Appendix

Table 8: Accuracy metrics for Gemma-3-4b-it across various shot counts (

K

) for ESAT dataset. ESAT images are already low-resolution, so Acc. P1(a) and P1(b) coincide.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.0792	–	–	–	–	–	–	–	–
1	0.2565	0.2371	0.30153	0.34177	0.37143	0.3894	0.3288	0.3774	0.405
2	0.31023	0.2885	0.33783	0.38793	0.4041	0.4119	0.3306	0.3717	0.3903
3	0.36513	0.3277	0.34713	0.42184	0.43444	0.44344	0.3399	0.3706	0.3819
5	0.40144	0.366	0.35823	0.447	0.4578	0.4647	0.3309	0.3675	0.3675

Appendix

Table 9: Accuracy metrics for Gemma-3-4b-it across various shot counts (

K

) for FGVC dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.1068	–	–	–	–	–	–	–	–
1	0.6901	0.688	0.7921	0.7803	0.8186	0.8343	0.8882	0.9176	0.9264
2	0.8284	0.813	0.8843	0.9	0.9058	0.9137	0.8931	0.9088	0.9225
3	0.8715	0.845	0.8892	0.9225	0.9382	0.9421	0.8941	0.9078	0.9215
5	0.9049	0.901	0.8715	0.9421	0.933	0.9382	0.8843	0.9186	0.9205

Appendix

Table 10: Accuracy metrics for Gemma-3-4b-it across various shot counts (

K

) for FLWR dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.1925	–	–	–	–	–	–	–	–
1	0.5618	0.5549	0.5817	0.5601	0.6016	0.639	0.6453	0.6837	0.6984
2	0.6319	0.6406	0.6512	0.6446	0.6671	0.6925	0.6849	0.6926	0.7029
3	0.67	0.6686	0.6756	0.6848	0.699	0.7087	0.6967	0.7053	0.7096
5	0.7065	0.7092	0.7111	0.7136	0.7214	0.7266	0.7051	0.7127	0.7215

Appendix

Table 11: Accuracy metrics for Gemma-3-4b-it across various shot counts (

K

) for S397 dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.3571	–	–	–	–	–	–	–	–
1	0.7915	0.7856	0.7969	0.8128	0.8336	0.8435	0.8253	0.8461	0.8601
2	0.8329	0.8291	0.8368	0.8387	0.8422	0.8501	0.8383	0.8596	0.8671
3	0.8466	0.8389	0.8472	0.8551	0.8563	0.8586	0.8467	0.8659	0.872
5	0.8615	0.8454	0.8599	0.8687	0.8702	0.8703	0.8563	0.8719	0.875

Appendix

Table 12: Accuracy metrics for Qwen2.5-VL-7B-Instruct across various shot counts (

K

) for FOOD dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.1672	–	–	–	–	–	–	–	–
1	0.759	0.7218	0.7452	0.7648	0.776	0.7898	0.7712	0.7787	0.7905
2	0.8	0.773	0.7735	0.7973	0.8042	0.8132	0.7877	0.7973	0.8026
3	0.8085	0.7936	0.7835	0.8106	0.8212	0.8281	0.7856	0.8015	0.8048
5	0.8266	0.7989	0.7957	0.8138	0.8239	0.8329	0.7745	0.7835	0.7888

Appendix

Table 13: Accuracy metrics for Qwen2.5-VL-7B-Instruct across various shot counts (

K

) for DTD dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.6903	–	–	–	–	–	–	–	–
1	0.8285	0.8077	0.8111	0.8574	0.8705	0.8743	0.8078	0.8323	0.846
2	0.8552	0.8331	0.843	0.8757	0.8841	0.8852	0.8192	0.8601	0.8746
3	0.8721	0.8504	0.8577	0.8887	0.892	0.8912	0.819	0.8708	0.886
5	0.8972	0.8864	0.8778	0.9013	0.9117	0.9084	0.8252	0.8781	0.8917

Appendix

Table 14: Accuracy metrics for Qwen2.5-VL-7B-Instruct across various shot counts (

K

) for PETS dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.2413	–	–	–	–	–	–	–	–
1	0.8529	0.84	0.8441	0.9147	0.9098	0.9127	0.9245	0.9421	0.946
2	0.9107	0.899	0.8588	0.949	0.9509	0.949	0.9303	0.9441	0.951
3	0.9264	0.9086	0.8725	0.9578	0.9617	0.9568	0.9334	0.948	0.9509
5	0.947	0.9194	0.8784	0.9627	0.9637	0.9637	0.9313	0.945	0.9529

Appendix

Table 15: Accuracy metrics for Qwen2.5-VL-7B-Instruct across various shot counts (

K

) for FLWR dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.2826	–	–	–	–	–	–	–	–
1	0.4794	0.4449	0.4392	0.4967	0.5265	0.5391	0.4689	0.5163	0.5433
2	0.5345	0.4875	0.4569	0.5313	0.5616	0.5535	0.4752	0.5304	0.5409
3	0.5598	0.5139	0.4784	0.5709	0.5778	0.5766	0.4836	0.5286	0.543
5	0.5916	0.5584	0.5148	0.5892	0.5979	0.5955	0.4926	0.5397	0.5442

Appendix

Table 16: Accuracy metrics for Qwen2.5-VL-7B-Instruct across various shot counts (

K

) for FGVC dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.1164	–	–	–	–	–	–	–	–
1	0.6797	0.6797	0.8521	0.752	0.7875	0.7816	0.8759	0.8841	0.883
2	0.8217	0.8217	0.8486	0.8218	0.8156	0.8141	0.8789	0.8878	0.8885
3	0.8452	0.8452	0.8529	0.8402	0.828	0.8277	0.87	0.8941	0.8915
5	0.8406	0.8406	0.8519	0.8362	0.844	0.8351	0.8563	0.8778	0.8785

Appendix

Table 17: Accuracy metrics for Qwen2.5-VL-7B-Instruct across various shot counts (

K

) for ESAT dataset. ESAT images are already low-resolution, so Acc. P1(a) and P1(b) coincide.

K	Acc. P1		Acc.	Accuracy P3			Accuracy P4
	P1(a)	P1(b)	P2	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$	$k^{\prime}=1$	$k^{\prime}=2$	$k^{\prime}=3$
0	0.2137	–	–	–	–	–	–	–	–
1	0.6255	0.6163	0.6233	0.6803	0.6996	0.7157	0.6685	0.6868	0.6977
2	0.6755	0.6544	0.6813	0.7022	0.7114	0.7228	0.7009	0.7151	0.7238
3	0.6986	0.6865	0.7048	0.7179	0.7269	0.7324	0.7186	0.73	0.7361
5	0.7202	0.7089	0.7247	0.7256	0.7348	0.7475	0.7376	0.7465	0.7516

Appendix

Table 18: Accuracy metrics for Qwen2.5-VL-7B-Instruct across various shot counts (

K

) for S397 dataset. Acc. P1(a) is with full-resolution images in context, P1(b) uses compressed images.

Dataset	Groundedness	Genericness	Hallucination
PETS	4.67	3.03	3.92
FOOD	4.82	3.01	3.47
DTD	4.84	3.22	4.66
FGVC	4.78	3.54	4.42
FLWR	4.80	3.26	4.53
S397	4.81	3.00	3.48

Appendix

Table 19: Instruction Quality. Mean scores given by LLM as a judge for various datasets acrross different parameters.

K	Label-only	Per-image	$\Delta$ (pp)
1	0.7172	0.7224	+0.5
2	0.7922	0.8077	+1.6
3	0.7991	0.8175	+1.8
5	0.8072	0.8276	+2.0
10	0.8272	0.8545	+2.7
25	0.8023	0.8390	+3.7

Appendix

Table 20: Label-prior control. Top-1 accuracy for label-only vs. per-image instructions on Oxford-IIIT Pets (top) and FGVC (bottom) across

K\in\{1,2,3,5,10,25\}

. Per-image instructions outperform the label-only control at every

K

on both datasets, with no exceptions.

K	Pipeline	Latency (s)	QPS	Speedup
K	Pipeline	Latency (s)	QPS	Latency	QPS
1	P1	0.2444	17.68	—	—
1	P2	0.1405	26.35	1.74x	1.49x
5	P1	0.5486	2.57	—	—
5	P2	0.1678	24.40	3.27x	9.49x
10	P1	1.1492	1.18	—	—

Appendix

Table 21: Latency and QPS comparison between P1 and P2 pipelines across different values of K.

Panel of Figure 6 — Figure 6: Per-query latency vs. shot count $K$ for P1 and P2 (Table 21 ). P1’s latency grows steeply with $K$ since each additional shot adds a full image ( $\approx$ 550 tokens); P2’s grows far more slowly, so the speedup widens from 1.74 $\times$ at $K{=}1$ to 5.44 $\times$ at $K{=}10$ .

Panel of Figure 7 — Figure 7: Amortization of the one-time instruction-generation cost. P1 has no upfront cost; P2 includes a worst-case $\approx$ 1-hour offline generation cost, amortized against per-query savings at $K{=}5$ .

Budget	Pipeline	Tokens	Input TPS	Output TPS	Total TPS
Low (~1.5k)	P1 (K=1)	1,152	2,436.65	17.42	2,454.07
	P2 (K=10)	1,700	6,480.92	21.25	6,502.18
	P3 (K=1, K’=3)	1,493	4,165.58	18.71	4,184.28
	P4 (K=3, K’=1)	1,600	3,237.63	19.83	4,547.96
Mid (~3.3k)	P1 (K=5)	3,400	3,045.66	8.20	3,053.86
	P2 (K=25)	3,305	8,548.72	13.28	8,561.99

Appendix

Table 22: Token-level comparison across low, mid, and high token budgets.

Panel of Figure 8 — Figure 8: Projected P1 vs. P2 completion time for 100K queries. Reductions: 69.4% (Serial, $K{=}5$ ), 89.5% (Throughput, $K{=}5$ ), 39.4% (Serial, token-matched Mid budget). At 1M queries the same ratios scale linearly, extending the savings to $\approx$ 105.8h (Serial) and $\approx$ 96.7h (Throughput).

Component	Tokens (avg)
System prompt	15–40
Per image example (patches)	$\approx$ 550
Per compressed image example (patches)	$\approx$ 258
Per image example (label text)	$\approx$ 12
Per instruction (3 rules)	$\approx$ 95
Per instruction (label text)	$\approx$ 12

Appendix

Table 23: Per-component token counts . Instruction tokens are measured as the mean token count of generated instructions across all training images and all models.

In-Context Learning

In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.

See all 406 papers

Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts
Oct 22, 2026Nimrod Shabtay, Asaf Yehudai, Avihu Dekel +2Multimodal ICL In-Context Learning
IBM Research · Tel-Aviv University
V-GIFT: Boosting Visual Instruction Tuning with Self-Supervised Guidance
Apr 14, 2026Sophia Sirko-Galouchenko, Monika Wysoczanska, Andrei Bursuc +2Visual Instruction Tuning Large Vision-Language Models
Sorbonne Universite, CNRS, ISIR, F-75005 Paris, France · Institut universitaire de France (IUF)
Qwen-Image-Flash: Rethinking the Training Recipe for Few-Step Distillation
Jun 2, 2026Tianhe Wu, Zikai Zhou, Kun Yan +23Diffusion Model Distillation Few-Step Diffusion Sampling

Oct 22, 2026J/K move · Enter open · S save

Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts

Nimrod Shabtay, Asaf Yehudai, Avihu Dekel, Eli Schwartz+1

IBM Research · Tel-Aviv University

Multimodal ICL In-Context Learning Visual Evidence Sensitivity in VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) struggle with in context learning compared to their textual counterparts. Our preliminary study shows that VLMs rely on textual shortcuts and underutilize the visual demonstrations. We therefore hypothesize that VLMs are insufficiently trained to perform visual in-context learning (vICL). To this end, we construct MIME, a task-agnostic vICL training set without textual shortcuts. We ensure the instruction does not state the rule linking the demonstration images to their answers, so the model must infer it from the visual demonstrations. Fine-tuning on MIME improves vICL accuracy by 8.1 to 18.0 points across three VLMs, preserves single-image performance, and improves multi-image and text only in-context learning. In contrast, stating the rule in the training data sharply reduces this gain. We further propose DTAD (Demonstration Text Attention Dropout), which randomly withholds demonstration labels during training so that the VLM must infer the rule from the demonstration images. While achieving comparable vICL accuracy to fine-tuning on MIME, DTAD yields more consistent attention to the demonstration images. On TrueMICL, Qwen3-VL-8B trained on MIME matches frontier models such as Claude Opus 5, and surpasses them with DTAD. Finally, adding MIME to the instruction-tuning mix of LLaVA 1.5 13B more than doubles its vICL accuracy without hurting single-image abilities, whereas a same-size multi-image set improves it only slightly. These results suggest that vICL data should be a standard component of VLM training.

Abstract

In-Context Learning

Explore similar work

Watch and Learn: Teaching Visual In-Context Learning by Removing Textual Shortcuts

V-GIFT: Boosting Visual Instruction Tuning with Self-Supervised Guidance

Qwen-Image-Flash: Rethinking the Training Recipe for Few-Step Distillation