Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

Yu Wang; Sharon Li

cs.CVApr 15, 2026

Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

Authors: Yu Wang, Sharon Li

Organizations: Department of Computer Sciences, University of Wisconsin-Madison

Abstract

In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. Despite its success in large language models, the extension of ICL to multimodal settings remains poorly understood in terms of its internal mechanisms and how it differs from text-only ICL. In this work, we conduct a systematic analysis of ICL in multimodal large language models. Using identical task formulations across modalities, we show that multimodal ICL performs comparably to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations. To understand this gap, we decompose multimodal ICL into task mapping construction and task mapping transfer, and analyze how models establish cross-modal task mappings, and transfer them to query samples across layers. Our analysis reveals that current models lack reasoning-level alignment between visual and textual representations, and fail to reliably transfer learned task mappings to queries. Guided by these findings, we further propose a simple inference-stage enhancement method that reinforces task mapping transfer. Our results provide new insights into the mechanisms and limitations of multimodal ICL and suggest directions for more effective multimodal adaptation. Our code is available here.

Figures & tables

Figure 1: ( a–b ) Illustration of constructed outlier detection task Chen et al. (2025a) , where (a) shows the zero-shot setup and (b) shows the few-shot setup. In the few-shot scenario (2-shot), the model must infer from the demonstrations whether the query should be solved based on shape or color, and then apply this inferred rule to identify the outlier item in either the image or the sentence. ( c ) Performance comparison of Qwen2.5-VL-7B and Gemma-3-12B, showing that multimodal ICL suffers a clear degradation under the few-shot setting (4-shot) relative to text-only ICL. ( d ) Illustration of the multimodal ICL pipeline, including task mapping construction from demonstrations and task mapping transfer to the query.

Model	Size	ICL Type	Zero-Shot	4-shot
Qwen2.5-VL	7B	Text Only	38.80	88.40
	7B	Multimodal	45.20 ( $\uparrow 6.40$ )	63.60 ( $\downarrow 24.80$ )
	32B	Text Only	50.00	80.47
	32B	Multimodal	43.60 ( $\downarrow 6.40$ )	76.13 ( $\downarrow 4.34$ )
Gemma-3	12B	Text Only	49.40	91.40
	12B	Multimodal	44.60 ( $\downarrow 4.80$ )	72.93 ( $\downarrow 18.47$ )

Table 1: Comparison between text-only and multimodal ICL across model families and sizes.

Figure 2: Qualitative examples of error cases . (a) False task recognition: the model misreads the task mapping in the demonstrations and outputs the wrong attribute (shape star) while the true minority is color gray. (b) Correct task recognition but false answer: although the model identifies the correct task mapping (detecting the OOD sample by shape feature), it still predicts the wrong minority because it attends to incorrect regions in the image (square instead of star).

Figure 3: Proportion of error types in text-only ICL and multimodal ICL. Case 1 corresponds to incorrect task recognition, and Case 2 corresponds to correct task recognition but an incorrect answer.

Figure 4: Layer-wise visualization of attention from demonstration’s labels tokens (or the last token) to image tokens in a multimodal ICL example. The four demonstrations are labeled by the color outlier, but the model exhibits the False Task Recognition on the query and incorrectly predicts star instead of gray . Red bounding boxes denote ground-truth evidence regions. Demonstration label tokens form clear object-level grounding only in mid layers, with early layers being diffuse and deeper layers becoming noisy again. In contrast, the query token shows no meaningful grounding until the final several layers, where localization finally appears.

Figure 5: Layer-wise attention ratios over different image regions for correct ( $\bullet$ ) vs. incorrect ( $+$ ) predictions. Relative attention from demonstration label tokens to correct, false, and irrelevant evidence regions. Both models show a strong peak on correct evidence at the midlayer, indicating stable visual grounding within the demonstrations.

Model	Qwen2.5-VL		Gemma-3
Size	7B	32B	12B	27B
Zero-shot	45.20	43.60	44.60	48.00
Original (4-shot)	69.09	78.45	72.93	78.62
After UAS (4-shot)	45.44	52.19	50.56	0.0

Table 2: Causal Intervention Study. We apply our Uniform Attention Suppression (UAS) intervention, which replaces the model’s attention over image tokens with a uniform distribution, leading to significant performance degradation.

Figure 6: Comparison of Correct and Error Samples in Layer-wise Last-Token Attention to Evidence Regions.

	Size	Method	Outlier	Clock	Operator	OK-VQA
Qwen2.5-VL	7B	Vanilla	69.09 ±0.81	63.45 ±0.00	77.59 ±0.00	48.13 ±0.11
	7B	Ours	70.17 ±0.00	64.49 ±0.00	77.93 ±0.00	48.17 ±0.27
	32B	Vanilla	78.45 ±0.00	74.14 ±0.00	97.93 ±0.00	52.67 ±0.64
	32B	Ours	78.45 ±0.00	74.48 ±0.00	98.28 ±0.00	53.13 ±0.10
	12B	Vanilla	72.93 ±0.00	66.90 ±0.00	47.93 ±0.00	48.39 ±0.75
	12B	Ours	73.85 ±0.14	66.90 ±0.00	48.28 ±0.00	49.51 ±0.26

Table 3: Results of 4-shot multimodal ICL. Experiments are averaged over 3 different seeds, and the performance is in percentage with the standard deviations. Rows shaded in gray correspond to MGI (ours).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: An overview of datasets in our papers. For the datasets sourced from TrueMICL Chen et al. (2025a) (Outlier Detection, Operator Induction, Clock Math), the label to the query requires the model to learn the relationship between images and text in the demos. Meanwhile, for the natural-image datasets (OK-VQA Marino et al. (2019) ), the model can leverage query-relevant demonstrations to enhance inference on the query instance.

Figure 8: Layer-wise Relative Attention per Token of demonstration text and image tokens for different MLLMs families under the 4-shot setup. Qwen2.5-VL-7B shows a consistently text-dominant pattern across all layers, while Gemma-3-12B displays a modality-switching pattern.

Figure 9: Layer-wise Attention Head Activation Patterns on Image and Text Demonstrations under the 4-shot setup. Only a few heads show noticeable attention, while most heads remain inactive.

$\lambda$	1	1.5	2	2.5	3	3.5	4	4.5	5
Acc.	68.28	68.28	68.97	68.28	68.28	68.62	68.62	68.97	67.93

Appendix

Table 4: Effect of the strength of the intervention

\lambda

$\lambda$	11	12	13	14	15	16	17	18	19
Acc.	68.28	67.93	68.02	68.97	68.28	68.28	67.93	67.93	67.93

Appendix

Table 5: Effect of the start layer

L_{start}

Figure 10: Additional layer-wise visualization of attention from demonstration label tokens (or the last token) to image tokens in a multimodal ICL example. The four demonstrations are labeled by the shape outlier, yet the model exhibits Correct Task Recognition but False Answer on the query, incorrectly predicting triangle instead of square . Red bounding boxes denote ground-truth evidence regions . Demonstration label tokens form clear object-level grounding only in mid layers, with early layers being diffuse and deeper layers becoming noisy again. In contrast, the query token shows no meaningful grounding until the final several layers, where localization finally appears.

Setup	1-shot	2-shot	3-shot	4-shot
Baseline	65.52%	66.55%	67.59%	69.09%
Ours (MGI)	66.90%	67.59%	67.93%	70.17%

Appendix

Table 6: Effect of the number of demonstrations (

k

) on Outlier Detection performance with Qwen2.5-VL-7B.

Multimodal ICL

Multimodal in-context learning (ICL) studies how models use examples and other context to solve tasks involving images, text, audio, and video.

See all 46 papers

TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals
Sep 14, 2026Zihan Xue, Po-Yi Lu, Serhii Honcharenko +5Multimodal ICL Modality Gap in VLMs
University of California, Los Angeles · 2National Taiwan University · 3Texas A&M University +2
HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks
Mar 13, 2026Xiaoyu Li, Yuhang Liu, Xuanshuo Kang +4Multimodal ICL Multimodal Large Language Models
University of Electronic Science and Technology of China · Institute of Electronics and Information Industry Technology of Kashgar
Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning
Jun 3, 2026Niloufar Alipour Talemi, Hossein Kashiani, Fatemeh AfghahEfficient Multimodal Inference Multimodal ICL
Holcombe Department of Electrical and Computer Engineering, Clemson University, Clemson, United States.

Sep 14, 2026J/K move · Enter open · S save

TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals

Zihan Xue, Po-Yi Lu, Serhii Honcharenko, Zih-Ching Chen+4

University of California, Los Angeles · 2National Taiwan University · 3Texas A&M University+2

Multimodal ICL Modality Gap in VLMs Multimodal Model Evaluation

Open paper AskPDF arXiv

In-context learning (ICL) enables models to infer tasks from demonstrations, but existing benchmarks generally lack matched text and image versions needed to compare ICL performance across modalities. We introduce TwinICL, a procedurally generated benchmark providing such pairs for controlled comparison. Across six open-weight models and 38 tasks, multimodal ICL consistently underperforms text-only ICL, with gaps varying by task family. To test whether this gap can be recovered, we target visual access, task framing, and reasoning through three interventions. Their combination recovers strong multimodal ICL performance on a diagnostic subset, despite limited or inconsistent individual effects. To distinguish difficulties in executing tasks from those in inferring them, we evaluate models with explicit task instructions, revealing a modality gap even when the task is known. We then examine how adding demonstration inputs and outputs reshapes this gap, highlighting demonstrations' dual role as additional context to process and evidence about the task. The dataset is available at https://github.com/lab-flair/TwinICL.

Abstract

Multimodal ICL

Explore similar work

TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals

HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks

Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning