In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. Despite its success in large language models, the extension of ICL to multimodal settings remains poorly understood in terms of its internal mechanisms and how it differs from text-only ICL. In this work, we conduct a systematic analysis of ICL in multimodal large language models. Using identical task formulations across modalities, we show that multimodal ICL performs comparably to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations. To understand this gap, we decompose multimodal ICL into task mapping construction and task mapping transfer, and analyze how models establish cross-modal task mappings, and transfer them to query samples across layers. Our analysis reveals that current models lack reasoning-level alignment between visual and textual representations, and fail to reliably transfer learned task mappings to queries. Guided by these findings, we further propose a simple inference-stage enhancement method that reinforces task mapping transfer. Our results provide new insights into the mechanisms and limitations of multimodal ICL and suggest directions for more effective multimodal adaptation. Our code is available here.
Figures & tables
Figure 1: ( a–b ) Illustration of constructed outlier detection task Chen et al. (2025a) , where (a) shows the zero-shot setup and (b) shows the few-shot setup. In the few-shot scenario (2-shot), the model must infer from the demonstrations whether the query should be solved based on shape or color, and then apply this inferred rule to identify the outlier item in either the image or the sentence. ( c ) Performance comparison of Qwen2.5-VL-7B and Gemma-3-12B, showing that multimodal ICL suffers a clear degradation under the few-shot setting (4-shot) relative to text-only ICL. ( d ) Illustration of the multimodal ICL pipeline, including task mapping construction from demonstrations and task mapping transfer to the query.
Model
Size
ICL Type
Zero-Shot
4-shot
Qwen2.5-VL
7B
Text Only
38.80
88.40
Multimodal
45.20 ( ↑6.40 )
63.60 ( ↓24.80 )
32B
Text Only
50.00
80.47
Multimodal
43.60 ( ↓6.40 )
76.13 ( ↓4.34 )
Gemma-3
12B
Text Only
49.40
91.40
Multimodal
44.60 ( ↓4.80 )
72.93 ( ↓18.47 )
Table 1: Comparison between text-only and multimodal ICL across model families and sizes.
Figure 2: Qualitative examples of error cases . (a) False task recognition: the model misreads the task mapping in the demonstrations and outputs the wrong attribute (shape star) while the true minority is color gray. (b) Correct task recognition but false answer: although the model identifies the correct task mapping (detecting the OOD sample by shape feature), it still predicts the wrong minority because it attends to incorrect regions in the image (square instead of star).
Figure 3: Proportion of error types in text-only ICL and multimodal ICL. Case 1 corresponds to incorrect task recognition, and Case 2 corresponds to correct task recognition but an incorrect answer.
Figure 4: Layer-wise visualization of attention from demonstration’s labels tokens (or the last token) to image tokens in a multimodal ICL example. The four demonstrations are labeled by the color outlier, but the model exhibits the False Task Recognition on the query and incorrectly predicts star instead of gray . Red bounding boxes denote ground-truth evidence regions. Demonstration label tokens form clear object-level grounding only in mid layers, with early layers being diffuse and deeper layers becoming noisy again. In contrast, the query token shows no meaningful grounding until the final several layers, where localization finally appears.
Figure 5: Layer-wise attention ratios over different image regions for correct ( ∙ ) vs. incorrect ( + ) predictions. Relative attention from demonstration label tokens to correct, false, and irrelevant evidence regions. Both models show a strong peak on correct evidence at the midlayer, indicating stable visual grounding within the demonstrations.
Model
Qwen2.5-VL
Gemma-3
Size
7B
32B
12B
27B
Zero-shot
45.20
43.60
44.60
48.00
Original (4-shot)
69.09
78.45
72.93
78.62
After UAS (4-shot)
45.44
52.19
50.56
0.0
Table 2: Causal Intervention Study. We apply our Uniform Attention Suppression (UAS) intervention, which replaces the model’s attention over image tokens with a uniform distribution, leading to significant performance degradation.
Figure 6: Comparison of Correct and Error Samples in Layer-wise Last-Token Attention to Evidence Regions.
Size
Method
Outlier
Clock
Operator
OK-VQA
Qwen2.5-VL
7B
Vanilla
69.09 ±0.81
63.45 ±0.00
77.59 ±0.00
48.13 ±0.11
7B
Ours
70.17 ±0.00
64.49 ±0.00
77.93 ±0.00
48.17 ±0.27
32B
Vanilla
78.45 ±0.00
74.14 ±0.00
97.93 ±0.00
52.67 ±0.64
32B
Ours
78.45 ±0.00
74.48 ±0.00
98.28 ±0.00
53.13 ±0.10
12B
Vanilla
72.93 ±0.00
66.90 ±0.00
47.93 ±0.00
48.39 ±0.75
12B
Ours
73.85 ±0.14
66.90 ±0.00
48.28 ±0.00
49.51 ±0.26
Table 3: Results of 4-shot multimodal ICL. Experiments are averaged over 3 different seeds, and the performance is in percentage with the standard deviations. Rows shaded in gray correspond to MGI (ours).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: An overview of datasets in our papers. For the datasets sourced from TrueMICL Chen et al. (2025a) (Outlier Detection, Operator Induction, Clock Math), the label to the query requires the model to learn the relationship between images and text in the demos. Meanwhile, for the natural-image datasets (OK-VQA Marino et al. (2019) ), the model can leverage query-relevant demonstrations to enhance inference on the query instance.
Figure 8: Layer-wise Relative Attention per Token of demonstration text and image tokens for different MLLMs families under the 4-shot setup. Qwen2.5-VL-7B shows a consistently text-dominant pattern across all layers, while Gemma-3-12B displays a modality-switching pattern.
Figure 9: Layer-wise Attention Head Activation Patterns on Image and Text Demonstrations under the 4-shot setup. Only a few heads show noticeable attention, while most heads remain inactive.
λ
1
1.5
2
2.5
3
3.5
4
4.5
5
Acc.
68.28
68.28
68.97
68.28
68.28
68.62
68.62
68.97
67.93
Appendix
Table 4: Effect of the strength of the intervention λ .
λ
11
12
13
14
15
16
17
18
19
Acc.
68.28
67.93
68.02
68.97
68.28
68.28
67.93
67.93
67.93
Appendix
Table 5: Effect of the start layer Lstart .
Figure 10: Additional layer-wise visualization of attention from demonstration label tokens (or the last token) to image tokens in a multimodal ICL example. The four demonstrations are labeled by the shape outlier, yet the model exhibits Correct Task Recognition but False Answer on the query, incorrectly predicting triangle instead of square . Red bounding boxes denote ground-truth evidence regions . Demonstration label tokens form clear object-level grounding only in mid layers, with early layers being diffuse and deeper layers becoming noisy again. In contrast, the query token shows no meaningful grounding until the final several layers, where localization finally appears.
Setup
1-shot
2-shot
3-shot
4-shot
Baseline
65.52%
66.55%
67.59%
69.09%
Ours (MGI)
66.90%
67.59%
67.93%
70.17%
Appendix
Table 6: Effect of the number of demonstrations ( k ) on Outlier Detection performance with Qwen2.5-VL-7B.
In-context learning (ICL) enables models to infer tasks from demonstrations, but existing benchmarks generally lack matched text and image versions needed to compare ICL performance across modalities. We introduce TwinICL, a procedurally generated benchmark providing such pairs for controlled comparison. Across six open-weight models and 38 tasks, multimodal ICL consistently underperforms text-only ICL, with gaps varying by task family. To test whether this gap can be recovered, we target visual access, task framing, and reasoning through three interventions. Their combination recovers strong multimodal ICL performance on a diagnostic subset, despite limited or inconsistent individual effects. To distinguish difficulties in executing tasks from those in inferring them, we evaluate models with explicit task instructions, revealing a modality gap even when the task is known. We then examine how adding demonstration inputs and outputs reshapes this gap, highlighting demonstrations' dual role as additional context to process and evidence about the task. The dataset is available at https://github.com/lab-flair/TwinICL.
Zihan Xue, Po-Yi Lu, Serhii Honcharenko +5
University of California, Los Angeles · 2National Taiwan University · 3Texas A&M University +2
In-Context Learning (ICL) is a significant paradigm for Large Multimodal Models (LMMs), using a few in-context demonstrations (ICDs) for new task adaptation. However, its performance is sensitive to demonstration configurations and computationally expensive. Mathematically, the influence of these demonstrations can be decomposed into a dynamic mixture of the standard attention output and the context values. Current approximation methods simplify this process by learning a "shift vector". Inspired by the exact decomposition, we introduce High-Fidelity In-Context Learning (HIFICL) to more faithfully model the ICL mechanism. HIFICL consists of three key components: 1) a set of "virtual key-value pairs" to act as a learnable context, 2) a low-rank factorization for stable and regularized training, and 3) a simple end-to-end training objective. From another perspective, this mechanism constitutes a form of context-aware Parameter-Efficient Fine-Tuning (PEFT). Extensive experiments show that HiFICL consistently outperforms existing approximation methods on several multimodal benchmarks. The code is available at https://github.com/bbbandari/HiFICL.
Xiaoyu Li, Yuhang Liu, Xuanshuo Kang +4
University of Electronic Science and Technology of China · Institute of Electronics and Information Industry Technology of Kashgar
Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks. Despite its flexibility, multimodal ICL incurs high inference latency and suffers from instability due to sensitivity to demonstration formatting, ordering, and content. To address these limitations, we propose Hyper-ICL, a lightweight, training-based framework for demonstration-free multimodal ICL that reconstructs demonstration effects directly without requiring ICDs at inference time. Hyper-ICL learns a parameter-efficient low-rank logit-level adapter that calibrates attention distributions to better match demonstration-induced attention redistribution. To capture how demonstration influence varies across queries, we introduce a query-adaptive modulation mechanism that adaptively controls intervention strength at token level across layers and heads based on the current query. Finally, we propose a layer-wise hyperbolic anchor distillation loss that aligns intermediate student features to a demonstration-conditioned teacher via Lorentz geodesic distance. This loss encourages the student to reconstruct the demonstration-query relationships induced by ICDs. Extensive experiments across six different multimodal benchmarks (including VQAv2, OK-VQA, and COCO Caption) demonstrate that Hyper-ICL consistently improves accuracy and stability over vanilla ICL and existing state-of-the-art methods.