Recent progress has rapidly advanced our understanding of the mechanisms underlying in-context learning in modern attention-based neural networks. However, existing results focus exclusively on unimodal data; in contrast, the theoretical underpinnings of in-context learning for multi-modal data remain poorly understood. We introduce a mathematically tractable framework for studying multi-modal learning and explore when transformer-like architectures can recover Bayes-optimal performance in-context. To model multi-modal problems, we assume the observed data arises from a latent factor model. Our first result comprises a negative take on expressibility: we prove that single-layer self-attention models fail to recover the Bayes-optimal predictor uniformly over the task distribution. To address this limitation, we introduce a novel, linearized cross-attention mechanism, which we study in the regime where both the number of cross-attention layers and the context length are large. We show that this cross-attention mechanism is provably Bayes optimal when optimized using gradient flow. Our results underscore the benefits of depth for in-context learning and establish the provable utility of cross-attention for multi-modal distributions.
Figures & tables
Figure 1: Re-injection of covariates X throughout the CAT ( LCAT ) embeddings. The propagation of the raw data X occurs both through St as well as through the CA block At .
Figure 2: Prediction error for the one- and two-parameter LCA-based models and the LSA model. Left: ICL error at fixed depth for varying Lte . Right: ICL error at fixed Lte for varying depth T .
Figure 3: Diagnostics for less-restricted LCA parametrizations (setup same as in Figure 2 ; see Appendix I ). Left : layerwise scalar projections; Right : whitening error across layers.
Model
Params
Best test acc. (%)
Best epoch
Full paper-inspired model (Inj. & cross-attn)
4.36×104
63.85
64
No injection
3.82×104
62.72
65
No cross-attn
4.36×104
62.10
79
No inj. & no cross-attn
3.82×104
60.79
90
Standard bi-CA transformer
6.07×104
62.72
62
Table 1: Classification results on avMNIST for the full paper-inspired model, three ablations thereof, and a standard bi-directional cross-attention transformer. All models use (pre) layer-norm, 4 -headed multi-head attention, 10 attention layers, embedding dim. d=16 , hidden dim. dh=32 , dropout rate of 0.1 , GELU MLP blocks, and an appended CLS token. Training: cross-entropy loss over 100 epochs using AdamW optimizer with learning rate 3×10−4 and weight decay 10−4 . Model parameter counts and best test accuracy with corresponding epoch are shown.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Log-scale heat maps of the two-parameter excess loss ℓ(α,β) with d1=d2=16 and ∥m∥∼Unif(0,2) . ( Left ) depth T=10 . ( Middle ) depth T=5 . ( Right ) depth T=5 with pre-LayerNorm added between the CA layers. The dashed line marks α=−β over β∈(−2/(m+1),0) . Each panel carries its own colour scale, as the three surfaces span different magnitudes.
Figure 5: In-context performance at various Lte ( T=10 , d1=d2=16 ). ( Left ) one- and two-parameter LCA-based models, the two ablations without St , and the sample mean yˉLte . ( Right ) one- and two-parameter LCA-based models, the two-parameter deep LSA model, the LSA model from ( 4.2 ), and the sample mean. Models are optimized on ℓN,Ltr ( Ltr=100,N=2000 ) using gradient descent. Performance is averaged over 1000 test-prompts where error bars represent standard deviation over 10 separate training runs. The sweep starts at Lte=128 , since for Lte comparable to d the sample covariance fluctuates enough that the depth- T recurrence leaves its stable region on a small fraction of prompts.
In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. Despite its success in large language models, the extension of ICL to multimodal settings remains poorly understood in terms of its internal mechanisms and how it differs from text-only ICL. In this work, we conduct a systematic analysis of ICL in multimodal large language models. Using identical task formulations across modalities, we show that multimodal ICL performs comparably to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations. To understand this gap, we decompose multimodal ICL into task mapping construction and task mapping transfer, and analyze how models establish cross-modal task mappings, and transfer them to query samples across layers. Our analysis reveals that current models lack reasoning-level alignment between visual and textual representations, and fail to reliably transfer learned task mappings to queries. Guided by these findings, we further propose a simple inference-stage enhancement method that reinforces task mapping transfer. Our results provide new insights into the mechanisms and limitations of multimodal ICL and suggest directions for more effective multimodal adaptation. Our code is available here.
Yu Wang, Sharon Li
Department of Computer Sciences, University of Wisconsin-Madison
Modern sequence models have a striking capacity for in-context learning (ICL); they can perform new tasks based only on examples given in the prompt. Understanding how this ability emerges requires theory that captures important properties of natural data. Linear regression has served as a useful sandbox for ICL theory, but existing work has largely focused on prompts with independent examples. In this work, we extend this setting to sequentially correlated data, a basic feature of real sequences. We present a solvable model based on linear attention and test our predictions on realistic transformer architectures. We identify two distinct effects: First, when the query token is independent of the context, within-context correlations induce an effective context length: correlated prompts behave like shorter i.i.d. prompts. Second, when the query is also correlated with its context, test error is reduced, particularly for softmax attention when compared to linear attention. These results suggest that correlated prompts alter not only the effective sample size of in-context learning, but also which attention architectures are best matched to the task.
Mary Letey, Yue M. Lu, Cengiz Pehlevan +1
John A. Paulson School of Engineering and Applied Sciences, Harvard University · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Society of Fellows and Center for Brain Science, Harvard University
In-context learning (ICL) enables fast task adaptation from demonstrations without per-task parameter updates but remains highly sensitive to example selection and formatting. In unified multimodal models spanning understanding and generation, this sensitivity is exacerbated by cross-modal interference and varying cognitive demands. Consequently, in-context learning efficacy is often non-monotonic and highly task-dependent. To diagnose these behaviors, we introduce a six-level Capability-Oriented Taxonomy that categorizes the functional role of demonstrations from basic perception to high-order discernment. Guided by this cognitive framework, we construct UniICL-760K, a large-scale corpus featuring curated 8-shot in-context learning episodes across 15 subtasks, alongside UniICL-Bench for rigorous, controlled evaluation. We show that this data-driven assembly is the primary source of our gains. As a complementary, lightweight stabilizer, we additionally propose the Context-Adaptive Prototype Modulator, a plug-and-play module that further improves few-shot stability. Evaluations on UniICL-Bench show that our approach yields highly competitive unified results, outperforming larger-parameter multimodal large language model baselines on most understanding in-context learning tasks. Data and code are available at https://github.com/xuyicheng-zju/UniICL.
Yicheng Xu, Jiangning Zhang, Zhucun Xue +5
1Zhejiang University, China. · 2Shanghai Jiaotong University, China. · 3National University of Singapore, Singapore. +1