Convergent Emergence of In-Context Learning Across Modalities

Nathan Breslow; Seungwook Han; Daniel Hyunsoo Lee; Aayush Mishra; Anqi Liu; Daniel Khashabi

cs.AISep 12, 2026

Convergent Emergence of In-Context Learning Across Modalities

Authors: Nathan Breslow, Seungwook Han, Daniel Hyunsoo Lee, Aayush Mishra, Anqi Liu, Daniel Khashabi

Organizations: Johns Hopkins University · MIT · University of Illinois Urbana-Champaign

Abstract

Few-shot in-context learning (ICL), the capacity of a model to infer abstract patterns from input-output examples provided in its prompt and apply them to new inputs, has been extensively studied in large language models trained for next-token prediction on human text. Recently, few-shot ICL has been demonstrated in autoregressive genomic models as well. This raises a question: does ICL emerge broadly across domains, and if so, what common structure is shared? To address both, we develop a controlled cross-modality framework that instantiates the same task suite in a variety of modalities to test what we call the Convergent Emergence Hypothesis: the idea that few-shot ICL, when it emerges, shares a common cross-modality difficulty profile - i.e., tasks that benefit from ICL in one modality tend to benefit in others. We show that paired-mapping ICL emerges across six modalities (language, genome, integer sequences, time series, images, and proteins), surpasses controlled baselines, and has correlated per-task effects across five of them. Together, these results provide support for the Convergent Emergence Hypothesis in some modalities, but not all.

Figures & tables

Panel of Figure 1 — Figure 1: Two hypotheses about how ICL task sets relate across modalities. (a) Disjoint sets imply that ICL is modality-specific, shaped by the idiosyncrasies of each training corpus; (b) a shared intersection implies a common substrate on which the same abstract tasks are easy or hard everywhere. Each colored circle denotes one modality, with colors held fixed across panels.

Panel of Figure 2 — Figure 2: Left: Evaluating ICL in different modalities on a shared task suite (detailed in Figure 3 ). Right: Pairwise Spearman correlation of per-task clean-minus-deranged accuracy (Eq. 4 ) across modalities, using the representative model for each modality at its maximum shot count and $T=128$ trials per task and condition (§ 4.1 ). Hatched off-diagonal cells have one-sided permutation $p\geq 0.05$ . The ordering places the shared five-modality block in the upper left. ICL in different modalities are positively correlated across five of the six modalities, indicating a large degree of shared ICL structure.

Figure 3: The cross-modality construction. The same abstract $8$ -bit task ( $f:\{0,1\}^{8}\to\{0,1\}^{8}$ ; here the identity, $f(x)=x$ ) is instantiated in every modality through its encoding map $\phi_{M}$ . Each discrete-symbol modality draws disjoint symbols for the two bits and for the demonstration separator $\psi_{M}$ (occasionally omitted for brevity), re-sampled every trial (one representative draw is shown): digits for Language and Integer sequence, nucleotides for Genome, amino acids for Protein, and color-cluster pixels for Image. Time series instead uses fixed signed half-sine sub-lobes ( $\pm s_{16}$ ) and all-zero separator patches. The resulting few-shot prompt (in-context exemplars followed by a held-out query whose output the model must produce) is fed to the corresponding decoder-only model. Language, Genome, Protein, and Image concatenate input and output directly, while Integer sequence uses literal comma separators (§ 3.2 ).

Single primitives
Function $f$	Input $I$	Output $f(I)$
identity	10110101	10110101
reverse_bits	10110101	10101101
rotl1	10110101	01101011
swap_halves	10110101	01011011
majority	10110101	11111111

Table 1: Twelve representative tasks from the 100-function suite, each instantiated on the same input to illustrate the diversity of transformations. See Appendix E for the full list.

Model	Shot counts $N$
Default	$\{1,2,4,8,16,32,64\}$
TimesFM	$\{1,2,4,8,16,24,32,48\}$
ProGen2	$\{1,2,4,8,16,32,48\}$
ImageGPT	$\{1,2,4,8,16,31\}$

Table 2: Shot-count sweeps per model.

Figure 4: Clean versus deranged few-shot ICL curves across six modalities, evaluated on 100 tasks with

T=128

trials per task. Points show mean exact-match accuracy and error bars show one task-cluster standard error. Clean accuracy rises across each modality’s shot grid, whereas the deranged control remains substantially flatter. Shot grids differ where model context limits require shorter prompts.

Modality	Model	$n_{\max}$	Clean	Deranged	Paired gap (pp) [95% CI]	Holm-adjusted $p_{\rm task}$
			acc.%	acc.%
Language	Qwen3-14B	64	30.0%	16.1%	13.9 [10.8, 17.1]	$1.6\times 10^{-20}$
Genome	Evo2-40B	64	33.2%	15.2%	18.0 [15.0, 21.0]	$6.3\times 10^{-22}$
Integer seq.	NextTerm-440M	64	48.1%	16.7%	31.4 [25.6, 37.6]	$1.1\times 10^{-22}$
Image	ImageGPT-large	31	43.5%	18.3%	25.2 [19.5, 31.3]	$1.6\times 10^{-19}$
Time series	TimesFM-2.5	48	32.6%	14.0%	18.5 [15.1, 22.0]	$4.5\times 10^{-22}$

Table 3: Per-modality summary from the

T=128

evaluation. All quantities are evaluated at the modality’s maximum shot count

n_{\max}

. The paired gap is the mean of the 100 within-task clean-minus-deranged accuracy differences in percentage points, with a 95% task-bootstrap confidence interval. The final column reports the exact one-sided within-task condition-swap

p

-value after Holm adjustment across the six modalities.

Figure 5: Mean per-task clean-minus-deranged exact-match accuracy, averaged within the

k=7

semantic task clusters (Appendix I ), for each representative model at its maximum shot count. Each condition uses

T=128

trials per task. The radial axis retains the

0.01

–

1

logarithmic scale; effects at or below

0.01

are shown at the

0.01

floor.

Figure 6: Paired-mapping ICL generally strengthens with model scale. Clean-minus-deranged exact-match accuracy for each model at the shot count where that model’s clean accuracy is highest; deranged accuracy is evaluated at the same shot count. Each condition uses

T=128

trials per task, and error bars show the standard error across the 100 paired task differences. Effects increase monotonically for ImageGPT and NextTerm, rise through Qwen3-8B and then plateau, and peak at Evo2-7B and ProGen2- base .

Panel of Figure 7 — Figure 7: Chess and music controls with $T=128$ trials per task (axes matched). Chess shows no maximum-shot clean advantage (gap $=-0.0016$ , exact one-sided within-task condition-swap $p=0.640$ ). Musicroll’s earlier gaps are nonsignificant, but its $n=32$ gap is small and positive ( $0.0107$ , $p=0.00349$ ; Holm-adjusted across five tested shots, $p=0.0174$ ). Error bars are task-level standard errors. Contrast with Figure 4 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	Global Broadcast	Half Segments	Parity Masking	Half Swaps	Bit Inversion	Bit Reversal	Local Shifts
Language (Qwen3-14B)	spread_last_bit +center_mask, $n{=}64$ in 11110111 exp 01111110 out 0000000	right_half, $n{=}64$ in 00001011 exp 00001011 out 00000010	shift_left_zero +swap_pairs, $n{=}64$ in 11011010 exp 01111000 out 01001100	swap_halves +shift_left_zero, $n{=}64$ in 01111110 exp 11001110 out 10011010	flip_bits, $n{=}64$ in 11010111 exp 00101000 out 0000001	flip_bits +reverse_bits, $n{=}64$ in 00110001 exp 01110011 out 1110101	center_mask, $n{=}64$ in 11100001 exp 01100000 out 0110000
Genome (Evo2-40B)	minority, $n{=}64$ in 01010000 exp 11111111 out 00000000	invert_prefix, $n{=}64$ in 01101111 exp 10011111 out 10011110	alternating_start_zero, $n{=}64$ in 10010100 exp 11000001 out 10111110	swap_halves +reverse_bits, $n{=}64$ in 01010111 exp 10101110 out 10101010	double_rotl +flip_bits, $n{=}64$ in 10011000 exp 10011101 out 10011111	flip_bits +reverse_bits, $n{=}64$ in 11110010 exp 10110000 out 11100011	rotl1, $n{=}64$ in 00010011 exp 00100110 out 01101110
Integer seq. (NextTerm-440M)	shift_left_zero +parity_fill, $n{=}64$ in 11000000 exp 11111111 out 00000000	flip_bits +mirror_half, $n{=}64$ in 01111100 exp 10000001 out 10000010	alternating_start_one, $n{=}64$ in 10110110 exp 00011100 out 00010100	double_rotl +reverse_bits, $n{=}64$ in 01000110 exp 10011000 out 10000101	rotr1 +flip_bits, $n{=}64$ in 00011000 exp 11110011 out 11100101	reverse_bits, $n{=}64$ in 11010001 exp 10001011 out 01000011	shift_left_zero, $n{=}64$ in 00001110 exp 00011100 out 00011110
Image (ImageGPT-large)	shift_left_zero +parity_fill, $n{=}31$ in 00101000 exp 00000000 out 11111111	flip_bits +right_half, $n{=}31$ in 11100000 exp 00001111 out 00000000	shift_left_zero +swap_pairs, $n{=}31$ in 10101001 exp 10100001 out 00100000	shift_right_zero +swap_halves, $n{=}31$ in 11101101 exp 01100111 out 11100111	flip_bits, $n{=}2$ in 10011011 exp 01100100 out 01010110	reverse_bits +center_mask, $n{=}31$ in 01011011 exp 01011010 out 11011010	swap_pairs, $n{=}31$ in 10111110 exp 01111101 out 11111101
Time series (TimesFM 2.5)	spread_last_bit +flip_bits, $n{=}24$ in 01101000 exp 11111111 out 00000000	left_half +reverse_bits, $n{=}48$ in 10110001 exp 00001101 out 00000101	shift_right_zero +keep_even_positions, $n{=}48$ in 01000111 exp 00100010 out 00000010	shift_right_zero +swap_halves, $n{=}48$ in 11100111 exp 00110111 out 10111001	shift_left_zero +flip_bits, $n{=}48$ in 00110111 exp 10010001 out 10010101	reverse_bits +rotl1, $n{=}48$ in 00110111 exp 11011001 out 11010111	identity, $n{=}48$ in 11000100 exp 11000100 out 00000100
Protein (ProGen2-base)	parity_fill +flip_bits, $n{=}48$ in 00101110 exp 11111111 out 00000000	invert_prefix, $n{=}48$ in 00101101 exp 11011101 out 00110111	alternating_start_one, $n{=}48$ in 01110101 exp 11011111 out 00000000	swap_halves, $n{=}48$ in 00011101 exp 11010001 out 00010001	rotr1 +flip_bits, $n{=}48$ in 11001011 exp 00011010 out 01110110	reverse_bits +rotl1, $n{=}32$ in 01111000 exp 00111100 out 11111000	swap_pairs, $n{=}48$ in 11111011 exp 11110111 out 11111101

Appendix

Table 4: One randomly sampled mistake per modality per task family, taken from the representative model’s completions at its family-best clean shot count

n

. Rows: modalities (radar-chart models). Columns: the seven semantic task families. Each cell shows the sampled task, the input ( in ), the correct output ( exp ), and the model’s decoded prediction ( out ).

Primitive	Description
alternating_start_one	Produce a mask marking positions that differ from the alternating 1010… pattern starting with 1 (1 = mismatch, 0 = match).
alternating_start_zero	Produce a mask marking positions that differ from the alternating 0101… pattern starting with 0 (1 = mismatch, 0 = match).
center_mask	Zero out the first and last bits while leaving the interior bits unchanged; strings of length $\leq 2$ become all zeros.
double_rotl	Circularly rotate the bitstring two positions to the left.
double_rotr	Circularly rotate the bitstring two positions to the right.
edge_mask	Preserve the first and last bits and zero out every interior bit (length 0/1 strings pass through).

Appendix

Table 5: The 30 unary primitives used to construct functions in

F

Function	Function	Function
identity	flip_bits $\to$ left_half	flip_bits $\to$ keep_even_positions
rotl1	flip_bits $\to$ right_half	flip_bits $\to$ keep_odd_positions
reverse_bits	double_rotl $\to$ reverse_bits	edge_mask $\to$ flip_bits
flip_bits	rotl1 $\to$ swap_halves	center_mask $\to$ flip_bits
swap_halves	xor_with_s0	shift_left_zero $\to$ keep_even_positions
majority	flip_bits $\to$ xor_with_s0	shift_left_zero $\to$ keep_odd_positions

Appendix

Table 6: The complete set of 100 functions in

F

, consisting of 30 single primitives and 70 composed functions (

(f\to g)(x)=g(f(x))

Figure 8: Language ( $\phi_{\mathrm{lang}}$ , Qwen3). Each bit is one digit token ( $0\to 5$ , $1\to 6$ in this trial) and the separator is a distinct digit ( $\to 3$ ); input and output are concatenated with no delimiter.

Figure 9: Genome ( $\phi_{\mathrm{genome}}$ , Evo2). Each bit maps to a nucleotide from {A,C,G,T} , with a disjoint nucleotide as the separator $\psi_{\mathrm{genome}}$ .

Figure 10: Integer sequences ( $\phi_{\mathrm{integer\_seq}}$ ). Each bit is a digit $1$ – $9$ and the separator is a comma, inserted between demonstrations and between the input and output of each demonstration.

Figure 11: Protein ( $\phi_{\mathrm{protein}}$ , ProGen2). Each bit maps to a canonical amino acid, with a disjoint residue as the separator; the prompt is prefixed with ProGen2’s sequence-start token.

Figure 12: Image ( $\phi_{\mathrm{image}}$ , ImageGPT). Each demonstration is a $16$ -pixel horizontal span (8 pixels input, 8 pixels output) with a left-padded $16$ -pixel separator span, packing one demonstration into a single $32$ -token raster row; bits are rendered as color-cluster tokens.

Figure 13: Time series ( $\phi_{\mathrm{timeseries}}$ , TimesFM). Each bit is a signed $16$ -step half-sine sub-lobe ( $+s_{16}$ for 1, $-s_{16}$ for 0); two sub-lobes fill a $32$ -step patch, and all-zero patches serve as separators in the input–zero–output–zero layout. The dashed blue segment is the target forecast.

In-Context Learning

In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.

See all 404 papers

Apr 15, 2026

Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

In-context learning (ICL) enables models to adapt to new tasks via inference-time demonstrations. Despite its success in large language models, the extension of ICL to multimodal settings remains poorly understood in terms of its internal mechanisms and how it differs from text-only ICL. In this work, we conduct a systematic analysis of ICL in multimodal large language models. Using identical task formulations across modalities, we show that multimodal ICL performs comparably to text-only ICL in zero-shot settings but degrades significantly under few-shot demonstrations. To understand this gap, we decompose multimodal ICL into task mapping construction and task mapping transfer, and analyze how models establish cross-modal task mappings, and transfer them to query samples across layers. Our analysis reveals that current models lack reasoning-level alignment between visual and textual representations, and fail to reliably transfer learned task mappings to queries. Guided by these findings, we further propose a simple inference-stage enhancement method that reinforces task mapping transfer. Our results provide new insights into the mechanisms and limitations of multimodal ICL and suggest directions for more effective multimodal adaptation. Our code is available here.

Yu Wang, Sharon Li

Department of Computer Sciences, University of Wisconsin-Madison

Multimodal ICL In-Context Learning Multimodal Large Language Models Modality Gap in VLMs

Ask PDF

Mar 25, 2026

UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy

In-context learning (ICL) enables fast task adaptation from demonstrations without per-task parameter updates but remains highly sensitive to example selection and formatting. In unified multimodal models spanning understanding and generation, this sensitivity is exacerbated by cross-modal interference and varying cognitive demands. Consequently, in-context learning efficacy is often non-monotonic and highly task-dependent. To diagnose these behaviors, we introduce a six-level Capability-Oriented Taxonomy that categorizes the functional role of demonstrations from basic perception to high-order discernment. Guided by this cognitive framework, we construct UniICL-760K, a large-scale corpus featuring curated 8-shot in-context learning episodes across 15 subtasks, alongside UniICL-Bench for rigorous, controlled evaluation. We show that this data-driven assembly is the primary source of our gains. As a complementary, lightweight stabilizer, we additionally propose the Context-Adaptive Prototype Modulator, a plug-and-play module that further improves few-shot stability. Evaluations on UniICL-Bench show that our approach yields highly competitive unified results, outperforming larger-parameter multimodal large language model baselines on most understanding in-context learning tasks. Data and code are available at https://github.com/xuyicheng-zju/UniICL.

Yicheng Xu, Jiangning Zhang, Zhucun Xue +5

1Zhejiang University, China. · 2Shanghai Jiaotong University, China. · 3National University of Singapore, Singapore. +1

Multimodal ICL Unified Multimodal Models In-Context Learning Multimodal Foundation Models Multimodal Model Evaluation Multimodal Understanding

Ask PDF

Sep 30, 2026

Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning

In-context learning (ICL) can be amortized into latent objects (task vectors, function vectors, context vectors) that recover few-shot behavior at zero-shot inference cost, but recent theory shows a static vector acts as a single synthetic demonstration and must fail on high-rank mappings such as word-level bijections. We ask a linguistic version of this question: which linguistic operations can be amortized out of the prompt? We train a 2.6M-parameter network that reads the geometry of a few-shot support set (centroid, principal subspace, spectrum, computed once and cached) and produces an input-conditioned additive update to the query's residual stream at a mid-depth layer of a frozen GPT-2-large/XL. Across eight inflectional directions and one lexical relation, under a canonical split that bars inverted-pair leakage between directions, three regimes emerge. On forward inflection, where 10-shot ICL is strong (0.67-0.89) and extracted task vectors collapse (<=0.06), the transform matches ICL at strictly zero-shot per-query cost. On lemmatization directions, which frozen GPT-2 can execute but 10 demonstrations systematically fail to convey (ICL 0.13-0.48 at 1.5B), the transform is not capped by ICL at all: it reaches 0.78-0.92, up to +72 points over ICL (past to present: 0.85 vs. 0.13). On arbitrary pairings (antonymy) every amortizer plateaus near half of ICL at every scale, capacity, and seed tested. Controls show the support manifold acts as a causally necessary task fingerprint: wrong-task manifolds collapse accuracy to <=0.06, query-only variants cannot disambiguate tasks sharing an input space, and leave-one-task-out transfer is zero. Productive rules amortize into latent task representations, sometimes better than prompting can convey them; memorized pairings do not.

Gunmay Jhingran

Delhi Technological University

Representation Learning In-Context Learning Representation Geometry in Language Models

Ask PDF

Abstract

In-Context Learning

Explore similar work

Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks

UniICL: Systematizing Unified Multimodal In-context Learning through a Capability-Oriented Taxonomy

Rules Amortize, Pairings Don't: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning