Transformer-based large language models exhibit in-context learning, enabling adaptation to downstream tasks via few-shot prompting with demonstrations. In practice, such models are often fine-tuned to improve zero-shot performance on downstream tasks, allowing them to solve tasks without examples and thereby reducing inference costs. However, fine-tuning can degrade in-context learning, limiting the performance of fine-tuned models on tasks not seen during fine-tuning. Using linear attention models, we provide a theoretical analysis that characterizes how fine-tuning objectives modify attention parameters and identifies conditions under which this leads to degraded few-shot performance. We show that fine-tuning all attention parameters can harm in-context learning, whereas restricting updates to the value matrix improves zero-shot performance while preserving in-context learning. We further show that incorporating an auxiliary few-shot loss enhances in-context learning primarily on the target task, at the expense of degraded in-context learning ability on tasks not seen during fine-tuning. We provide empirical evidence from synthetic and real-world datasets consistent with the qualitative predictions of our theory.
Figures & tables
Section
Training Regime
Zero-shot error on θ0
Few-shot error on θ0
Few-shot error on −θ0
Results
(in-distribution)
(in-distribution)
(out-of-distribution)
§ 4.1
Pretraining
σ2+θ0⊤Σθ0
σ2
σ2
Corollary 4.1
( worst )
( best )
( best )
Corollary 4.2
§ 4.2
Fine-tuning all parameters
σ2
σ2+θ0⊤Σθ0
σ2+θ0⊤Σθ0
Theorem 4.3
with the zero-shot loss
( best )
( worst )
( worst )
Corollary 4.4
§ 4.3
Fine-tuning the value matrix
σ2+d+42θ0⊤Σθ0
σ2+(d+4)21θ0⊤Σθ0
σ2+(d+4)21θ0⊤Σθ0
Theorem 4.6
Table 1: Summary of our theoretical results in Section 4 . We consider four training regimes and report the test error (Definition 3.1 ) evaluated at the optimal parameters that minimize the training objective for each regime. Fine-tuning is performed on data generated from the target task vector θ0∈Rd . Zero-shot errors are evaluated on the in-distribution task θ0 used for fine-tuning, whereas asymptotic few-shot errors ( m,n→∞ ) are evaluated on both θ0 and the out-of-distribution task −θ0 . The test error ranges from σ2 ( best ) to σ2+θ0⊤Σθ0 ( worst ). The tags ( best ), ( better ), ( worse ), and ( worst ) indicate the relative ordering of test errors across regimes, with comparisons made separately within the zero-shot and asymptotic few-shot settings. Although the asymptotic few-shot test errors are smaller than the zero-shot error in some cases, this advantage relies on the assumption of infinitely many shots. In practical settings with a limited number of shots, fine-tuned models can achieve better in-distribution zero-shot performance than few-shot methods. The last column references the corresponding theoretical results for each training regime, including the analysis of few-shot errors with finite shots.
Figure 1: n -shot test error on the target task θ0 of the pretrained model using the theory-derived parameters in Corollary 4.1 . Each curve corresponds to a model pretrained with context length m=1000 and a different input dimension d . We set σ2=0 , under which the condition in ( 14 ) predicts that the n -shot performance is worse than zero-shot performance for n ranging from 1 to d−2 .
Figure 2: n -shot test error on the target task θ0 of the fully fine-tuned model using the theory-derived parameters in Theorem 4.3 . Each curve corresponds to a model with a different choice of the parameter w , evaluated under σ2=0.1 and θ0⊤Σθ0=1 . For all values of w , the zero-shot error is 0.1 , whereas the few-shot error is larger than the zero-shot error and converges to 1 as n→∞ .
Figure 3: (Top) n -shot test error on the target task θ0 for the value-matrix fine-tuned model with the theory-derived parameters (V^(w),Q^) in Theorem 4.6 . Each curve corresponds to a different choice of w . We use d=5 , θ0⊤Σθ0=1 , and σ2=0.1 , for which the zero-shot test error equals 3.1(=σ2+d+42θ0⊤Σθ0) for all models. (Bottom) Optimal w⋆(n;θ0) that minimizes the n -shot test error on the target task θ0 . The optimal value increases monotonically with n and converges to 8/9(=d+4d+3) . For example, the 20 -shot test error is minimized at w≈0.66 .
Figure 4: n -shot test errors on the target task θ0 for linear attention models under different training regimes. Curves show theoretical predictions from Section 4 , while points correspond to models trained empirically in Section 5.1 . Empirical results match the theoretical values. The parameters used for the theoretical predictions are given by: (a) ( 10 ) from Corollary 4.1 . (b) ( 15 ) from Theorem 4.3 with w=0.52 . (c) ( 21 ) from Theorem 4.6 with w=0.77(=m+1+dm) (Proposition 4.7 ). (d) ( 21 ) with w=0.66 (Theorem 4.9 ; see Figure 3 ).
Figure 5: n -shot test errors evaluated on the in-distribution task θ0 (Left) and the out-of-distribution task −θ0 (Right) . Empty circles denote models fine-tuned with the zero-shot (ZS) loss only, whereas filled circles denote models fine-tuned with both the zero-shot and few-shot (FS) losses. Parentheses indicate which parameters are updated: all updates all parameters, Q updates the query-key matrix, and V updates the value matrix. Value-matrix fine-tuning without the auxiliary few-shot loss gives the lowest error on the out-of-distribution task −θ0 .
Figure 6: Zero-shot (x-axis) and few-shot (y-axis) accuracy of Qwen2.5-3B-Instruct on the Humanities and STEM categories of the MMLU benchmark. We compare the pretrained model with three models fine-tuned to improve zero-shot performance on Humanities . (a) The pretrained model exhibits higher few-shot than zero-shot accuracy on Humanities . (b) Fine-tuning with the zero-shot loss improves zero-shot accuracy relative to the pretrained model but degrades few-shot accuracy. (c) Value-matrix fine-tuning with the zero-shot loss preserves few-shot accuracy while achieving zero-shot accuracy comparable to (b) . (d) Value-matrix fine-tuning with both zero-shot and few-shot losses improves few-shot accuracy on Humanities , at the cost of reduced few-shot accuracy on other categories such as STEM .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Test errors on (Left) the in-distribution task θ0 and (Right) the out-of-distribution task −θ0 , using a 1-head attention model.
Figure 8: Test errors on (Left) the in-distribution task θ0 and (Right) the out-of-distribution task −θ0 , using a 2-head attention model.
0-shot accuracies
7-shot accuracies
Model
Humanities
Social Sciences
STEM
Other
Humanities
Social Sciences
STEM
Other
Base model
63.62 (0.11)
70.12 (0.14)
70.69 (0.17)
63.52 (0.12)
70.13 (0.15)
75.33 (0.12)
71.77 (0.17)
64.06 (0.07)
ZS fine-tuning on Q/K/V
69.30 (0.16)
75.15 (0.14)
68.27 (0.20)
63.65 (0.09)
66.06 (0.06)
73.26 (0.19)
66.72 (0.11)
62.91 (0.11)
+5.69
+5.03
-2.42
+0.13
-4.07
-2.06
-5.05
-1.15
ZS fine-tuning on Q
66.05 (0.11)
71.77 (0.36)
67.42 (0.38)
60.83 (0.13)
68.57 (0.14)
73.13 (0.16)
71.17 (0.27)
64.01 (0.23)
+2.44
+1.65
-3.28
-2.69
-1.55
-2.19
-0.60
-0.05
Appendix
Table 2: Zero-shot and 7-shot accuracies evaluated on four MMLU categories ( Humanities , Social Sciences , STEM , and Other ). We fine-tune the base model ( Qwen2.5-3B-Instruct ) on the Humanities subset. ZS and FS denote fine-tuning with the zero-shot objective and the auxiliary few-shot objective, respectively, and Q, K, and V indicate that LoRA adapters are applied to the query, key, and value matrices, respectively. The second row for each model reports the difference from the base model in percentage points.
In-context learning (ICL) derives its power from enabling Large Language Models to adapt to new tasks via prompt-based reasoning alone, entirely bypassing the need for parameter updates. Existing theories primarily study ICL in single-task settings, while real-world prompts often contain sequences of heterogeneous tasks, leaving a gap in understanding whether Large Language Models implicitly perform continual learning during inference. To bridge this gap, we propose the first theoretical framework for in-context continual learning, modeling how a pretrained Transformer processes multiple sequential tasks within a single prompt through shared attention mechanisms. Focusing on linear and masked linear self-attention, we derive error expressions for model predictions under sequential task prompts and analyze their generalization and forgetting behavior. Our results reveal that standard attention mechanisms inevitably induce intertask interference by uniformly or causally aggregating historical contexts, leading to systematic bias. We further provide a bias-variance-interference decomposition of prediction error, characterizing when historical in-context information yields positive transfer or provable negative transfer. This analysis exposes fundamental limits of attention-based continual inference and offers theoretical explanations for order sensitivity and performance degradation in long prompts.
Guangyu Li, Meng Ding, Lijie Hu
1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · University of Buffalo
In-context fine-tuning (IC-Train), training an LLM with labeled examples in-context, is increasingly used in place of standard fine-tuning for domain adaptation and continual absorption of labeled data. We study the robustness of the in-context learning ability that emerges from such training: does the fine-tuned model perform well across test inputs whose in-context examples range from unrelated to nearly identical? Across 32 configurations spanning four open-source LLMs and eight test sets over machine translation, Text-to-SQL, and multilingual semantic parsing, we show that robustness hinges on an overlooked design choice: how in-context examples are selected relative to the target during training. The two prevailing strategies turn out to be accurate over complementary parts of this spectrum: random contexts yield a model that gains little from related examples even when they are placed in its context, while retrieved similar contexts weaken accuracy on targets lacking close neighbors and raise the propensity to copy labels from context. Probes tracking in-weights learning, in-context learning, and copying trace these failures to distinct training dynamics, and show that introducing contrast in target-context similarity both within a context and across batches, restores robustness across the entire spectrum.
Linear attention is a tractable model for understanding the mechanisms governing in-context learning in transformers. For linear regression tasks, recent asymptotic analyses have characterised its learning and generalisation behaviour. We extend this theory to nonlinear single-index targets, y=f(x⊤w)+ε. Our main result establishes a nonlinearity-noise equivalence: linear attention extracts only the linear Hermite component of f, while the remaining nonlinear structure contributes to the generalisation error as effective noise. This reduction allows results from the corresponding linear theory to be transferred to nonlinear tasks. We illustrate its implications for finite pretraining data and for the transition from task memorisation to task generalisation as task diversity increases. These results identify a limitation of the reduced linear-attention model and provide a tractable starting point for studying nonlinear in-context learning.
Mary Letey, Arman Rysmakhanov, Yue M. Lu +2
Applied Mathematics, Harvard University Kempner Institute, Harvard University · Williams College · Center for Brain Science, Harvard University Society of Fellows, Harvard University