Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models

Chungpa Lee; Jy-yong Sohn; Kangwook Lee

cs.CLFeb 26, 2026

Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models

Authors: Chungpa Lee, Jy-yong Sohn, Kangwook Lee

Organizations: Yonsei University, Korea · University of Wisconsin–Madison, USA · KRAFTON, Korea · Ludo Robotics, Korea

Abstract

Transformer-based large language models exhibit in-context learning, enabling adaptation to downstream tasks via few-shot prompting with demonstrations. In practice, such models are often fine-tuned to improve zero-shot performance on downstream tasks, allowing them to solve tasks without examples and thereby reducing inference costs. However, fine-tuning can degrade in-context learning, limiting the performance of fine-tuned models on tasks not seen during fine-tuning. Using linear attention models, we provide a theoretical analysis that characterizes how fine-tuning objectives modify attention parameters and identifies conditions under which this leads to degraded few-shot performance. We show that fine-tuning all attention parameters can harm in-context learning, whereas restricting updates to the value matrix improves zero-shot performance while preserving in-context learning. We further show that incorporating an auxiliary few-shot loss enhances in-context learning primarily on the target task, at the expense of degraded in-context learning ability on tasks not seen during fine-tuning. We provide empirical evidence from synthetic and real-world datasets consistent with the qualitative predictions of our theory.

Figures & tables

Section	Training Regime	Zero-shot error on $\theta_{0}$	Few-shot error on $\theta_{0}$	Few-shot error on $-\theta_{0}$	Results
		(in-distribution)	(in-distribution)	(out-of-distribution)
§ 4.1	Pretraining	$\sigma^{2}+{\mathbf{\theta}}_{0}^{\top}{\mathbf{\Sigma}}{\mathbf{\theta}}_{0}$	$\sigma^{2}$	$\sigma^{2}$	Corollary 4.1
		( worst )	( best )	( best )	Corollary 4.2
§ 4.2	Fine-tuning all parameters	$\sigma^{2}$	$\sigma^{2}+{\mathbf{\theta}}_{0}^{\top}{\mathbf{\Sigma}}{\mathbf{\theta}}_{0}$	$\sigma^{2}+{\mathbf{\theta}}_{0}^{\top}{\mathbf{\Sigma}}{\mathbf{\theta}}_{0}$	Theorem 4.3
	with the zero-shot loss	( best )	( worst )	( worst )	Corollary 4.4
§ 4.3	Fine-tuning the value matrix	$\sigma^{2}+\frac{2}{d+4}{\mathbf{\theta}}_{0}^{\top}{\mathbf{\Sigma}}{\mathbf{\theta}}_{0}$	$\sigma^{2}+\frac{1}{(d+4)^{2}}{\mathbf{\theta}}_{0}^{\top}{\mathbf{\Sigma}}{\mathbf{\theta}}_{0}$	$\sigma^{2}+\frac{1}{(d+4)^{2}}{\mathbf{\theta}}_{0}^{\top}{\mathbf{\Sigma}}{\mathbf{\theta}}_{0}$	Theorem 4.6

Table 1: Summary of our theoretical results in Section 4 . We consider four training regimes and report the test error (Definition 3.1 ) evaluated at the optimal parameters that minimize the training objective for each regime. Fine-tuning is performed on data generated from the target task vector

{\mathbf{\theta}}_{0}\in\mathbb{R}^{d}

. Zero-shot errors are evaluated on the in-distribution task

{\mathbf{\theta}}_{0}

used for fine-tuning, whereas asymptotic few-shot errors (

m,n\to\infty

) are evaluated on both

{\mathbf{\theta}}_{0}

and the out-of-distribution task

-{\mathbf{\theta}}_{0}

. The test error ranges from

\sigma^{2}

( best ) to

\sigma^{2}+{\mathbf{\theta}}_{0}^{\top}{\mathbf{\Sigma}}{\mathbf{\theta}}_{0}

( worst ). The tags ( best ), ( better ), ( worse ), and ( worst ) indicate the relative ordering of test errors across regimes, with comparisons made separately within the zero-shot and asymptotic few-shot settings. Although the asymptotic few-shot test errors are smaller than the zero-shot error in some cases, this advantage relies on the assumption of infinitely many shots. In practical settings with a limited number of shots, fine-tuned models can achieve better in-distribution zero-shot performance than few-shot methods. The last column references the corresponding theoretical results for each training regime, including the analysis of few-shot errors with finite shots.

Figure 1:

n

-shot test error on the target task

{\mathbf{\theta}}_{0}

of the pretrained model using the theory-derived parameters in Corollary 4.1 . Each curve corresponds to a model pretrained with context length

m=1000

and a different input dimension

d

. We set

\sigma^{2}=0

, under which the condition in ( 14 ) predicts that the

n

-shot performance is worse than zero-shot performance for

n

ranging from

1

d-2

Figure 2:

n

-shot test error on the target task

{\mathbf{\theta}}_{0}

of the fully fine-tuned model using the theory-derived parameters in Theorem 4.3 . Each curve corresponds to a model with a different choice of the parameter

w

, evaluated under

\sigma^{2}=0.1

and

{\mathbf{\theta}}_{0}^{\top}{\mathbf{\Sigma}}{\mathbf{\theta}}_{0}=1

. For all values of

w

, the zero-shot error is

0.1

, whereas the few-shot error is larger than the zero-shot error and converges to

1

n\to\infty

Figure 3: (Top)

n

-shot test error on the target task

{\mathbf{\theta}}_{0}

for the value-matrix fine-tuned model with the theory-derived parameters

\big(\hat{{\mathbf{V}}}(w),\hat{{\mathbf{Q}}}\big)

in Theorem 4.6 . Each curve corresponds to a different choice of

w

. We use

d=5

{\mathbf{\theta}}_{0}^{\top}{\mathbf{\Sigma}}{\mathbf{\theta}}_{0}=1

, and

\sigma^{2}=0.1

, for which the zero-shot test error equals

3.1\,(=\sigma^{2}+\tfrac{2}{d+4}{\mathbf{\theta}}_{0}^{\top}{\mathbf{\Sigma}}{\mathbf{\theta}}_{0})

for all models. (Bottom) Optimal

w^{\star}(n;{\mathbf{\theta}}_{0})

that minimizes the

n

-shot test error on the target task

{\mathbf{\theta}}_{0}

. The optimal value increases monotonically with

n

and converges to

8/9\,(=\tfrac{d+3}{d+4})

. For example, the

20

-shot test error is minimized at

w\approx 0.66

Panel of Figure 4 — Figure 4: $n$ -shot test errors on the target task ${\mathbf{\theta}}_{0}$ for linear attention models under different training regimes. Curves show theoretical predictions from Section 4 , while points correspond to models trained empirically in Section 5.1 . Empirical results match the theoretical values. The parameters used for the theoretical predictions are given by: (a) ( 10 ) from Corollary 4.1 . (b) ( 15 ) from Theorem 4.3 with $w=0.52$ . (c) ( 21 ) from Theorem 4.6 with $w=0.77\;(=\tfrac{m}{m+1+d})$ (Proposition 4.7 ). (d) ( 21 ) with $w=0.66$ (Theorem 4.9 ; see Figure 3 ).

Figure 5:

n

-shot test errors evaluated on the in-distribution task

{\mathbf{\theta}}_{0}

(Left) and the out-of-distribution task

-{\mathbf{\theta}}_{0}

(Right) . Empty circles denote models fine-tuned with the zero-shot (ZS) loss only, whereas filled circles denote models fine-tuned with both the zero-shot and few-shot (FS) losses. Parentheses indicate which parameters are updated: all updates all parameters, Q updates the query-key matrix, and V updates the value matrix. Value-matrix fine-tuning without the auxiliary few-shot loss gives the lowest error on the out-of-distribution task

-{\mathbf{\theta}}_{0}

Figure 6: Zero-shot (x-axis) and few-shot (y-axis) accuracy of Qwen2.5-3B-Instruct on the Humanities and STEM categories of the MMLU benchmark. We compare the pretrained model with three models fine-tuned to improve zero-shot performance on Humanities . (a) The pretrained model exhibits higher few-shot than zero-shot accuracy on Humanities . (b) Fine-tuning with the zero-shot loss improves zero-shot accuracy relative to the pretrained model but degrades few-shot accuracy. (c) Value-matrix fine-tuning with the zero-shot loss preserves few-shot accuracy while achieving zero-shot accuracy comparable to (b) . (d) Value-matrix fine-tuning with both zero-shot and few-shot losses improves few-shot accuracy on Humanities , at the cost of reduced few-shot accuracy on other categories such as STEM .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Test errors on (Left) the in-distribution task

{\mathbf{\theta}}_{0}

and (Right) the out-of-distribution task

-{\mathbf{\theta}}_{0}

, using a 1-head attention model.

Figure 8: Test errors on (Left) the in-distribution task

{\mathbf{\theta}}_{0}

and (Right) the out-of-distribution task

-{\mathbf{\theta}}_{0}

, using a 2-head attention model.

	0-shot accuracies				7-shot accuracies
Model	Humanities	Social Sciences	STEM	Other	Humanities	Social Sciences	STEM	Other
Base model	63.62 (0.11)	70.12 (0.14)	70.69 (0.17)	63.52 (0.12)	70.13 (0.15)	75.33 (0.12)	71.77 (0.17)	64.06 (0.07)
ZS fine-tuning on Q/K/V	69.30 (0.16)	75.15 (0.14)	68.27 (0.20)	63.65 (0.09)	66.06 (0.06)	73.26 (0.19)	66.72 (0.11)	62.91 (0.11)
	+5.69	+5.03	-2.42	+0.13	-4.07	-2.06	-5.05	-1.15
ZS fine-tuning on Q	66.05 (0.11)	71.77 (0.36)	67.42 (0.38)	60.83 (0.13)	68.57 (0.14)	73.13 (0.16)	71.17 (0.27)	64.01 (0.23)
	+2.44	+1.65	-3.28	-2.69	-1.55	-2.19	-0.60	-0.05

Appendix

Table 2: Zero-shot and 7-shot accuracies evaluated on four MMLU categories ( Humanities , Social Sciences , STEM , and Other ). We fine-tune the base model ( Qwen2.5-3B-Instruct ) on the Humanities subset. ZS and FS denote fine-tuning with the zero-shot objective and the auxiliary few-shot objective, respectively, and Q, K, and V indicate that LoRA adapters are applied to the query, key, and value matrices, respectively. The second row for each model reports the difference from the base model in percentage points.

Explore similar work

Understanding Generalization and Forgetting in In-Context Continual Learning
May 27, 2026Guangyu Li, Meng Ding, Lijie HuContinual Learning for LLMs In-Context Learning
1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · University of Buffalo
Contrast Matters: Understanding Robustness of In-Context Fine-Tuning to Target-Context Relatedness
Apr 2, 2026Deeptanshu Malu, Deevyanshu Malu, Aditya Nemiwal +1In-Context Example Selection LLM Fine-Tuning
IIT Bombay
What can linear attention learn from nonlinear teachers in-context?
Oct 7, 2026Mary Letey, Arman Rysmakhanov, Yue M. Lu +2In-Context Learning Linear Attention
Applied Mathematics, Harvard University Kempner Institute, Harvard University · Williams College · Center for Brain Science, Harvard University Society of Fellows, Harvard University

May 27, 2026J/K move · Enter open · S save

Understanding Generalization and Forgetting in In-Context Continual Learning

Guangyu Li, Meng Ding, Lijie Hu

1Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · University of Buffalo

Continual Learning for LLMs In-Context Learning

Open paper AskPDF arXiv

In-context learning (ICL) derives its power from enabling Large Language Models to adapt to new tasks via prompt-based reasoning alone, entirely bypassing the need for parameter updates. Existing theories primarily study ICL in single-task settings, while real-world prompts often contain sequences of heterogeneous tasks, leaving a gap in understanding whether Large Language Models implicitly perform continual learning during inference. To bridge this gap, we propose the first theoretical framework for in-context continual learning, modeling how a pretrained Transformer processes multiple sequential tasks within a single prompt through shared attention mechanisms. Focusing on linear and masked linear self-attention, we derive error expressions for model predictions under sequential task prompts and analyze their generalization and forgetting behavior. Our results reveal that standard attention mechanisms inevitably induce intertask interference by uniformly or causally aggregating historical contexts, leading to systematic bias. We further provide a bias-variance-interference decomposition of prediction error, characterizing when historical in-context information yields positive transfer or provable negative transfer. This analysis exposes fundamental limits of attention-based continual inference and offers theoretical explanations for order sensitivity and performance degradation in long prompts.

Abstract

Explore similar work

Understanding Generalization and Forgetting in In-Context Continual Learning

Contrast Matters: Understanding Robustness of In-Context Fine-Tuning to Target-Context Relatedness

What can linear attention learn from nonlinear teachers in-context?