Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

Arnav Chavan; Kusha Sareen; Rishabh Tiwari; Abhranil Chandra; Johan Obando-Ceron; Kurt Keutzer; Scott Niekum; Inderjit S Dhillon

cs.LGOct 15, 2026

Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

Authors: Arnav Chavan, Kusha Sareen, Rishabh Tiwari, Abhranil Chandra, Johan Obando-Ceron, Kurt Keutzer, Scott Niekum, Inderjit S Dhillon,

Organizations: UC Berkeley · Mila · McGill University · UMass Amherst ·

Abstract

Large language models (LLMs) acquire new skills through fine-tuning, but it can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to continual learning in LLMs. Recent works connect forgetting to how far, and along which directions, fine-tuning moves the weights. On-policy methods such as reinforcement learning and on-policy distillation avoid updating the principal singular directions of weight matrices and thus forget less. In contrast, SFT concentrates updates along these directions and is susceptible to catastrophic forgetting. In this work, we introduce Subspace Tuning (SST), which restricts each weight update to the bottom singular subspace, eliminating changes along the principal directions. SST is optimizer-agnostic and composes well with fine-tuning recipes including SFT and self-distillation. SST combines tight subspace protection, dense gradient estimation, and a simple refresh mechanism, enabling stronger learning than prior orthogonal fine-tuning methods. With these mechanisms, protecting as much as 99% of the weight subspace substantially improves retention while preserving strong target-task performance. This behavior strengthens with model and data size and nearly eliminates forgetting in the large-model regime. SST also keeps the model close to its base distribution, inducing $2$ -- $4\times$ less KL drift than the baseline recipes. Overall, SST advances the learning--forgetting Pareto frontier, making it a practical default for fine-tuning pre-trained language models.

Figures & tables

Figure 1: SST freezes the top of each weight matrix’s spectrum, and improves learning–forgetting trade-off across fine-tuning recipes. Left: the method. SST projects the update on low energy singular directions of each weight matrix. Right: the result on Qwen3-8B, averaged over four domains (Table 4 ). Both axes show changes from the base model in percentage points (pp); higher is better. Circles denote training without SST, stars denote training with SST, and the black triangle marks the base model.

Figure 2: Scaling model size. Qwen3 models from

1.7

B to

14

B fine-tuned on Science for five epochs. SFT forgets heavily at every scale, whereas forgetting under SST shrinks as models grow and nearly vanishes from

8

B onward. SST also reaches higher new-task accuracy at every size.

Figure 3: Scaling training data. Qwen3-8B fine-tuned on Science under a matched

45

k-example budget: the novel arms see a fresh stream once, the repeated arms cycle

9

k examples for five epochs. Under both exposures SFT drifts progressively farther from the base model, while SST stays flat and learns the task better.

Figure 4: Controlled study on Qwen2.5-0.5B. Left: Increase in held-out NLL after fitting the toy task (lower is better). Right: Share of the weight change entering protected directions (lower is better). Only update projection achieves exact confinement while keeping the held-out NLL nearly unchanged.

Figure 5: Increase in in-domain accuracy for each recipe and its

+

SST

+

R10 version (Table 4 ). Light bars are the recipe alone, full-color bars the

+

SST version, and the gray rule in each group is the base model. Refresh mechanism improves the in-domain accuracy matching SFT performance.

Figure 6: Prior capability change compared to the base model Unconstrained recipes incur broad capability losses. SST variants stay much closer to the base model across benchmarks and domains.

Figure 7: Policy drift from the base model ,

\mathrm{KL}(\pi_{\text{base}}\,\|\,\pi_{\theta})

on WikiText-2, a general-domain dataset containing no task data. Left: drift over training. Right: the same drift against the in-domain accuracy, marker size growing with data seen.

Figure 8: Each recipe damages the base model in its own way, and SST pulls it back. Olmo-3-7B-Think fine-tuned on answer-only medical data (

+

R10 arms shown); the gray bar in each panel is the base model. Left: Mean response length, with the dotted rule marking the base model’s response length. Right: Rate of fabricating a “reference solution” the prompt never contained.

Figure 9: Two-stage continual fine-tuning (

A\rightarrow B

) on Qwen3-8B. Left: accuracy lost on domain

A

relative to its stage-1 level while stage 2 trains on

B

. Right: accuracy gained on

B

over the base model. SST arms use R10 refresh. SST improves both quantities in every tested chain.

Figure 10: Three-stage continual fine-tuning on Qwen3-8B. The SFT family is trained on FinQA

\rightarrow

Pathology

\rightarrow

Science. Lines track task accuracy on all three domains, dotted lines mark each domain’s base-model accuracy, each stage’s background is tinted with the domain trained at that stage, and the enlarged marker is the domain just trained. A line should rise in its own stage and hold its level afterward. Under SFT the dormant domains drop sharply and recover only when trained, while the SST variants hold all three domains much closer to their trained levels.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Interference from task-arithmetic merging. Each bar is the mean change in task accuracy relative to the same method’s own single-domain experts.

Figure 12: Two-stage continual fine-tuning with SDFT (

A\rightarrow B

) on Qwen3-8B. Left: accuracy lost on domain

A

relative to its stage-1 level after training on

B

. Right: accuracy gained on

B

over the base model. All arms use the same task chains and stage budgets; SST+R10 denotes SFT with SST and R10 refresh.

Figure 13: Three-stage SDFT chain on Qwen3-8B. The SDFT family is trained on Medicine

\rightarrow

Pathology

\rightarrow

Science. Lines track all three domains, dotted lines mark base-model accuracy, and the background tint identifies the domain trained at each stage.

Domain	Method	ID	HellaSwag	TQA-mc1	MMLU	IFEval	Winogr.	LM avg ( $\Delta$ )
ToolUse	Baseline	—	80.5	47.6	71.8	71.5	71.0	68.5 ( $+0.0$ )
	SFT	70.1	74.3	39.5	57.7	45.7	68.7	57.2 ( $-11.3$ )
	SFT+SST	70.1	65.5	47.6	68.4	70.4	62.6	62.9 ( $-5.6$ )
	SFT+SST+R100	69.1	65.3	47.2	68.3	69.5	63.0	62.7 ( $-5.8$ )
	SFT+SST+R10	71.1	67.6	47.4	68.5	69.7	63.3	63.3 ( $-5.2$ )
	SDFT	70.1	70.8	44.2	68.9	65.8	67.7	63.5 ( $-5.0$ )

Appendix

Table 1: Qwen2.5-7B-Instruct. ID

=

in-distribution accuracy; LM average over HellaSwag, TruthfulQA-MC1, MMLU, IFEval, and WinoGrande;

\Delta

is relative to the pretrained baseline (

68.5

). Bold entries mark column maxima among non-base recipes within each domain.

Model	Method	ID	GPQA	MMLU-Pro	GSM8K	HumanEval	IFEval	LM avg	$\Delta$
Qwen3-14B	Base	69.4	62.1	75.5	95.6	93.9	85.8	82.6	$+0.0$
	SFT	78.0	39.9	48.9	72.4	36.6	26.8	44.9	$-37.7$
	SFT+SST	86.5	63.6	75.4	94.5	95.1	79.7	81.7	$\mathbf{-0.9}$
	SFT+SST+R10	88.1	62.6	76.2	94.7	89.6	82.1	81.0	$-1.6$
Qwen3-8B	Base	60.2	60.1	73.0	94.8	91.5	85.6	81.0	$+0.0$
	SFT	76.1	35.4	48.7	81.0	36.0	34.2	47.1	$-33.9$

Appendix

Table 2: Five-epoch model-scale results underlying Figure 2 . Science fine-tuning with the same configuration at each size. LM avg is the five-benchmark average and

\Delta

is relative to each model’s own base. Bold entries mark column maxima among non-base recipes within each model size.

Model	Method	ID	GPQA	MMLU-Pro	GSM8K	HumanEval	IFEval	LM avg	$\Delta$
Qwen3-32B	Baseline	68.2	64.6	73.3	95.2	94.5	85.2	82.6	$+0.0$
	SFT	83.6	50.5	62.4	93.6	80.5	48.2	67.0	$-15.6$
	SFT+SST	84.1	62.6	73.5	94.8	97.0	79.9	81.6	$-1.0$
	SFT+SST+R100	85.6	58.6	73.8	95.2	95.1	81.0	80.7	$-1.9$
	SFT+SST+R10	87.3	65.7	74.3	95.1	94.5	79.1	81.7	$\mathbf{-0.9}$
Qwen3-14B	Baseline	69.4	62.1	75.5	95.6	93.9	85.8	82.6	$+0.0$

Appendix

Table 3: Two-epoch model-scale sweep on Science.

\Delta

is relative to each model’s own baseline. Cells marked — were not produced; bold entries mark column maxima among non-base recipes within each model size.

Domain	Method	ID	GPQA	MMLU-Pro	GSM8K	HumanEval	IFEval	LM avg	$\Delta$
Science	Baseline	60.2	60.1	73.0	94.8	91.5	85.6	81.0	$+0.0$
	SFT	77.6	38.4	62.3	93.9	36.6	46.2	55.5	$-25.5$
	SFT+SST	80.1	57.6	73.1	95.1	92.1	82.1	80.0	$-1.0$
	SFT+SST+R100	81.7	57.6	73.1	95.1	92.1	82.1	80.0	$-1.0$
	SFT+SST+R10	84.0	58.6	74.4	94.9	92.7	80.4	80.2	$-0.8$
	SDFT	88.1	42.4	62.5	92.9	61.6	25.3	57.0	$-24.0$

Appendix

Table 4: Qwen3-8B (thinking mode). Methods are grouped by recipe. Bold entries mark column maxima among non-base recipes within each domain.

\Delta

= LM avg

-

baseline (

81.0

Figure 14: Policy drift at the final fine-tuned checkpoint across Qwen3 sizes. Light bars are the recipe alone, full-color bars their

+

SST version (no-refresh).

Figure 15: Data scaling against subspace-constrained baselines on Qwen3-8B. The top two rows show Medicine (left) and Pathology (right), each using

9{,}000

unique examples repeated for five epochs. The bottom two rows show FinQA with matched

6{,}000

-example exposure:

2

k unique traces repeated for three epochs (left) or

6

k unique traces seen once (right). Within each pair, the upper panels show change in the five-benchmark prior-capability average from the base value of

81.0

, and the lower panels show target accuracy. Methods share the data and evaluation schedule within each setting. Curves compare no-refresh SST, SST+R10, Sculpting, and OPLoRA; SFT is omitted to keep the constrained trajectories legible.

	Science		FinQA		Pathology		Medicine		Mean
Method	ID	LM	ID	LM	ID	LM	ID	LM	ID	LM
Baseline	60.2	81.0	70.5	81.0	78.3	81.0	74.8	81.0	71.0	81.0
SFT	77.6	55.5	72.3	23.7	80.4	34.7	72.9	36.3	75.8	37.6
Sculpting	83.2	77.2	74.5	57.7	79.4	43.5	74.6	41.3	77.9	54.9
OPLoRA	78.5	79.1	72.8	61.7	77.4	51.6	70.0	60.0	74.7	63.1
SFT+SST	80.1	80.0	73.0	82.3	77.3	77.2	72.4	78.9	75.7	79.6

Appendix

Table 5: Comparison with subspace-constrained baselines using the official method-specific settings from their papers. All methods use the same datasets and evaluation suite. ID is in-domain accuracy and LM is the five-benchmark capability average (base

81.0

). Bold entries mark column maxima among non-base recipes.

Task	Method	ID	GPQA	MMLU-Pro	GSM8K	HumanEval	IFEval	Prior cap.
—	Base		58.1	59.7	94.4	92.1	85.8	78.0
FinQA	$\lambda{=}0$	73.4	10.1	13.4	41.8	24.4	28.1	23.6
	$\lambda{=}0.001$	73.4	0 7.6	14.2	43.5	24.4	26.4	23.2
	$\lambda{=}0.01$	71.3	0 7.1	13.2	42.0	17.7	26.6	21.3
	$\lambda{=}0.1$	66.0	13.6	15.8	14.2	0 0.0	0 9.4	10.6
Science	$\lambda{=}0.001$	81.7	49.5	54.4	93.3	60.4	55.1	62.5

Appendix

Table 6: SFT with reverse-KL regularisation, Qwen3-8B. Prior capability is the mean of the five held-out benchmarks shown. Bold entries mark column maxima among non-base recipes across the reported rows.

Figure 16: Reverse-KL regularisation across three coefficients. Prior capability (left) and in-domain accuracy (right) against SFT with KL regularization, averaged over FinQA and Science. The grey rule is the un-finetuned base model and the solid blue rule is SST at

r{=}0.99

on the same two tasks.

Figure 17: Protected rank fraction on FinQA. Prior capability (left) and in-domain accuracy (right) against the fraction of each weight’s subspace held fixed, with plain SFT as the

r{=}0

end of the same sweep. The grey rule is the un-finetuned base model. Where no KL coefficient recovers prior capability (Figure 16 ), rank recovers it almost completely and in-domain accuracy rises rather than falls.

Figure 18: FinQA sensitivity to learning rate and schedule. Each dot is one of six configurations, the thick segment spans the minimum and maximum, and the diamond marks the mean. SST greatly reduces variation in both in-domain accuracy and the out-of-domain (OOD) prior-capability average.

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 231 papers

May 6, 2026

Attribution-Guided Continual Learning for Large Language Models

Large language models (LLMs) often suffer from catastrophic forgetting in continual learning: after learning new tasks sequentially, they perform worse on earlier tasks. Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization. However, these methods lack understanding of LLM mechanisms and cannot distinguish which parameters store important knowledge from previous tasks and which parameters can be updated for new tasks. To address this, we propose the attribution-guided continual fine-tuning framework that leverages Layer-wise Relevance Propagation (LRP) to estimate parameter importance based on the internal computational process of LLMs. During continual learning, parameters critical to previous tasks are constrained to receive smaller updates, while less relevant parameters remain available for learning new tasks. Extensive experiments show that, compared with baseline methods, our approach reduces catastrophic forgetting while preserving adaptability to new tasks, highlighting the value of mechanistic attribution for continual fine-tuning of LLMs.

Yazheng Liu, Yuxuan Wan, Rui Xu +3

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Beijing University of Posts and Telecommunications, Beijing, China

Continual Learning for LLMs LLM Fine-Tuning Catastrophic Forgetting

Ask PDF

Oct 1, 2026

Local Support Learning

We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal. Following this observation, we propose Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data. During a new learning phase, LSL pairs two components with distinct roles: a standard weight adapter, trained as usual to minimize the loss, and a gating function that enables the adapter only on input activations from its own training distribution, making the update local to that distribution. The key challenge is that this gate must route data from all learning phases while training only on data from the current one. We address this with a gate based on a Gaussian Mixture Model (GMM), whose likelihood decays rapidly away from its training data, giving it a natural tendency to stay closed on data from prior phases. We show that this post-training approach can resolve forgetting in LLMs of up to 7 billion parameters, retaining both pretrained and finetuned capabilities across multiple training phases, while being efficient in memory and compute, robust to hyperparameter choice, and showing scaling potential.

Assaf Ben-Kish, Akarsh Kumar, James Glass +1

Tel Aviv University · MIT CSAIL

Continual Learning for LLMs Adapter-Based Continual Learning Catastrophic Forgetting

Ask PDF

May 28, 2026

On-Policy Replay for Continual Supervised Fine-Tuning

Continual supervised fine-tuning (SFT) is the de facto recipe for adapting large language models (LLMs) to a stream of downstream tasks, but it suffers from catastrophic forgetting of earlier capabilities. Recent work shows that on-policy signals -- training on the model's own outputs -- reduce forgetting more reliably than off-policy supervision. Existing on-policy methods route this signal through a new training objective (e.g., self-distillation losses with a teacher copy), inheriting an extra forward pass, schedule sensitivity, and stylistic drift from the teacher.We instead route the on-policy signal through the training data source. Our method, On-Policy Replay (OPR), rolls out the most recent checkpoint on a small budget of historical prompts, filters the generations by a task reward, and replays the surviving (prompt, model response) pairs as ordinary SFT examples. There is no teacher, no auxiliary loss, and no on-the-fly distillation. Across three 7--8B instruction-tuned backbones (Qwen2.5-7B-Instruct, Qwen3-8B, Llama3.1-8B-Instruct) on the TRACE continual-learning benchmark, OPR consistently reduces forgetting; on the sharpest stress test (Qwen2.5-7B-Instruct, Sequential SFT BWT -13.93), OPR lifts BWT to -0.65 at a 10% replay budget and to -2.29 at a 1% budget -- a 46% reduction in |BWT| over a tuned Vanilla Replay baseline, with 42--46% reductions observed across all three backbones. We give a KL-shrinkage interpretation that places OPR and prior on-policy distillation methods on a single axis, and we present a counterintuitive finding that explains why Vanilla Replay is already a strong baseline: low-score replay is uniformly worse than Vanilla Replay, demonstrating that the active ingredient in OPR is the on-policy distribution, not the response quality alone.Our code is available at https://github.com/Yancey2024/OnPolicyReplay.

Yan Chen, Taojie Zhu, Meng Zhang +4

Tsinghua University · Alibaba Group

Continual Learning for LLMs Replay-Based Continual Learning Catastrophic Forgetting

Ask PDF

Abstract

Continual Learning for LLMs

Explore similar work

Attribution-Guided Continual Learning for Large Language Models

Local Support Learning

On-Policy Replay for Continual Supervised Fine-Tuning