Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

Arnav Chavan; Kusha Sareen; Rishabh Tiwari; Abhranil Chandra; Johan Obando-Ceron; Kurt Keutzer; Scott Niekum; Inderjit S Dhillon

cs.LGOct 15, 2026

Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

Authors: Arnav Chavan, Kusha Sareen, Rishabh Tiwari, Abhranil Chandra, Johan Obando-Ceron, Kurt Keutzer, Scott Niekum, Inderjit S Dhillon,

Organizations: UC Berkeley · Mila · McGill University · UMass Amherst ·

Abstract

Large language models (LLMs) acquire new skills through fine-tuning, but it can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to continual learning in LLMs. Recent works connect forgetting to how far, and along which directions, fine-tuning moves the weights. On-policy methods such as reinforcement learning and on-policy distillation avoid updating the principal singular directions of weight matrices and thus forget less. In contrast, SFT concentrates updates along these directions and is susceptible to catastrophic forgetting. In this work, we introduce Subspace Tuning (SST), which restricts each weight update to the bottom singular subspace, eliminating changes along the principal directions. SST is optimizer-agnostic and composes well with fine-tuning recipes including SFT and self-distillation. SST combines tight subspace protection, dense gradient estimation, and a simple refresh mechanism, enabling stronger learning than prior orthogonal fine-tuning methods. With these mechanisms, protecting as much as 99% of the weight subspace substantially improves retention while preserving strong target-task performance. This behavior strengthens with model and data size and nearly eliminates forgetting in the large-model regime. SST also keeps the model close to its base distribution, inducing $2$ -- $4\times$ less KL drift than the baseline recipes. Overall, SST advances the learning--forgetting Pareto frontier, making it a practical default for fine-tuning pre-trained language models.

Figures & tables

Figure 1: SST freezes the top of each weight matrix’s spectrum, and improves learning–forgetting trade-off across fine-tuning recipes. Left: the method. SST projects the update on low energy singular directions of each weight matrix. Right: the result on Qwen3-8B, averaged over four domains (Table 4 ). Both axes show changes from the base model in percentage points (pp); higher is better. Circles denote training without SST, stars denote training with SST, and the black triangle marks the base model.

Figure 2: Scaling model size. Qwen3 models from

1.7

B to

14

B fine-tuned on Science for five epochs. SFT forgets heavily at every scale, whereas forgetting under SST shrinks as models grow and nearly vanishes from

8

B onward. SST also reaches higher new-task accuracy at every size.

Figure 3: Scaling training data. Qwen3-8B fine-tuned on Science under a matched

45

k-example budget: the novel arms see a fresh stream once, the repeated arms cycle

9

k examples for five epochs. Under both exposures SFT drifts progressively farther from the base model, while SST stays flat and learns the task better.

Figure 4: Controlled study on Qwen2.5-0.5B. Left: Increase in held-out NLL after fitting the toy task (lower is better). Right: Share of the weight change entering protected directions (lower is better). Only update projection achieves exact confinement while keeping the held-out NLL nearly unchanged.

Figure 5: Increase in in-domain accuracy for each recipe and its

+

SST

+

R10 version (Table 4 ). Light bars are the recipe alone, full-color bars the

+

SST version, and the gray rule in each group is the base model. Refresh mechanism improves the in-domain accuracy matching SFT performance.

Figure 6: Prior capability change compared to the base model Unconstrained recipes incur broad capability losses. SST variants stay much closer to the base model across benchmarks and domains.

Figure 7: Policy drift from the base model ,

\mathrm{KL}(\pi_{\text{base}}\,\|\,\pi_{\theta})

on WikiText-2, a general-domain dataset containing no task data. Left: drift over training. Right: the same drift against the in-domain accuracy, marker size growing with data seen.

Figure 8: Each recipe damages the base model in its own way, and SST pulls it back. Olmo-3-7B-Think fine-tuned on answer-only medical data (

+

R10 arms shown); the gray bar in each panel is the base model. Left: Mean response length, with the dotted rule marking the base model’s response length. Right: Rate of fabricating a “reference solution” the prompt never contained.

Figure 9: Two-stage continual fine-tuning (

A\rightarrow B

) on Qwen3-8B. Left: accuracy lost on domain

A

relative to its stage-1 level while stage 2 trains on

B

. Right: accuracy gained on

B

over the base model. SST arms use R10 refresh. SST improves both quantities in every tested chain.

Figure 10: Three-stage continual fine-tuning on Qwen3-8B. The SFT family is trained on FinQA

\rightarrow

Pathology

\rightarrow

Science. Lines track task accuracy on all three domains, dotted lines mark each domain’s base-model accuracy, each stage’s background is tinted with the domain trained at that stage, and the enlarged marker is the domain just trained. A line should rise in its own stage and hold its level afterward. Under SFT the dormant domains drop sharply and recover only when trained, while the SST variants hold all three domains much closer to their trained levels.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Interference from task-arithmetic merging. Each bar is the mean change in task accuracy relative to the same method’s own single-domain experts.

Figure 12: Two-stage continual fine-tuning with SDFT (

A\rightarrow B

) on Qwen3-8B. Left: accuracy lost on domain

A

relative to its stage-1 level after training on

B

. Right: accuracy gained on

B

over the base model. All arms use the same task chains and stage budgets; SST+R10 denotes SFT with SST and R10 refresh.

Figure 13: Three-stage SDFT chain on Qwen3-8B. The SDFT family is trained on Medicine

\rightarrow

Pathology

\rightarrow

Science. Lines track all three domains, dotted lines mark base-model accuracy, and the background tint identifies the domain trained at each stage.

Domain	Method	ID	HellaSwag	TQA-mc1	MMLU	IFEval	Winogr.	LM avg ( $\Delta$ )
ToolUse	Baseline	—	80.5	47.6	71.8	71.5	71.0	68.5 ( $+0.0$ )
	SFT	70.1	74.3	39.5	57.7	45.7	68.7	57.2 ( $-11.3$ )
	SFT+SST	70.1	65.5	47.6	68.4	70.4	62.6	62.9 ( $-5.6$ )
	SFT+SST+R100	69.1	65.3	47.2	68.3	69.5	63.0	62.7 ( $-5.8$ )
	SFT+SST+R10	71.1	67.6	47.4	68.5	69.7	63.3	63.3 ( $-5.2$ )
	SDFT	70.1	70.8	44.2	68.9	65.8	67.7	63.5 ( $-5.0$ )

Appendix

Table 1: Qwen2.5-7B-Instruct. ID

=

in-distribution accuracy; LM average over HellaSwag, TruthfulQA-MC1, MMLU, IFEval, and WinoGrande;

\Delta

is relative to the pretrained baseline (

68.5

). Bold entries mark column maxima among non-base recipes within each domain.

Model	Method	ID	GPQA	MMLU-Pro	GSM8K	HumanEval	IFEval	LM avg	$\Delta$
Qwen3-14B	Base	69.4	62.1	75.5	95.6	93.9	85.8	82.6	$+0.0$
	SFT	78.0	39.9	48.9	72.4	36.6	26.8	44.9	$-37.7$
	SFT+SST	86.5	63.6	75.4	94.5	95.1	79.7	81.7	$\mathbf{-0.9}$
	SFT+SST+R10	88.1	62.6	76.2	94.7	89.6	82.1	81.0	$-1.6$
Qwen3-8B	Base	60.2	60.1	73.0	94.8	91.5	85.6	81.0	$+0.0$
	SFT	76.1	35.4	48.7	81.0	36.0	34.2	47.1	$-33.9$

Appendix

Table 2: Five-epoch model-scale results underlying Figure 2 . Science fine-tuning with the same configuration at each size. LM avg is the five-benchmark average and

\Delta

is relative to each model’s own base. Bold entries mark column maxima among non-base recipes within each model size.

Model	Method	ID	GPQA	MMLU-Pro	GSM8K	HumanEval	IFEval	LM avg	$\Delta$
Qwen3-32B	Baseline	68.2	64.6	73.3	95.2	94.5	85.2	82.6	$+0.0$
	SFT	83.6	50.5	62.4	93.6	80.5	48.2	67.0	$-15.6$
	SFT+SST	84.1	62.6	73.5	94.8	97.0	79.9	81.6	$-1.0$
	SFT+SST+R100	85.6	58.6	73.8	95.2	95.1	81.0	80.7	$-1.9$
	SFT+SST+R10	87.3	65.7	74.3	95.1	94.5	79.1	81.7	$\mathbf{-0.9}$
Qwen3-14B	Baseline	69.4	62.1	75.5	95.6	93.9	85.8	82.6	$+0.0$

Appendix

Table 3: Two-epoch model-scale sweep on Science.

\Delta

is relative to each model’s own baseline. Cells marked — were not produced; bold entries mark column maxima among non-base recipes within each model size.

Domain	Method	ID	GPQA	MMLU-Pro	GSM8K	HumanEval	IFEval	LM avg	$\Delta$
Science	Baseline	60.2	60.1	73.0	94.8	91.5	85.6	81.0	$+0.0$
	SFT	77.6	38.4	62.3	93.9	36.6	46.2	55.5	$-25.5$
	SFT+SST	80.1	57.6	73.1	95.1	92.1	82.1	80.0	$-1.0$
	SFT+SST+R100	81.7	57.6	73.1	95.1	92.1	82.1	80.0	$-1.0$
	SFT+SST+R10	84.0	58.6	74.4	94.9	92.7	80.4	80.2	$-0.8$
	SDFT	88.1	42.4	62.5	92.9	61.6	25.3	57.0	$-24.0$

Appendix

Table 4: Qwen3-8B (thinking mode). Methods are grouped by recipe. Bold entries mark column maxima among non-base recipes within each domain.

\Delta

= LM avg

-

baseline (

81.0

Figure 14: Policy drift at the final fine-tuned checkpoint across Qwen3 sizes. Light bars are the recipe alone, full-color bars their

+

SST version (no-refresh).

Figure 15: Data scaling against subspace-constrained baselines on Qwen3-8B. The top two rows show Medicine (left) and Pathology (right), each using

9{,}000

unique examples repeated for five epochs. The bottom two rows show FinQA with matched

6{,}000

-example exposure:

2

k unique traces repeated for three epochs (left) or

6

k unique traces seen once (right). Within each pair, the upper panels show change in the five-benchmark prior-capability average from the base value of

81.0

, and the lower panels show target accuracy. Methods share the data and evaluation schedule within each setting. Curves compare no-refresh SST, SST+R10, Sculpting, and OPLoRA; SFT is omitted to keep the constrained trajectories legible.

	Science		FinQA		Pathology		Medicine		Mean
Method	ID	LM	ID	LM	ID	LM	ID	LM	ID	LM
Baseline	60.2	81.0	70.5	81.0	78.3	81.0	74.8	81.0	71.0	81.0
SFT	77.6	55.5	72.3	23.7	80.4	34.7	72.9	36.3	75.8	37.6
Sculpting	83.2	77.2	74.5	57.7	79.4	43.5	74.6	41.3	77.9	54.9
OPLoRA	78.5	79.1	72.8	61.7	77.4	51.6	70.0	60.0	74.7	63.1
SFT+SST	80.1	80.0	73.0	82.3	77.3	77.2	72.4	78.9	75.7	79.6

Appendix

Table 5: Comparison with subspace-constrained baselines using the official method-specific settings from their papers. All methods use the same datasets and evaluation suite. ID is in-domain accuracy and LM is the five-benchmark capability average (base

81.0

). Bold entries mark column maxima among non-base recipes.

Task	Method	ID	GPQA	MMLU-Pro	GSM8K	HumanEval	IFEval	Prior cap.
—	Base		58.1	59.7	94.4	92.1	85.8	78.0
FinQA	$\lambda{=}0$	73.4	10.1	13.4	41.8	24.4	28.1	23.6
	$\lambda{=}0.001$	73.4	0 7.6	14.2	43.5	24.4	26.4	23.2
	$\lambda{=}0.01$	71.3	0 7.1	13.2	42.0	17.7	26.6	21.3
	$\lambda{=}0.1$	66.0	13.6	15.8	14.2	0 0.0	0 9.4	10.6
Science	$\lambda{=}0.001$	81.7	49.5	54.4	93.3	60.4	55.1	62.5

Appendix

Table 6: SFT with reverse-KL regularisation, Qwen3-8B. Prior capability is the mean of the five held-out benchmarks shown. Bold entries mark column maxima among non-base recipes across the reported rows.

Figure 16: Reverse-KL regularisation across three coefficients. Prior capability (left) and in-domain accuracy (right) against SFT with KL regularization, averaged over FinQA and Science. The grey rule is the un-finetuned base model and the solid blue rule is SST at

r{=}0.99

on the same two tasks.

Figure 17: Protected rank fraction on FinQA. Prior capability (left) and in-domain accuracy (right) against the fraction of each weight’s subspace held fixed, with plain SFT as the

r{=}0

end of the same sweep. The grey rule is the un-finetuned base model. Where no KL coefficient recovers prior capability (Figure 16 ), rank recovers it almost completely and in-domain accuracy rises rather than falls.

Figure 18: FinQA sensitivity to learning rate and schedule. Each dot is one of six configurations, the thick segment spans the minimum and maximum, and the diamond marks the mean. SST greatly reduces variation in both in-domain accuracy and the out-of-domain (OOD) prior-capability average.

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 231 papers

Attribution-Guided Continual Learning for Large Language Models
May 6, 2026Yazheng Liu, Yuxuan Wan, Rui Xu +3Continual Learning for LLMs LLM Fine-Tuning
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Beijing University of Posts and Telecommunications, Beijing, China
Local Support Learning
Oct 1, 2026Assaf Ben-Kish, Akarsh Kumar, James Glass +1Continual Learning for LLMs Adapter-Based Continual Learning
Tel Aviv University · MIT CSAIL
On-Policy Replay for Continual Supervised Fine-Tuning
May 28, 2026Yan Chen, Taojie Zhu, Meng Zhang +4Continual Learning for LLMs Replay-Based Continual Learning
Tsinghua University · Alibaba Group

May 6, 2026J/K move · Enter open · S save

Attribution-Guided Continual Learning for Large Language Models

Yazheng Liu, Yuxuan Wan, Rui Xu, Xi Zhang+2

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Beijing University of Posts and Telecommunications, Beijing, China

Continual Learning for LLMs LLM Fine-Tuning Catastrophic Forgetting

Open paper AskPDF arXiv

Large language models (LLMs) often suffer from catastrophic forgetting in continual learning: after learning new tasks sequentially, they perform worse on earlier tasks. Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization. However, these methods lack understanding of LLM mechanisms and cannot distinguish which parameters store important knowledge from previous tasks and which parameters can be updated for new tasks. To address this, we propose the attribution-guided continual fine-tuning framework that leverages Layer-wise Relevance Propagation (LRP) to estimate parameter importance based on the internal computational process of LLMs. During continual learning, parameters critical to previous tasks are constrained to receive smaller updates, while less relevant parameters remain available for learning new tasks. Extensive experiments show that, compared with baseline methods, our approach reduces catastrophic forgetting while preserving adaptability to new tasks, highlighting the value of mechanistic attribution for continual fine-tuning of LLMs.

Abstract

Continual Learning for LLMs

Explore similar work

Attribution-Guided Continual Learning for Large Language Models

Local Support Learning

On-Policy Replay for Continual Supervised Fine-Tuning