Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models

Bing Wang; Changchun Li; Xin-Qiang Cai; Lin Yuanbo Wu; Ximing Li; Gang Niu; Masashi Sugiyama

cs.CLSep 25, 2026

Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models

Authors: Bing Wang, Changchun Li, Xin-Qiang Cai, Lin Yuanbo Wu, Ximing Li, Gang Niu, Masashi Sugiyama

Organizations: College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin University · RIKEN Center for Advanced Intelligence Project · University of Warwick ·

Abstract

Continual fine-tuning is essential for large language models (LLMs) to dynamically adapt to real-world environments, yet it inevitably suffers from catastrophic forgetting, particularly the performance degradation of previous tasks and LLMs' general-purpose knowledge. Although existing methods, such as orthogonal gradient projection, mitigate the forgetting across various fine-tuning tasks, they fundamentally fail to preserve pre-training LLMs' inherent general-purpose knowledge because the original data and gradients of off-the-shelf pre-training LLMs required by these methods are strictly unknown and highly diverse. To bridge this critical gap, we propose EoupCT, a novel framework designed to Estimate and Orthogonalize Unknown Pre-training gradients for Continual LLM fine-Tuning. Specifically, EoupCT estimates pre-training gradients by dynamically generating pseudo data that is most susceptible to forgetting for new tasks through a learnable soft prompt equipped with Gumbel-Softmax relaxation. Furthermore, we formulate a multi-objective optimization problem and introduce a first-order efficient Pareto optimizer that jointly optimizes LLM parameters and the soft prompt, rigorously enforcing orthogonality between new task updates and the estimated pre-training gradients. Extensive experiments across multiple LLMs demonstrate that EoupCT effectively preserves both task-specific proficiency and inherent general-purpose knowledge, successfully mitigating the catastrophic forgetting.

Figures & tables

Figure 1: Framework of EoupCT with pre-training gradient estimation and Pareto orthogonal optimization.

		SuperNI						MMLU (average three task orders)
	Method	Order 1		Order 2		Order 3		STEM		Humanity		Other
		Rou.	Fgt.	Rou.	Fgt.	Rou.	Fgt.	Acc.	Fgt.	Acc.	Fgt.	Acc.	Fgt.
Qwen3-8B	+ LoRA	50.1	7.7	48.2	9.8	49.9	8.5	62.6	3.5	71.3	7.6	72.5	4.4
	+ LoRAMoE	50.8	6.3	49.4	8.7	48.9	8.4	61.6	4.5	71.3	7.5	72.0	4.9
	+ OLoRA	51.2	6.7	47.4	9.4	50.1	7.6	63.3	2.9	72.0	6.9	72.9	4.0
	+ GainLoRA	50.0	6.6	49.7	8.0	49.2	8.1	62.7	3.4	72.7	6.2	72.4	4.4

Table 1: Results of EoupCT under two task types: task-specific knowledge by SuperNI and general-purpose knowledge by MMLU ( Rou.

\uparrow

, Acc.

\uparrow

, Fgt.

\downarrow

). The bold results represent the best scores.

Figure 2: Sensitivity analysis of soft prompt length

L

and surrogate sequence length

M

Ablation	Qwen3-4B				Llama3-3B				Gemma2-2B
	SuperNI		MMLU		SuperNI		MMLU		SuperNI		MMLU
	Rou.	Fgt.	Acc.	Fgt.	Rou.	Fgt.	Acc.	Fgt.	Rou.	Fgt.	Acc.	Fgt.
EoupCT	50.7	6.0	60.2	0.7	44.0	10.5	41.4	-0.8	46.7	10.3	38.5	0.8
w/o C2	47.6	10.3	59.0	2.0	42.3	13.0	40.4	0.2	44.8	10.1	36.9	2.3
w/o P&PO	47.3	9.2	57.8	3.1	42.1	12.2	40.3	0.3	43.5	13.0	36.9	2.4
w/o N&PO	47.4	8.7	57.9	3.0	41.9	12.2	40.0	0.6	44.2	13.3	36.6	2.6

Table 2: Ablation experiments of different constraints in EoupCT .

Figure 3: Forgetting dynamics of Qwen3-4B (left) and Llama3-3B (right) across MMLU .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

LLM	LoRA	LoRAMoE (#expert=4)	LoRAMoE (#expert=8)	OLoRA	CLoRA	EoupCT
Qwen3-8B	88 (1 $\times$ )	223 (2.5 $\times$ )	225 (2.5 $\times$ )	237 (2.7 $\times$ )	191 (2.2 $\times$ )	162 (1.8 $\times$ )
Qwen3-4B	62 (1 $\times$ )	194 (3.1 $\times$ )	195 (3.1 $\times$ )	204 (3.3 $\times$ )	155 (2.5 $\times$ )	129 (2.1 $\times$ )
Llama3-8B	60 (1 $\times$ )	197 (3.3 $\times$ )	199 (3.3 $\times$ )	211 (3.5 $\times$ )	156 (2.6 $\times$ )	130 (2.1 $\times$ )
Llama3-3B	35 (1 $\times$ )	144 (4.1 $\times$ )	144 (4.1 $\times$ )	151 (4.3 $\times$ )	114 (3.2 $\times$ )	65 (1.8 $\times$ )
Gemma2-9B	76 (1 $\times$ )	260 (3.4 $\times$ )	262 (3.4 $\times$ )	271 (3.5 $\times$ )	210 (2.7 $\times$ )	178 (2.3 $\times$ )
Gemma2-2B	44 (1 $\times$ )	140 (3.2 $\times$ )	141 (3.2 $\times$ )	149 (3.4 $\times$ )	116 (2.6 $\times$ )	69 (1.5 $\times$ )

Appendix

Table 3: Empirical computational time (minutes) on 8

\times

GPUs.

Notation	Description
$\mathcal{F}_{\boldsymbol{\theta}},\boldsymbol{\theta}$	Frozen pre-training LLM and trainable LLM weights
$\mathcal{T}_{t},\mathcal{T}_{0}$	$t$ -th new downstream task and original pre-training distribution
$\mathbf{P}_{t},\mathbf{S}(\mathbf{P}_{t})$	Continuous soft prompt and generated surrogate sequence
$\mathcal{L}_{\text{new}},\mathcal{L}_{\text{pre}}$	Plasticity and stability objective functions
$\mathbf{g}_{\text{new}},\mathbf{g}_{\text{pre}}$	Task-specific and protection gradients w.r.t. $\boldsymbol{\theta}$
$\boldsymbol{\theta}_{\text{virt}}$	Virtually updated weights simulating new task impact

Appendix

Table 4: Summary of primary notations used in EoupCT .

ID	Dataset	Task Type	Train	Eval
t1290	xsum_summarization	text summarization	1,000	100
t511	reddit_tifu_long_text_summarization	text summarization	1,000	100
t1572	samsum_summary	text summarization	160	20
t1729	personachat_generate_next	dialogue generation	1,000	100
t639	multi_woz_user_utterance_generation	dialogue generation	142	18
t1590	diplomacy_text_generation	dialogue generation	126	16

Appendix

Table 5: Overview of selected multi-task datasets from SuperNI [ Wang et al., 2022 ] .

Seed	Task Ordering
1	$\text{t591}\rightarrow\text{t073}\rightarrow\text{t1687}\rightarrow\text{t875}\rightarrow\text{t1572}\rightarrow\text{t639}\rightarrow\text{t1510}\rightarrow\text{t1590}\rightarrow\text{t511}\rightarrow\text{t181}\rightarrow\text{t363}\rightarrow\text{t1729}\rightarrow\text{t002}\rightarrow\text{t1290}\rightarrow\text{t748}$
2	$\text{t1290}\rightarrow\text{t1510}\rightarrow\text{t1572}\rightarrow\text{t002}\rightarrow\text{t073}\rightarrow\text{t511}\rightarrow\text{t1590}\rightarrow\text{t363}\rightarrow\text{t748}\rightarrow\text{t639}\rightarrow\text{t181}\rightarrow\text{t1729}\rightarrow\text{t1687}\rightarrow\text{t591}\rightarrow\text{t875}$
3	$\text{t002}\rightarrow\text{t073}\rightarrow\text{t1572}\rightarrow\text{t181}\rightarrow\text{t591}\rightarrow\text{t1687}\rightarrow\text{t363}\rightarrow\text{t1729}\rightarrow\text{t511}\rightarrow\text{t875}\rightarrow\text{t639}\rightarrow\text{t748}\rightarrow\text{t1590}\rightarrow\text{t1290}\rightarrow\text{t1510}$

Appendix

Table 6: Three distinct task orderings under the continual LLM fine-tuning.

Subject	Category	#Eval
anatomy	other	135
global_facts	other	100
marketing	other	234
philosophy	humanities	311
formal_logic	humanities	283
prehistory	humanities	324

Appendix

Table 7: Statistic of MMLU subjects.

		Evaluation Task ID
		591	073	1687	875	1572	639	1510	1590	511	181	363	1729	002	1290	748
Training Task ID	591	67.2	80.1	42.5	41.0	32.6	8.2	24.2	11.2	14.8	13.9	60.6	10.6	77.1	18.8	42.9
	073	68.9	81.0	61.4	45.0	32.0	7.2	15.3	13.7	14.3	13.8	92.0	10.5	78.2	19.1	42.8
	1687	66.8	77.0	86.0	42.0	32.4	7.3	54.4	13.6	14.4	11.6	91.0	10.9	77.6	18.7	43.9
	875	62.9	77.0	85.0	81.0	35.2	6.6	74.8	12.0	13.4	23.8	84.0	10.2	73.8	20.7	43.3
	1572	61.0	67.0	86.0	74.0	40.7	7.0	85.9	10.5	14.8	29.7	87.0	14.1	73.5	18.2	44.5

Appendix

Table 8: Performance dynamics during continual fine-tuning, using the Qwen3-4B model with the EoupCT method ( Rouge = 50.7, Forget Rate = 6.0).

		Evaluation Task ID
		591	073	1687	875	1572	639	1510	1590	511	181	363	1729	002	1290	748
Training Task ID	591	67.8	79.0	78.0	43.0	34.3	4.0	95.7	12.8	14.6	12.3	88.0	11.9	77.1	19.5	43.6
	073	66.3	78.0	75.0	42.0	35.2	9.4	20.2	10.3	14.9	14.2	91.0	10.4	76.9	19.2	43.3
	1687	66.7	80.0	86.0	38.0	34.7	6.3	89.9	11.4	14.9	16.1	89.0	11.9	76.8	18.9	43.5
	875	61.1	66.0	82.0	83.0	37.2	5.3	92.7	14.0	13.9	6.2	79.0	11.4	75.2	18.3	42.6
	1572	55.2	69.0	80.0	77.0	44.8	4.3	93.3	9.1	14.5	28.2	82.0	15.9	76.5	18.7	43.6

Appendix

Table 9: Performance dynamics during continual fine-tuning, using the Qwen3-4B model with the GainLoRA method ( Rouge = 48.4, Forget Rate = 9.0).

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 231 papers

Attribution-Guided Continual Learning for Large Language Models
May 6, 2026Yazheng Liu, Yuxuan Wan, Rui Xu +3Continual Learning for LLMs LLM Fine-Tuning
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Beijing University of Posts and Telecommunications, Beijing, China
Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models
May 14, 2026Yuehao Liu, Shanyan Guan, Weijia Zhang +4Continual Learning for LLMs Continual Learning for VLMs
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · vivo Mobile Communication Co., Ltd.
Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
Oct 15, 2026Arnav Chavan, Kusha Sareen, Rishabh Tiwari +6Continual Learning for LLMs LLM Fine-Tuning
UC Berkeley · Mila · McGill University +5

May 6, 2026J/K move · Enter open · S save

Attribution-Guided Continual Learning for Large Language Models

Yazheng Liu, Yuxuan Wan, Rui Xu, Xi Zhang+2

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Beijing University of Posts and Telecommunications, Beijing, China

Continual Learning for LLMs LLM Fine-Tuning Catastrophic Forgetting

Open paper AskPDF arXiv

Large language models (LLMs) often suffer from catastrophic forgetting in continual learning: after learning new tasks sequentially, they perform worse on earlier tasks. Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization. However, these methods lack understanding of LLM mechanisms and cannot distinguish which parameters store important knowledge from previous tasks and which parameters can be updated for new tasks. To address this, we propose the attribution-guided continual fine-tuning framework that leverages Layer-wise Relevance Propagation (LRP) to estimate parameter importance based on the internal computational process of LLMs. During continual learning, parameters critical to previous tasks are constrained to receive smaller updates, while less relevant parameters remain available for learning new tasks. Extensive experiments show that, compared with baseline methods, our approach reduces catastrophic forgetting while preserving adaptability to new tasks, highlighting the value of mechanistic attribution for continual fine-tuning of LLMs.

Abstract

Continual Learning for LLMs

Explore similar work

Attribution-Guided Continual Learning for Large Language Models

Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models

Towards Continual Learning: Mitigating Forgetting With Subspace Tuning