Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning

Zekai Lin; Chao Xue; Di Liang; Xingsheng Han; Peiyang Liu; Xianjie Wu; Lei Jiang; Yu Lu

cs.LGApr 15, 2026

Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning

Authors: Zekai Lin, Chao Xue, Di Liang, Xingsheng Han, Peiyang Liu, Xianjie Wu, Lei Jiang, Yu Lu, +3 more

Organizations: Tencent Hunyuan · University of New South Wales · Tencent Yuanbao · Peking University · UESTC, China

Abstract

Supervised Fine-Tuning (SFT) of large language models often suffers from task interference and catastrophic forgetting. Recent approaches alleviate this issue by isolating task-critical parameters during training. However, these methods represent a static solution to a dynamic problem, assuming that parameter importance remains fixed once identified. In this work, we empirically demonstrate that parameter importance exhibits temporal drift over the course of training. To address this, we propose Evolving Parameter Isolation (EPI), a fine-tuning framework that adapts isolation decisions based on online estimates of parameter importance. Instead of freezing a fixed subset of parameters, EPI periodically updates isolation masks using gradient-based signals, enabling the model to protect emerging task-critical parameters while releasing outdated ones to recover plasticity. Experiments on diverse multi-task benchmarks demonstrate that EPI consistently reduces interference and forgetting compared to static isolation and standard fine-tuning, while improving overall generalization. Our analysis highlights the necessity of synchronizing isolation mechanisms with the evolving dynamics of learning diverse abilities.

Figures & tables

Figure 1: Conceptual illustration of parameter isolation strategies. (a) Standard SFT: parameters are updated globally, leading to interference. Arrows denote task-specific gradient directions. (b) Static Isolation: The pre-computed mask fails to cover critical parameters (red) as their importance distribution shifts during training. (c) Dynamic Isolation (Ours): The protection mask is dynamically updated to follow the trajectory of parameter importance.

Figure 2: Overview of the Evolving Parameter Isolation (EPI) framework. The model trains on heterogeneous data (Left) while the EPI Engine (Middle) continuously estimates parameter sensitivity using exponential moving averages of squared gradients. These signals drive the Evolutionary Locking mechanism (Right), which dynamically updates isolation masks from $T_{start}$ to $T_{end}$ to protect shifting task-critical parameters.

Figure 3: Conceptual illustration of temporal smoothing. The Instantaneous Gradient (Gray) exhibits high variance. In contrast, the EMA Estimate (Orange) effectively filters high-frequency noise, revealing the underlying importance trend ( $\boldsymbol{S}_{t}$ ).

Figure 4: Conceptual illustration of necessity of Layer-wise Normalization. (a) Without normalization, global ranking causes severe bias, concentrating protection only on layers with high gradient norms (Red bars). (b) Our Adaptive Layer-wise Normalization rectifies this by projecting scores to a local scale, resulting in a uniform distribution of critical parameters (Blue bars) regardless of the layer’s position.

Figure 5: Dynamic Mask Evolution. Instead of a fixed constraint, the protection mask evolves periodically ( $T_{1}\rightarrow T_{3}$ ). Critical parameters (Red) are locked to prevent forgetting, while redundant ones (Grey) are released. This creates a moving shield that adapts to the temporal drift of parameter importance.

Base Model	Method	GSM8K	CodeAlpaca	LogiQA	Alpaca	UltraChat	Avg. Norm.
LLaMA-3-8B	Full SFT	55.6	28.3	60.8	7.8	8.1	7.32
	Multi-Stage (Random)	56.9	27.9	61.5	8.0	8.2	7.44
	Multi-Stage (Heuristic)	57.4	28.7	62.1	7.7	8.0	7.48
	Static Isolation ( $p=1\%$ )	59.2	29.4	63.0	8.1	8.3	7.68
	EPI (Ours)	61.8	31.2	65.4	8.4	8.6	7.98
Mistral-7B	Full SFT	53.1	26.7	58.9	7.5	7.9	7.05

Table 1: Main experimental results on heterogeneous SFT benchmarks. EPI consistently outperforms standard SFT, multi-stage training, and static parameter isolation across all base models. Dynamic isolation yields the largest gains on reasoning-intensive tasks (GSM8K, LogiQA) and improves overall generalization as measured by Avg. Norm. Score. Background colors denote the top-3 methods (Darker Orange indicates higher performance).

Mask Strategy	LLaMA-3	Gemma-2
Static Mask ( $p=1\%$ )	7.68	8.21
Static Mask + Layer Norm	7.82	8.36
EPI (w/o Layer Norm)	7.83	8.35
EPI (Full)	7.98	8.57

Table 2: Ablation of mask strategy. Both layer-wise normalization and dynamic evolution contribute meaningfully, and their combination yields the strongest performance.

Strategy	Avg. Norm. Score
Per-Layer Fixed Budget	7.82
Global Raw Scores	7.71
EPI (Ours)	7.98

Table 3: Comparison of alternative allocation strategies on LLaMA-3-8B. Both per-layer fixed quotas and global raw ranking underperform our layer-wise normalization plus global top-

p\%

selection.

Smoothing Factor	LLaMA-3	Gemma-2
No EMA (Instantaneous)	7.81	8.39
EMA ( $\beta=0.9$ )	7.90	8.50
EMA ( $\beta=0.99$ )	7.98	8.57

Table 4: Effect of Temporal Accumulation. A high smoothing factor (

\beta=0.99

) is essential to filter stochastic noise and reveal stable importance trends.

Figure 6: Sensitivity Analysis to Isolation Ratio ( $p$ ). The plot illustrates the average normalized score across different isolation ratios for LLaMA-3 and Gemma-2. Both models exhibit a distinct peak performance at $p=1.0\%$ , representing an optimal trade-off between retaining prior knowledge (Stability) and adapting to new tasks (Plasticity).

Update Interval ( $H$ )	Avg. Norm. Score
100	7.92
500	7.98
1000	7.95
2000	7.89

Table 5: Sensitivity to the mask-update interval

H

on LLaMA-3-8B.

H=500

provides the best trade-off between responsiveness and stability.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Base Model	Method	Forgetting Ratio (FR, %) $\downarrow$					Avg. FR
Base Model	Method	GSM8K	Code	LogiQA	Alpaca	UChat	Avg. FR
LLaMA-3-8B	Full SFT (Multi-task)	12.6	10.4	9.8	7.5	6.9	9.44
	Multi-Stage (Random)	9.7	8.9	8.1	6.8	6.2	7.94
	Multi-Stage (Heuristic)	8.9	8.2	7.5	6.5	6.0	7.42
	Static Isolation ( $p=1\%$ )	6.1	5.8	5.2	4.5	4.1	5.14
	EPI (Ours, $p=1\%$ )	5.3	5.1	4.8	4.2	3.9	4.66

Appendix

Table 6: Forgetting analysis across tasks and base models. Lower FR (%) indicates better retention of previously learned knowledge. EPI consistently reduces forgetting compared to standard SFT and static isolation baselines.

Base Model	Method	Task Gradient Conflict (TGC) $\downarrow$					Avg. TGC
Base Model	Method	Code	LogiQA	Alpaca	Code-Logi	Code-Alp	Avg. TGC
LLaMA-3-8B	Full SFT (Multi-task)	0.42	0.38	0.36	0.33	0.30	0.358
	Multi-Stage (Random)	0.31	0.28	0.26	0.24	0.22	0.262
	Multi-Stage (Heuristic)	0.28	0.25	0.23	0.21	0.20	0.234
	Static Isolation ( $p=1\%$ )	0.18	0.15	0.13	0.12	0.11	0.138
	EPI (Ours, $p=1\%$ )	0.15	0.12	0.10	0.09	0.08	0.108

Appendix

Table 7: Seesaw effect analysis via Task Gradient Conflict (TGC). Lower TGC signifies reduced task interference. EPI minimizes gradient conflicts more effectively than full multi-task and static baselines.

Task	Input Prompt	Full SFT Output	Static Mask Output	EPI Output (Ours)
GSM8K (Reasoning)	A train leaves city A at 9:00 AM… (Time/Speed problem)	1:30 PM	1:35 PM	1:30 PM, with step-by-step reasoning showing distance covered by each train.
CodeAlpaca (Python)	Write a function that returns Fibonacci numbers up to n.	Returns first 5 numbers only, ignores n.	Correct function but inefficient loop, $O(n^{2})$ .	Correct and efficient function using iterative approach, handles all n.
Alpaca (Instruction)	Summarize the plot of ’Pride and Prejudice’ in 2-3 sentences.	Misses key relationships, inconsistent with timeline.	Basic summary, minor omissions.	Concise and accurate summary including major characters, timeline, and main conflict.
UltraChat (Dialogue)	User: How do I cook pasta perfectly?	Overgeneralized advice, misses boiling details.	Correct steps but unordered.	Step-by-step guidance with tips for timing, salt, and texture; natural conversational tone.

Appendix

Table 8: Qualitative comparison of model outputs. EPI demonstrates superior reasoning consistency and completeness compared to baselines, effectively mitigating the trade-off between forgetting and learning.

Base Model	Method	GSM8K	CodeAlpaca	LogiQA	Alpaca	UltraChat	Avg. Norm.
LLaMA-3-8B	Full SFT	55.6 $\pm$ 0.4	28.3 $\pm$ 0.5	60.8 $\pm$ 0.5	7.8 $\pm$ 0.2	8.1 $\pm$ 0.1	7.32 $\pm$ 0.06
	Static Isolation	59.2 $\pm$ 0.6	29.4 $\pm$ 0.4	63.0 $\pm$ 0.4	8.1 $\pm$ 0.1	8.3 $\pm$ 0.1	7.68 $\pm$ 0.04
	EPI (Ours)	61.8 $\pm$ 0.3	31.2 $\pm$ 0.3	65.4 $\pm$ 0.5	8.4 $\pm$ 0.1	8.6 $\pm$ 0.1	7.98 $\pm$ 0.03
Gemma-2-9B	Full SFT	58.5 $\pm$ 0.5	30.1 $\pm$ 0.6	64.0 $\pm$ 0.6	8.3 $\pm$ 0.2	8.5 $\pm$ 0.2	7.78 $\pm$ 0.07
	Static Isolation	62.2 $\pm$ 0.4	31.5 $\pm$ 0.5	66.4 $\pm$ 0.5	8.6 $\pm$ 0.1	8.8 $\pm$ 0.1	8.21 $\pm$ 0.05
	EPI (Ours)	65.0 $\pm$ 0.4	33.2 $\pm$ 0.5	69.1 $\pm$ 0.4	8.9 $\pm$ 0.1	9.1 $\pm$ 0.1	8.57 $\pm$ 0.04

Appendix

Table 9: Main results with standard deviations (3 runs). EPI consistently outperforms baselines with small standard deviations across tasks, demonstrating that the gains reported in the main text are statistically robust rather than artifacts of a favorable seed.

Isolation Ratio $p$	GSM8K	CodeAlpaca	LogiQA	Alpaca	UltraChat	Avg. Norm.
0.5%	58.7	28.9	62.3	8.0	8.2	7.59
1%	59.2	29.4	63.0	8.1	8.3	7.68
2%	58.4	28.5	61.8	7.8	7.9	7.42
5%	56.9	27.6	60.1	7.4	7.6	7.12

Appendix

Table 10: Static Isolation Ratio Sweep on LLaMA-3-8B . Static isolation peaks at

p=1\%

and drops sharply for

p\geq 2\%

, directly supporting over-protection as a failure mode of static masks.

Training Progress	25%	50%	75%	100% (Final)
Mask Overlap (%)	78.3	62.1	51.4	44.7

Appendix

Table 11: Temporal drift in isolation masks. The Jaccard overlap between the initial mask and later masks steadily decreases during training.

Layer Group	Average Mask Flip Rate (%)
Layers 1–8	12.3
Layers 9–16	21.7
Layers 17–24	33.5
Layers 25–32	41.2

Appendix

Table 12: Layer-wise mask evolution. Upper layers exhibit higher flip rates, indicating stronger adaptation to evolving task demands.

Task Pair	Jaccard Similarity (Top-1%)
GSM8K vs. CodeAlpaca	0.31
GSM8K vs. LogiQA	0.42
CodeAlpaca vs. LogiQA	0.28

Appendix

Table 13: Task-conditioned overlap of important parameters. Different tasks rely on partially distinct critical parameter subsets.

Method	$p=0.5\%$	$p=1\%$	$p=2\%$	$p=5\%$
Static Isolation	8.2	8.3	7.9	7.6
EPI	8.4	8.6	8.4	8.2

Appendix

Table 14: UltraChat performance vs. isolation ratio . EPI maintains higher new-task plasticity at large

p

by releasing outdated parameters, whereas static isolation exhibits substantially larger degradation.

Training Stage	Pearson $r$	Spearman $\rho$
Early	0.72	0.68
Middle	0.75	0.71
Late	0.73	0.69

Appendix

Table 15: Correlation between online importance and old-task sensitivity (LLaMA-3-8B). We sample parameters across layers, estimate true old-task sensitivity via small Gaussian perturbations (measured as

\Delta\mathcal{L}_{\text{old}}

on a held-out batch), and correlate it with our online importance score. Strong correlations persist across all training stages.

Explore similar work

TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning
May 29, 2026Xiaosong Han, Ke Chen, Xindi Dai +7Continual Learning for LLMs LLM Fine-Tuning
Key Laboratory of Symbolic Computation and Knowledge Engineering of the Ministry of Education · College of Computer Science and Technology, Jilin University Changchun, China · College of Software, Jilin University Changchun, China +3
Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization
Apr 18, 2026Weijie Wan, Jiangjiang ZhaoContinual Learning for LLMs LLM Fine-Tuning
Shortest Path Technology · China Mobile Communications Group Co.,Ltd
PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
May 27, 2026Yangyi Huang, Ruotian Peng, Zeju Qiu +4LLM Evaluation Fine-Tuning
The Chinese University of Hong Kong · Westlake University · MPI for Intelligent Systems

May 29, 2026J/K move · Enter open · S save

TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning

Xiaosong Han, Ke Chen, Xindi Dai, Di Liang+6

Key Laboratory of Symbolic Computation and Knowledge Engineering of the Ministry of Education · College of Computer Science and Technology, Jilin University Changchun, China · College of Software, Jilin University Changchun, China+3

Continual Learning for LLMs LLM Fine-Tuning Selective Fine-Tuning Parameter-Efficient Fine-Tuning Catastrophic Forgetting

Open paper AskPDF arXiv

In real-world deployment, LLMs are often adapted continually across tasks to keep LLMs up-to-date in production, where new fine-tuning should preserve previously learned skills. However, indiscriminately mixing tasks can dilute task specialization, while sequential fine-tuning (full-parameter or low rank adaptation) often causes catastrophic forgetting due to destructive overwriting. Replay-based continual tuning and maintaining separate task-specific adapters can mitigate forgetting, but introduce additional compute, storage, and management overhead. Recognizing the redundancy of LLM parameters for any single task, we reframe continual task adaptation as task-specific parameter discovery via adaptation-aware probing: a short warm-start probe exposes a task's adaptation trace, enabling us to identify and isolate the small subset of parameters essential for each task to mitigate catastrophic forgetting. Building on this view, we introduce TRACE, a novel approach for discovering Task-specific paRameters via Adaptation-aware probing for Continual finE-tuning. We perform a short warm-start fine-tune to derive task-specific core parameters by comparing the warm-started and pre-trained models. Core parameters are identified via two strategies: importance scoring (L

_2

norm and Fisher Information) and specificity analysis (cosine similarity of parameter updates). In continual fine-tuning settings, only the active task's core parameters are updated while others remain frozen, preserving prior knowledge. We conduct extensive experiments across multiple standard benchmarks to demonstrate the superior performance of our proposed method. Additionally, we validate the generalization of our method through a cross-model and scale transferability study, demonstrating a "small-to-large" paradigm that guides the fine-tuning of large-scale models under resource constraints.

Abstract

Explore similar work

TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning

Efficient Task Adaptation in Large Language Models via Selective Parameter Optimization

PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective