Confine to the Task, Anchor the Rest: Concept-Guided Retention Regularizer for Parameter-Efficient Fine-Tuning

Ibna Kowsar; Ivan Medri; Fanyu Meng; Xin Liu

cs.LGOct 13, 2026

Confine to the Task, Anchor the Rest: Concept-Guided Retention Regularizer for Parameter-Efficient Fine-Tuning

Authors: Ibna Kowsar, Ivan Medri, Fanyu Meng, Xin Liu

Organizations: Department of Computer Science, University of California, Davis · University of Virginia

Abstract

Parameter-efficient fine-tuning (PEFT) has been widely used, yet the adapted model suffers catastrophic forgetting of pre-trained knowledge. While most existing work hardens the adapter by constraining where the weight update may lie, we note that constraining this weight channel does not constrain the activation channel. In this paper, we propose Concept-Guided Retention (CGR), a plug-and-play regularizer that constrains both for any PEFT base. CGR reads a frozen sparse autoencoder (SAE) and treats two disjoint concept sets differently. It softly confines the update to the task-concept subspace (via weight channel), and anchors a selected set of general-purpose concepts to the frozen model (via activation channel). We also provide anchor value bounds based on weight updates to understand how they interact, and show that both terms are necessary. On a CLIP vision panel, CGR significantly improves retention at comparable accuracy for every base, and the same pattern holds for world-knowledge retention under Llama-3.1-8B instruction tuning. Because the protected directions are named SAE concepts, CGR remains inspectable and controllable.

Figures & tables

Panel of Figure 1 — Figure 1: CGR confines the weight update to task concepts and anchors retained concepts to the frozen model, both read from one frozen SAE. Left: one pass of task data through the frozen backbone and SAE selects the task set $\mathcal{S}_{\mathrm{tsk}}$ by causal attribution and a disjoint retention set $\mathcal{S}_{\mathrm{ret}}$ by firing presence. Right: two CGR regularizers attach to any PEFT base. Confine penalizes the part of $\Delta W$ outside the task subspace and anchor holds the retention-concept read-out at its frozen-model values.

Method	PEFT	Weight	Activ.	Pers.	Split	Bound	V+L
OPLoRA	✓	hard	$\times$	✓	$\times$	$\times$	$\times$
LoRA-Null	✓	hard	$\times$	$\times$	$\times$	$\times$	$\times$
CorDA	✓	hard	$\times$	$\times$	$\times$	$\times$	$\times$
SC-LoRA	✓	hard	$\times$	$\times$	$\times$	$\times$	$\times$
SAILS	✓	hard	$\times$	$\times$	$\times$	$\times$	$\times$
CWAC	$\times$	hard	soft	✓	$\times$	$\times$	$\times$

Table 1: Mechanism coverage (as reported by each paper). Weight / Activ.: form of the weight- / activation-space retention term, hard (exact projection or initialization) vs. soft (penalty);

\times

: channel unconstrained; Role split: disjoint task and retention concept sets with opposite treatments; Bound: a stated bound on the retained capability’s drop.

Figure 2: CGR pushes the attainable-retention frontier toward the ceiling (CLIP vision panel). One point per (task, base) of Table 2 where gray is for the base and red for

{+}

CGR . The shaded band shows the retention CGR adds.

	FGVC		CUB		DTD		Flow		Pets		EuroS		RESISC		DiabR		dSpr-O		SVHN		Avg
Method	Acc	Ret	Acc	Ret	Acc	Ret	Acc	Ret	Acc	Ret	Acc	Ret	Acc	Ret	Acc	Ret	Acc	Ret	Acc	Ret	Ret
Zero-shot	19.3	82.6	70.9	82.6	54.6	82.6	73.0	82.6	89.8	82.6	40.5	82.6	61.9	82.6	35.4	82.6	9.3	82.6	60.4	82.6	82.6
LoRA	59.3	67.9	74.6	70.1	71.5	74.1	93.2	76.3	82.6	73.8	93.6	78.3	83.0	76.7	72.3	78.6	43.6	70.9	89.4	73.8	$74.1$
${+}$ CGR	60.1	78.0	76.4	78.7	74.3	80.5	95.0	81.6	86.1	80.8	93.4	81.2	84.0	81.8	72.9	81.2	42.9	77.9	85.8	80.6	$80.2$ ⋆
OPLoRA	58.9	69.9	75.1	69.3	72.1	72.1	93.4	76.9	81.5	75.9	95.1	78.9	82.7	77.3	72.6	76.4	44.2	72.2	89.8	74.6	$74.4$
${+}$ CGR	59.5	79.2	76.6	79.2	74.6	80.5	95.4	81.6	85.7	81.1	93.4	81.4	84.3	82.1	73.3	81.7	43.4	80.0	86.3	81.5	$80.8$ ⋆

Table 2: Results of CGR on ten CLIP ViT-B/32 tasks. Each adapter is shown with and without CGR . AF is AdaptFormer. Acc is target-task accuracy. Ret is zero-shot retention, the mean accuracy on four held-out probes. Avg is the mean retention over the tasks. Best retention per task and best Avg in bold. The star ( ⋆ ) marks a statistically significant CGR retention gain across tasks.

	World-knowledge retention (%)					Target
Method	Trivia	NQ-O	WebQS	Avg1	Per	MTB
Zero-shot	61.5	7.9	11.6	27.0	100.0	–
LoRA	27.6	11.1	12.4	17.1	81.6	5.8
+ CGR	43.5	12.0	12.4	22.6	90.2	6.2
LoRA-Null	47.0	9.5	9.8	22.1	87.1	3.5
+ CGR	55.9	9.1	9.7	24.9	91.5	3.3

Table 3: Results of CGR on Llama-3.1-8B instruction-following tasks. Each adapter is shown with and without CGR . Avg1 is the average preservation score. Per is the percentage of preserved performance relative to the pre-trained model. MTB is the MTBench score. Trivia and NQ-O abbreviate TriviaQA and NQ-Open.

	FGVC		CUB
Configuration	Acc	Ret	Acc	Ret
LoRA (no CGR )	$59.3$	$53.1$	$74.6$	$59.5$
Full CGR ∗	$\mathbf{60.1}$	$\mathbf{72.0}$	$\mathbf{76.4}$	$\mathbf{73.2}$
Confine only	$59.8$	$66.3$	$75.6$	$66.9$
Anchor only	$59.2$	$66.2$	$75.6$	$70.6$
SAE-subspace init only	$59.0$	$50.4$	$74.4$	$59.0$

Table 4: Contribution of each CGR component on top of a LoRA base. Ret is CIFAR-100 zero-shot retention.

Anchor set	Acc	Ret	Acc	Ret	Anch. act.
	FGVC		CUB
Top- $k$ firing (ours)	$\mathbf{60.1}$	$\mathbf{72.0}$	$\mathbf{76.4}$	$\mathbf{73.2}$	$53.3$
Random selection	$59.6$	$69.2$	$\mathbf{76.4}$	$70.6$	$3.6$
Last- $k$ (least firing)	$59.4$	$64.8$	$76.1$	$67.9$	${<}0.1$
SAE-FD ( Zhang et al. 2026 )	$59.6$	$66.6$	$75.9$	$68.6$	$100$
All off-task concepts	$60.0$	$66.4$	$75.9$	$68.6$	$98.3$

Table 5: Concept-selection ablation on CLIP for retention on top of LoRA Base. Ret is CIFAR-100 zero-shot retention. Anchored activation (%) is the share of the total SAE activation on the task inputs that the anchored set accounts for.

Anchor set	Trivia	NQ-O	WebQS	Avg1
Top- $k$ firing (ours)	43.5	12.0	12.4	22.6
Random selection	43.4	10.7	12.1	22.1
SAE-FD ( Zhang et al. 2026 )	42.1	11.1	12.1	21.7

Table 6: Concept-selection ablation on Llama-3.1-8B for retention on top of LoRA base.

Figure 3: The quantity the

\mathrm{anchor}

reads predicts realized retention (FGVC). Measured

\mathcal{S}_{\mathrm{ret}}

drift vs. zero-shot retention (CIFAR-10/100 mean) over all runs.

Figure 4: CGR pins the concepts that drift most under plain LoRA (FGVC). Each concept is named by its top-firing Caltech-101 exemplars. Bars show the activation drift under LoRA (gray) and under ${+}$ CGR (red).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

	Fine-tuning tasks										Retention probes
	FGVC	CUB	DTD	Flow	Pets	EuroS	RESISC	DiabR	dSpr-O	SVHN	C-10	C-100	Calt-101	SUN397
Classes	$100$	$200$	$47$	$102$	$37$	$10$	$45$	$5$	$16$	$10$	$10$	$100$	$101$	$397$

Appendix

Table 7: Vision task datasets. Class counts for the ten fine-tuning tasks and the four held-out retention probes, with column abbreviations as in Table 2 . Training budgets are given in the text. Probes are evaluated zero-shot only and are never fine-tuning tasks.

Backbone & frozen SAE
	Vision (CLIP)	Language (Llama)
Backbone	CLIP ViT-B/32	Llama-3.1-8B (bf16)
SAE site	resid., block $8$	resid., layer $19/32$
SAE	Prisma ( $64\times$ , ReLU $+$ L1)	Llama-Scope ( $8\times$ , top- $32$ )
$\|\mathcal{S}_{\mathrm{tsk}}\|,\|\mathcal{S}_{\mathrm{ret}}\|$	$128,128$	$256,128$
Optimization

Appendix

Table 8: Hyperparameters. Shared across all adapter families unless noted; CGR adds only the two loss coefficients and the concept-set sizes. The anchor coefficient

\lambda_{\mathrm{anc}}

differs across modalities because the anchor is a mean squared activation drift, whose scale is set by the backbone’s residual-stream magnitude, not a tuned trade-off.

Figure 5: The weight update concentrates on the task-driven concepts (FGVC). Left: per-concept residual-write projection $\|W_{dec}[:,j]^{\top}\Delta W\|$ under a plain LoRA adapter (horizontal) against the same quantity under CGR (vertical), on a linear scale. Blue marks the task set $\mathcal{S}_{\mathrm{tsk}}$ , red the retention set $\mathcal{S}_{\mathrm{ret}}$ , and light gray all remaining concepts. Both sets are computed on the frozen backbone, independently of either adapter, so the same partition applies to both axes. The diagonal marks equal write under both methods. A concept above it receives more update under CGR than under LoRA, a concept below it less. LoRA alone does not separate the two sets (both horizontal-axis means are $0.20$ ). CGR lifts the task concepts above the diagonal (from $0.20$ under LoRA to $0.26$ under CGR ) and drops the retention concepts below it (from $0.20$ to $0.11$ ), so the update is reallocated, not enlarged. Right: the four task concepts CGR concentrates its update on, named by their top-activating FGVC images. Bars are blue throughout because all four belong to the task set, hollow under LoRA and solid under CGR .

Figure 6: Weight-space control and activation-space drift across task and retention concepts. Each point corresponds to one SAE concept, grouped into the task set $\mathcal{S}_{\mathrm{tsk}}$ and the retention set $\mathcal{S}_{\mathrm{ret}}$ . The left column reports the weight change magnitude, $\|W_{\mathrm{dec}}[:,j]^{\top}\Delta W\|$ . The right column reports the corresponding activation drift, $\mathbb{E}_{x}|f_{j}(h_{\mathrm{ft}}(x))-f_{j}(h_{0}(x))|$ . Horizontal lines denote per-set means, and the boxes report task accuracy and CIFAR-100 zero-shot retention. Confine concentrates the weight update on the task concepts and suppresses its projection onto the retention concepts, but does not by itself prevent retention-concept activation drift. Anchor directly suppresses activation drift, including when applied only to the selected retention set. Combining the two terms in full CGR separates their roles: task concepts remain free to support adaptation, while the retention concepts constrain the resulting representation change. Anchoring all concepts instead reduces this flexibility and yields lower task accuracy than selectively anchoring $\mathcal{S}_{\mathrm{ret}}$ . Results are means over three seeds; off-scale retention-concept drifts are clipped.

Figure 7: Drift over training: full CGR pins the activation channel and confines the weight write on all three datasets. Columns: FGVC-Aircraft, CUB-200, DTD (

3

bases

\times

weight-only dashed /

+\textsc{CGR}

solid, mean

\pm

std over

3

seeds, rows share y-limits). Top row: every weight-only method’s retention-concept activation drift climbs (to

{\sim}0.06

–

0.09

) while all

+\textsc{CGR}

arms stay pinned at

{\sim}0.01

. Bottom row: the confine term also bounds the adapter write magnitude on every base (e.g. LoRA-Null on FGVC

37\!\to\!22

), so the drift gap is not explained by larger weight moves.

Figure 8: Drift predicts forgetting: replication (CUB-200, DTD). Per-run measured

\mathcal{S}_{\mathrm{ret}}

activation drift vs. zero-shot retention (CIFAR-10/100 mean), for every drift-instrumented run per dataset. Anchored runs are red, weight-only runs are open. The negative relation from Fig. 3 holds on both datasets.

Figure 9: Selected concepts are causally load-bearing. In each trained model we zero the

|\mathcal{S}_{ret}|=128

selected retention concepts at layer 8, subtracting their SAE contribution from the residual stream with no retraining, and re-measure zero-shot retention. The control applies the same surgery to

128

random alive concepts. Bars show retention drop per natural probe (mean

\pm

std over

24

FGVC runs, with the intervention instrumenting three of the four retention probes and SUN397 excluded). The random set is inert. Removing

\mathcal{S}_{ret}

destroys natural-image retention.

Method	CIFAR-10	CIFAR-100	Caltech-101	SUN397	Avg
Zero-shot (no FT)	93.8	78.2	90.9	67.6	82.6
LoRA	$84.3$ $\pm 4.1$	$63.1$ $\pm 5.8$	$87.3$ $\pm 2.5$	$61.6$ $\pm 2.7$	$74.1$ $\pm 3.5$
${+}$ CGR ∗	$91.4$ $\pm 1.8$	$74.0$ $\pm 2.2$	$89.5$ $\pm 1.9$	$66.0$ $\pm 1.1$	$80.2$ $\pm 1.5$
OPLoRA	$84.5$ $\pm 3.4$	$63.3$ $\pm 5.5$	$87.6$ $\pm 2.3$	$61.9$ $\pm 2.7$	$74.4$ $\pm 3.3$
${+}$ CGR ∗	$92.0$ $\pm 1.2$	$75.2$ $\pm 1.4$	$90.0$ $\pm 1.2$	$66.2$ $\pm 1.2$	$80.8$ $\pm 1.1$
LoRA-Null	$89.9$ $\pm 2.5$	$72.5$ $\pm 4.6$	$89.6$ $\pm 1.9$	$66.5$ $\pm 1.3$	$79.6$ $\pm 2.0$

Appendix

Table 9: Per-probe zero-shot retention (%). The four natural probes that make up the Ret column of Table 2 , each shown separately; every cell is meaned over the same

10

fine-tuning tasks and

3

seeds, so each row’s Avg matches its Avg retention in Table 2 . Each cell shows mean

{\pm}

std, the spread across the

10

tasks (as in Table 2 ; seed noise

{\leq}0.9

); CGR tightens it on every probe. Best per column bold;

{}^{*}\!=\!

ours.

Figure 10: The selected head leads the control at every level. Both heads see only concept values, so these accuracies are not comparable to the fine-tuned models of Table 2 .

Figure 11: Steering the selected concepts controls retention (FGVC). On the frozen backbone we scale the selected

\mathcal{S}_{\mathrm{ret}}

concepts’ layer-8 contribution by

\alpha

(red). Retention peaks at the unmodified model (

\alpha{=}1

) and falls on both sides, toward removal at

\alpha{=}0

and under amplification at

\alpha{=}1.5

. A size-matched random control is flat (gray). Retention is averaged over the CIFAR-10 and CIFAR-100 probes.

	World-knowledge retention (%)
Method	Trivia	NQ-O	WebQS	Avg1	Per
Zero-shot	$61.5$	$7.9$	$11.6$	$27.0$	$100.0$
LoRA	$27.6$ $\pm 1.2$	$11.1$ $\pm 0.6$	$12.4$ $\pm 0.7$	$17.1$ $\pm 0.7$	$81.6$ $\pm 0.7$
+ CGR	$43.5$ $\pm 2.2$	$12.0$ $\pm 0.8$	$12.4$ $\pm 0.7$	$22.6$ $\pm 0.4$	$90.2$ $\pm 1.1$
LoRA-Null	$47.1$ $\pm 0.3$	$9.5$ $\pm 0.3$	$9.8$ $\pm 0.1$	$22.1$ $\pm 0.1$	$87.1$ $\pm 0.3$
+ CGR	$55.9$ $\pm 0.3$	$9.1$ $\pm 0.2$	$9.7$ $\pm 0.2$	$24.9$ $\pm 0.1$	$91.5$ $\pm 0.8$

Appendix

Table 11: Seed variability of the Llama-3.1-8B retention results. Every cell of Table 3 is reported as mean

\pm

sample std over seeds. Per is capped per seed here, whereas Table 3 applies the cap to the seed mean, which shifts Per by at most

1.0

where a seed exceeds the zero-shot base. MTBench is single-seed by design and omitted.

Natural		Specialized / Structured
Task	EV	Task	EV
Caltech-101	0.87	RESISC-45	0.74
SUN397	0.86	EuroSAT	0.69
DTD	0.85	Diabetic-Retino	0.49
CUB-200	0.82	dSprites-loc/ori	0.23
Oxford-Pets	0.81

Appendix

Table 12: SAE reconstruction EV (layer-8 CLS, 1000 samples, frozen backbone). Higher

=

SAE in-distribution, concepts faithful. Natural tasks are high-EV; the structured/synthetic task (dSprites) is low-EV.

	$\lambda_{\mathrm{anc}}$	10	30	60	100	150
FGVC	Acc	$59.4$ $\pm 1.0$	$58.8$ $\pm 1.7$	$60.1$ $\pm 0.4$	$59.8$ $\pm 0.2$	$59.4$ $\pm 0.3$
FGVC	C-100 ret.	$69.0$ $\pm 0.6$	$70.0$ $\pm 1.1$	$72.0$ $\pm 0.8$	$73.3$ $\pm 0.8$	$73.8$ $\pm 0.4$
CUB-200	Acc	$76.0$ $\pm 0.2$	$76.3$ $\pm 0.1$	$76.4$ $\pm 0.3$	$76.3$ $\pm 0.2$	$76.2$ $\pm 0.2$
CUB-200	C-100 ret.	$71.2$ $\pm 0.6$	$72.1$ $\pm 0.5$	$73.2$ $\pm 0.7$	$74.2$ $\pm 0.5$	$74.0$ $\pm 0.6$

Appendix

Table 13: Anchor-strength sweep ( CGR on LoRA r8). Retention rises with

\lambda_{\mathrm{anc}}

and flattens above

100

on both datasets, at accuracy within seed noise. The default

\lambda_{\mathrm{anc}}{=}60

reproduces the full- CGR row of Table 4 . Mean over

3

seeds,

\pm

std.

		Anchor only	$\lambda_{\mathrm{conf}}$				Hard proj. ${+}$ anchor
		Anchor only	$0.03$	$0.1$	$0.3$	$1.0$	Hard proj. ${+}$ anchor
FGVC	Acc	$59.2$	$60.4$ $\pm 0.9$	$59.4$ $\pm 1.0$	$60.7$ $\pm 1.0$	$60.4$ $\pm 0.7$	$58.4$
FGVC	C-100 ret.	$66.2$	$70.3$ $\pm 0.6$	$72.1$ $\pm 0.3$	$71.8$ $\pm 0.5$	$70.9$ $\pm 0.9$	$63.8$
CUB-200	Acc	$75.6$	$76.2$ $\pm 0.1$	$76.0$ $\pm 0.4$	$76.4$ $\pm 0.2$	$76.3$ $\pm 0.4$	$75.2$
CUB-200	C-100 ret.	$70.6$	$73.0$ $\pm 1.5$	$72.9$ $\pm 1.2$	$73.2$ $\pm 1.2$	$73.1$ $\pm 1.6$	$69.4$

Appendix

Table 14: Confine-strength sweep ( CGR on LoRA r8). Accuracy and CIFAR-100 zero-shot retention stay on a plateau over a

33\times

range of

\lambda_{\mathrm{conf}}

, and the default

\lambda_{\mathrm{conf}}{=}0.1

sits inside it. The flanking columns give the endpoints. Anchor alone is the

\lambda_{\mathrm{conf}}{=}0

limit, and replacing the soft confine with a hard projection onto the task subspace retains less still. Every setting inside the sweep beats both.

	$\|\mathcal{S}_{\mathrm{ret}}\|$	32	64	128	256	512
FGVC	Acc	$59.4$ $\pm 0.4$	$60.0$ $\pm 0.5$	$60.1$ $\pm 0.4$	$59.7$ $\pm 0.6$	$59.8$ $\pm 0.8$
FGVC	C-100 ret.	$72.8$ $\pm 0.4$	$73.2$ $\pm 0.4$	$72.0$ $\pm 0.8$	$71.8$ $\pm 0.5$	$70.9$ $\pm 0.8$
CUB-200	Acc	$76.2$ $\pm 0.6$	$76.3$ $\pm 0.2$	$76.4$ $\pm 0.3$	$76.3$ $\pm 0.1$	$76.0$ $\pm 0.1$
CUB-200	C-100 ret.	$74.0$ $\pm 0.4$	$73.3$ $\pm 0.8$	$73.2$ $\pm 0.7$	$72.7$ $\pm 0.7$	$72.2$ $\pm 0.7$

Appendix

Table 15: Retention-set size sweep ( CGR on LoRA r8). Accuracy and CIFAR-100 zero-shot retention are flat over a

16\times

range of

|\mathcal{S}_{\mathrm{ret}}|

on both FGVC-Aircraft and CUB-200. The default

k{=}128

sits inside the plateau.

	$\|\mathcal{S}_{\mathrm{tsk}}\|$	32	64	128	256	512
FGVC	Acc	$60.3$ $\pm 0.6$	$59.9$ $\pm 1.3$	$60.1$ $\pm 0.4$	$60.0$ $\pm 0.7$	$59.9$ $\pm 0.5$
FGVC	C-100 ret.	$75.3$ $\pm 1.3$	$73.2$ $\pm 1.0$	$72.0$ $\pm 0.8$	$70.8$ $\pm 0.4$	$67.2$ $\pm 1.1$
CUB-200	Acc	$77.0$ $\pm 0.1$	$77.0$ $\pm 0.3$	$76.4$ $\pm 0.3$	$76.1$ $\pm 0.3$	$76.0$ $\pm 0.2$
	C-100 ret.	$75.7$ $\pm 1.6$	$75.0$ $\pm 0.6$	$73.2$ $\pm 0.7$	$72.0$ $\pm 1.1$	$70.7$ $\pm 0.8$

Appendix

Table 16: Task-set size sweep ( CGR on LoRA r8). Accuracy is flat over a

16\times

range of

|\mathcal{S}_{\mathrm{tsk}}|

on both datasets. Bounding the task set improves retention at no accuracy cost. The default

|\mathcal{S}_{\mathrm{tsk}}|{=}128

matches

|\mathcal{S}_{\mathrm{ret}}|

and is not tuned.

Model and adaptation
Symbol	Meaning
$W_{0}$	Frozen pretrained weight at the adapted site, $W_{0}\in\mathbb{R}^{d\times d_{in}}$
$\Delta W$	Additive weight update; $\Delta W{=}BA$ for LoRA
$B,A$	LoRA factors, $B\in\mathbb{R}^{d\times r}$ , $A\in\mathbb{R}^{r\times d_{in}}$
$r$	LoRA rank
$\ell$	Fixed read-out layer, never tuned

Appendix

Table 17: Notation used throughout the paper.

Fine-Tuning

Fine-tuning adapts pretrained models to new tasks, domains, and behaviors by updating some or all of their parameters.

See all 250 papers

Aug 12, 2026

AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention

Multimodal Large Language Models (MLLMs) exhibit strong generalization and reasoning abilities due to large-scale multimodal pre-training. However, fine-tuning these models on downstream tasks often leads to catastrophic forgetting, where newly learned task-specific knowledge degrades previously acquired capabilities. This issue arises because gradient updates for new tasks overwrite parameters critical to prior knowledge, limiting the practical deployment of MLLMs. To address this challenge, we propose Activation-Weighted Adaptive REtention (AWARe), a fine-tuning method that mitigates catastrophic forgetting by dynamically controlling parameter updates based on activation patterns. AWARe assigns activation-based importance scores to parameters, selectively freezing those essential for preserving prior capabilities while allowing less important parameters to adapt to new tasks. Importantly, AWARe operates without modifying model architectures, ensuring compatibility with existing inference engines. Extensive experiments demonstrate that AWARe effectively preserves upstream capabilities while achieving superior downstream performance compared to existing methods. Code is available at https://github.com/kaln27/AWARe.

Juncheng Liao, Jinfan Lv, Guoming Wang +3

School of Software Technology, Zhejiang University, Hangzhou, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China · School of Artificial Intelligence, Sun Yat-Sen University, Guangzhou, China +1

Continual Learning for LLMs LLM Fine-Tuning Multimodal Large Language Models Catastrophic Forgetting

Ask PDF

Oct 6, 2026

Are Parameter-Efficient Fine-tuning Methods Really Different?

Parameter-efficient fine-tuning (PEFT) offers many parameterizations, yet their methodological and functional differences remain unclear. We compare six methods in language and diffusion models to examine how their parameterizations relate to task performance, forgetting, and changes in pretrained weight geometry. Motivated by the spectrum-preserving design of orthogonal fine-tuning (OFT), we first ask whether spectral preservation is itself important for adaptation and retention. We find that the selected LoRA-family methods also approximately preserve pretrained geometry, and that restoring their slightly drifted singular-value spectra largely preserves task performance, questioning the necessity of explicit geometric preservation. Beyond this, we observe that some methods exhibit distinct adaptation--retention trade-offs that vary across settings: LoRA most consistently limits forgetting at competitive performance, DoRA achieves higher mean task scores than LoRA in most comparisons, while PiSSA often incurs greater retention costs. Further intervention experiments suggest that while performance gains from different PEFT methods can be attributed to modifications in different groups of spectral components, we consistently find that restoring dominant rather than intermediate or trailing components produces the largest mean reduction in general-text NLL or base-image drift. Together, these results motivate evaluating geometric constraints through their functional consequences rather than preservation alone. Code is available at https://github.com/Kuaaannn/PEFT_methods.

Yikuan Li, Pinyan Lu, Fanghui Liu

Department of Statistics, The University of Warwick, UK. · Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics, China. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.

Parameter-Efficient Fine-Tuning Low-Rank Adaptation Catastrophic Forgetting Diffusion Model Fine-Tuning

Ask PDF

May 27, 2026

PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective

Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize downstream accuracy while overlooking the retention of pretrained capabilities. We argue that PEFT should be assessed through the stability-plasticity dilemma: the trade-off between target-task adaptation and resistance to forgetting. We introduce PEFT-Arena, a benchmark that jointly measures downstream performance and general capability retention. Across methods, we find distinct stability-plasticity profiles; under comparable parameter budgets, orthogonal finetuning achieves the most favorable Pareto frontier. To explain these differences, we analyze PEFT updates from two geometric perspectives. In weight space, spectral analysis reveals how parameterizations interact with the pretrained singular-value structure. In activation space, retention metrics show whether finetuning preserves or distorts general-capability representations, with forgetting linked to non-isometric representation distortion. Finally, an analysis shows that final SFT checkpoints often overshoot a better target-retention operating point. Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.

Yangyi Huang, Ruotian Peng, Zeju Qiu +4

The Chinese University of Hong Kong · Westlake University · MPI for Intelligent Systems

LLM Evaluation Fine-Tuning Parameter-Efficient Fine-Tuning Catastrophic Forgetting

Ask PDF

Abstract

Fine-Tuning

Explore similar work

AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention

Are Parameter-Efficient Fine-tuning Methods Really Different?

PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective