Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix

Yu Sun; Junwei Zhou; Zuodong Xiang; Yike Zhang; Pengcheng Xu; Xinyu Guan; Ruoyun Ma; Hailu Xu

cs.LGOct 8, 2026

Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix

Authors: Yu Sun, Junwei Zhou, Zuodong Xiang, Yike Zhang, Pengcheng Xu, Xinyu Guan, Ruoyun Ma, Hailu Xu

Organizations: University of Pennsylvania · Matter Innovation Inc. · University of California, Davis · Johns Hopkins University ·

Abstract

Parameter-efficient continual learning updates a restricted subset of adapter parameters to incorporate sequential tasks while limiting catastrophic forgetting. Gradient attribution is a common rule for selecting that subset, but it requires an additional backward pass over calibration data. We identify a parameter-pool confound in comparisons between initialization-time gradient masks and unrestricted random masks under standard zero-initialized LoRA ( $B=0$ ): the gradient with respect to $A$ is identically zero ( $\partial\mathcal{L}/\partial A=0$ ), so gradient masks select strictly from the $B$ matrix whereas unrestricted random masks draw from both $A$ and $B$ . Such a comparison changes both the selection rule and the candidate parameter pool. Isolating the selection criterion by restricting both approaches to the same parameter pool, we find a negative result: a uniform random mask consistently achieves lower loss-based backward transfer than gradient attribution across seven seeds on Qwen2.5-3B. The effect is concentrated on the medical task, which contributes 80% of the aggregate; code and math move in the same direction, and legal shows no reliable difference. This advantage is consistent with a stability--plasticity trade-off under the tested task sequence, in which random masking slightly reduces single-task fitting. On the two discrete-option tasks (Medical and Legal), the mean accuracy-BWT difference is not statistically distinguishable from zero.

Figures & tables

Figure 1: The density-matched comparison. (a) Both arms draw 10% of the rank-16 lora_B pool, so they train exactly the same 1.6M parameters and differ only in how those coordinates are chosen; because

B=0

at initialization, the gradient score is zero for every lora_A coordinate, and any baseline that draws from both pools changes the pool as well as the criterion. (b) Under that matched density the random mask achieves lower loss-based BWT than the gradient mask in all seven seeds.

Symbol	Meaning
$f_{\theta}$ , $W_{0}$	base model and its frozen pretrained weight matrix
$B$ , $A$	low-rank LoRA factors, $B\in\mathbb{R}^{d_{\text{out}}\times r}$ , $A\in\mathbb{R}^{r\times d_{\text{in}}}$
$r$ , $\alpha$	LoRA rank ( $r=16$ ) and scaling ( $\alpha=32$ )
$W$	adapted weight, $W=W_{0}+\alpha BA$
$t$ , $T$	task index and number of tasks ( $T=5$ )
$\mathcal{D}_{t}$	training split of task $t$

Table 1: Notation. Indices run over tasks

t=1,\dots,T

with

T=5

, and over prior tasks

j\in\mathcal{P}=\{1,\dots,T-1\}

Figure 2: Per-seed within- lora_B differences (

\Delta=\text{Random-B}-\text{Grad-Full}

, loss-based BWT, 7 seeds). All seven are negative (Random-B forgets less), mean

\Delta=-0.072

(orange line); seed 42 is the largest effect (

-0.165

). Leave-one-out excluding seed 42 remains significant (paired

t(5)

p=0.015

Seed	Random-B BWT $\downarrow$	Gradient-B BWT $\downarrow$	$\Delta$ (matched)	$\Delta$ (unmatched)
42	$\mathbf{0.278}$	0.453	$-0.175$	$-0.220$
123	$\mathbf{0.264}$	0.453	$-0.188$	$-0.119$
456	$\mathbf{0.283}$	0.406	$-0.123$	$-0.131$
789	$\mathbf{0.352}$	0.387	$-0.035$	$-0.074$
1024	$\mathbf{0.352}$	0.474	$-0.121$	$-0.109$
2048	$\mathbf{0.231}$	0.491	$-0.261$	$-0.187$

Table 2: Headline result (D1, density-matched). Both arms select the same 1,651,508 parameters (10% of lora_B ); only the criterion differs. Per-seed loss-based BWT and

\Delta=\text{Random-B}-\text{Grad-B}

(negative = random-B forgets less), seven sessions, one seed each. The last column is the density- unmatched contrast from the same sessions, that is, Random-B against the unrestricted gradient mask (

\Delta=\text{Random-B}-\text{Grad-Full}

) rather than against Grad-B.

Figure 3: The density-matched D1 per-task decomposition, averaged over seven seeds. For each prior task, the bars show the loss-based BWT contribution of Random-B and Grad-B, with the corresponding values reported above each pair. Because the task-level contributions span roughly two orders of magnitude, the left panel uses a broken

y

-axis: the slashes mark the omitted interval from 0.52 to 0.92, and the labels above the Medical bars report their full values (

+0.998

and

+1.493

) rather than truncated heights. Creative is the final task and therefore has no backward-transfer contribution. The right panel shows the aggregate loss-based BWT on a separate scale. Random-B exhibits less loss-based forgetting on Code, Medical, and Math; the Legal difference is negligible and changes sign across seeds. Most of the aggregate difference is concentrated on Medical. Per-seed results are reported in Appendix Q .

Figure 4: Accuracy-based validation. Blue points are the seven per-seed accuracy-BWT differences (Grad-B minus Random-B, percentage points) on the two discrete-option tasks, spanning

-6.98

+6.00

pp; green is the seven-seed mean with its 95% interval

[-2.91,+4.80]

pp. The plotted quantity is a difference of accuracy-BWT (mean final-minus-learned accuracy over prior tasks), a retention measure, not a difference of final accuracies. The interval spans zero, so no statistically reliable accuracy difference is detected, and it reaches past the 4 pp non-inferiority margin, so that criterion is not met. Per-seed values are in Appendix N .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Peak GPU (GB)	Time (s)	Backward?
Gradient mask	$13.8^{{\dagger}}$	584.9	Yes
Wanda proxy	7.4	12.1	No
Shared random / RRM	0 ‡	0	No
No mask	0 ‡	0	No

Appendix

Table 3: Attribution cost on Qwen2.5-3B (BF16). Both columns are from a single full-parameter measurement run ( --include-frozen , --full-precision ); the exact correspondence and its limits are described below. Peak GPU is torch.cuda.max_memory_allocated converted to decimal GB (profiling fields *_peak_alloc_mb = 13168 / 7102 MiB, i.e. 13.8 / 7.4 GB); Time is the per-task attribution wall-clock recorded by the run (584.9 s gradient / 12.1 s Wanda).

Figure 5: Spearman $\rho=0.086$ between Wanda and gradient importance (LoRA parameters) on Qwen2.5-3B. These metrics rank parameters nearly independently.

Seed	Gradient BWT $\downarrow$	Random BWT $\downarrow$	No Mask BWT $\downarrow$
42	0.491	0.435	0.507
123	0.463	0.417	$\mathbf{0.409}$
456	$\mathbf{0.388}$	0.414	0.539
789	$\mathbf{0.357}$	0.417	0.484
1024	0.486	0.458	0.479
Mean	0.437	$\mathbf{0.428}$	0.484

Appendix

Table 4: Per-seed BWT breakdown (Qwen2.5-3B, rank-16 LoRA, 5 tasks).

Method	BWT $\downarrow$	Loss $\downarrow$	Calibration pass?
Shared random	$\mathbf{0.428}\pm 0.019$	$\mathbf{1.940}\pm 0.021$	No
Gradient mask	$0.437\pm 0.061$	$1.966\pm 0.037$	Yes
No mask	$0.484\pm 0.047$	$1.970\pm 0.033$	No

Appendix

Table 5: Confounded comparison (Qwen2.5-3B, rank-16 LoRA, 5 tasks,

n{=}5

seeds). The last column records whether an additional calibration forward–backward pass is required; its time and data costs are described in Appendix B . Across all tables, bold marks the better value in its column.

Seed	random_B_only BWT $\downarrow$	full_alora BWT $\downarrow$	$\Delta$
42	$\mathbf{0.326}$	0.491	$-0.165$
123	$\mathbf{0.320}$	0.413	$-0.093$
456	$\mathbf{0.337}$	0.356	$-0.019$
789	$\mathbf{0.346}$	0.364	$-0.018$
1024	$\mathbf{0.336}$	0.442	$-0.106$
2048	$\mathbf{0.367}$	0.402	$-0.035$

Appendix

Table 6: Within- lora_B comparison (Qwen2.5-3B, rank-16 LoRA, 5 tasks, 7 seeds). All comparisons are within-run (same method ordering and RNG state).

\Delta=\text{random\_B\_only}-\text{Grad-Full}

; negative

\Delta

indicates random-B achieves lower (better) BWT.

Level	What varies	Magnitude	What it measures
(a) protocol-equivalence check	execution path only	$9.6\times 10^{-5}$	numerical agreement between two code paths; not an inferential error term
(b) across-seed paired contrast	independent seed	SD $(\Delta_{s})=0.071$	variability used by the paired significance test
(c) position (ordering) sensitivity	position of the two arms	$0.064$ (seed 42, gradient_B )	method-position interaction in a three-seed check
(d) cross-session variation	session, hardware, and method	up to $0.120$ ( full_alora , seed 42)	method-specific offset of unknown origin

Appendix

Table 7: Distinct quantities that are sometimes all called run-to-run noise . The paired tests use the across-seed SD of the within-session contrasts; the protocol-equivalence discrepancy is an implementation check and does not enter inference. Position and cross-session effects are reported as sensitivities rather than as interchangeable error estimates.

Seed	$\|\Delta_{\text{pos}}(\text{rB})\|$	$\|\Delta_{\text{pos}}(\text{gB})\|$	rB / main	gB / main
42	$1.5\times 10^{-3}$	$6.4\times 10^{-2}$	0.01	0.41
789	$3.1\times 10^{-5}$	$6.2\times 10^{-3}$	0.00	0.04
2048	$8.4\times 10^{-6}$	$1.3\times 10^{-2}$	0.00	0.08

Appendix

Table 8: Method-ordering sensitivity (D1 density-matched, three seeds).

\Delta_{\text{pos}}

is the BWT difference between running at position 1 and position 2. The last two columns express each arm’s positional shift as a fraction of the main D1 contrast (

|\Delta|=0.155

Method	Run A	Run B	Run C	Run D / spread
Gradient mask ( full_alora )	0.491	0.378	0.498	— (0.120)
Gradient-B@10% ( gradient_B )	—	—	0.453	0.423 (0.030)
Random mask ( random_A+B )	0.435	0.349	—	—
Random B-only ( random_B_only )	0.326	0.324	0.278	0.278 (0.048)

Appendix

Table 9: Absolute loss-based BWT for the same seed 42 under identical method definitions, across sessions and hardware. Run C is the D1 density-matched session and Run D the uniform-B suppression control (the M2 controller) session, both on a separate GPU and software stack; random_A+B was not run in either. A Run-D entry exists only for methods Run D ran, and it did not train a plain full_alora arm, so that cell is --- and its gradient-side arm is gradient_B , entered on its own row. Spread is

\max-\min

across the runs in which the method appears.

Condition	RRM BWT	Random BWT	$\Delta$	Direction
Rank 4	0.464	0.423	$+$ 0.041	Random better
Rank 8	0.427	0.373	$+$ 0.055	Random better
Rank 16 (ref.)	0.401	0.435	$-$ 0.034	RRM better
Rank 32	0.414	0.390	$+$ 0.024	Random better
Reverse order	0.115	0.109	$+$ 0.006	Random better

Appendix

Table 10: RRM boundary analysis (Qwen2.5-3B, single seed per condition except rank-16 reference).

Figure 6: Sparsity sweep (Qwen2.5-3B, single seed). Shared random achieves numerically lower BWT than gradient attribution at all four tested levels.

Sparsity	Random BWT $\downarrow$	Gradient BWT $\downarrow$
5%	0.327	0.422
10%	0.435	0.491
20%	0.444	0.474
30%	0.361	0.467

Appendix

Table 11: Sparsity sweep numerical values (Qwen2.5-3B, single seed). Random BWT

\leq

Gradient BWT at all tested levels.

Figure 7: The suppression control’s outcome is seed-dependent . All three arms train 1,651,508 parameters, which is 5.5% of the full LoRA pool (each arm draws 10% of the lora_B sub-pool, and

B

holds 55% of the pool’s parameters), so this is the density-matched setting throughout. The uniform-B controller redistributes the gradient mask’s budget uniformly across LoRA modules. At seed 42 it sits in the gradient band (0.440 above gradient-B’s 0.423 and well above random-B-only’s 0.278); seeds 123 and 456 also land on the gradient side; at seed 789 it falls to the random band (0.355 below gradient-B’s 0.416, alongside random-B-only’s 0.352). Three of four seeds fall on the gradient side. M2 is an independent session from D1, so the same-seed gradient-B values differ between them (e.g. seed 42: 0.423 here vs 0.453 in the D1 table; seed 789: 0.416 vs 0.387; § F ). We therefore do not treat this ablation as establishing the o_proj -concentration mechanism (§ 6 ).

Figure 8: Gradient-mask concentration by LoRA module (Qwen2.5-3B). Gradient attribution selects 29% of o_proj parameters (2.9

\times

the 10% uniform rate) and is under-represented in down_proj (4.1%); the dashed line marks uniform 10% selection.

Module	Gradient density	Random density	Ratio
o_proj	29.0%	10.0%	2.9 $\times$
up_proj	12.8%	10.0%	1.3 $\times$
v_proj	10.3%	10.0%	1.0 $\times$
q_proj	8.1%	10.0%	0.8 $\times$
k_proj	8.0%	10.0%	0.8 $\times$
gate_proj	8.0%	10.0%	0.8 $\times$

Appendix

Table 12: Gradient mask concentration by LoRA module (Qwen2.5-3B). Random masking distributes uniformly at 10% per module. Gradient attribution selects exclusively lora_B parameters (all lora_A receive zero importance due to

B=0

initialization).

Scale	Random BWT $\downarrow$	Gradient BWT $\downarrow$	No Mask BWT $\downarrow$
0.5B	0.343	—	0.431
1.5B	0.309	—	0.356
7B	0.447	0.441	0.514

Appendix

Table 13: BWT at additional scales (single seed, observational).

Architecture	Comparison	Grad. BWT $\downarrow$	Rand. BWT $\downarrow$	$\Delta$
Qwen2.5-3B ( $n{=}5$ )	Confounded	$0.437\pm 0.061$	$0.428\pm 0.019$	$-0.009$
Phi-2 ( $n{=}5$ )	Confounded	$0.158\pm 0.019$	$0.183\pm 0.016$	$+0.025$
Qwen2.5-3B ( $n{=}7$ )	Within-B	$0.409\pm 0.046$	$0.338\pm 0.015$	$-0.072$
Phi-2 ( $n{=}4$ )	Within-B	$0.149\pm 0.036$	$0.065\pm 0.017$	$-0.084$

Appendix

Table 14: Cross-architecture comparison on Qwen2.5-3B and Phi-2 (2.78B, GPT-NeoX-style), with both arms within a row sharing one target set. Qwen2.5-3B targets q/k/v/o/gate/up/down_proj (pool

29{,}933{,}568

), whereas Phi-2 uses GPT-NeoX-style q,k,v,fc1,fc2,dense projections (pool

23{,}592{,}960

) in the confounded design. In the within- lora_B design, the Phi-2 rows omit dense (pool

20{,}971{,}520

, 12.5% smaller), so the two Phi-2 designs are not comparable row-for-row. The within- lora_B rows are matched on pool but not density (the gradient mask activates approximately

2\times

the B parameters of Random-B), as on Qwen2.5-3B.

Method	Unrestricted BWT	B-only BWT	Gap
Gradient	0.378	0.378	$+0.000$
Random	0.349	0.324	$-0.025$

Appendix

Table 15: Effect of restricting the mask to lora_B (seed 42). For gradient attribution, the restriction is a no-op by construction (B=0 collapses every lora_A importance score to zero); for random, restricting to B removes lora_A dilution and improves BWT by 0.025. This asymmetry demonstrates that the within-B comparison varies the selection criterion, not the parameter pool.

Seed	Accuracy-BWT $\Delta$ [Grad $-$ Rand], pp	Loss BWT $\Delta$ [Grad $-$ Rand]
42	$+2.00$	$+0.2320$
123	$0.00$	$+0.1647$
456	$+0.48$	$+0.1300$
789	$-6.98$	$+0.0786$
1024	$+4.60$	$+0.0514$
2048	$+0.51$	$+0.2572$

Appendix

Table 16: Per-seed results, both columns in the direction [Grad

-

Random] (positive = gradient forgets more). Col. 2 is the accuracy-BWT difference: the difference in mean final-minus-learned accuracy over the prior discrete-option tasks, not a difference of final accuracies. It is therefore a retention quantity on the same footing as the loss column and does not mix learning with retention. Col. 2 is reported in percentage points and is measured in the A800 accuracy runs; Col. 3 is the loss-based BWT difference from the A800 loss runs. Together they form a second environment relative to the original V100 loss runs. Col. 2 carries the 95%

t

-interval on the seven-seed mean. Col. 3 has mean

+0.152

(SD

0.0749

t=5.37

, CI

[+0.083,+0.221]

, 7/7 same direction). The V100 loss contrast is

+0.155

under the same [Grad

-

Random] convention. Across matched seeds, the A800-minus-V100 loss-contrast difference has mean

-0.0033

and SD

0.0441

, so individual seeds move between environments even though the average does not.

Method	B=0 pool	Within-pool grad-vs-random
EWC / SI / MAS	No	No
PiSSA ( $B\neq 0$ )	N/A by init	No
Attribution-Guided CL (post-training attribution)	N/A by design	No
Ours (within- lora_B )	Yes	Yes

Appendix

Table 17: Positioning against gradient-attribution families. “B=0 pool” marks a method whose attribution is computed at a zero-initialized LoRA, where the gradient with respect to lora_A is zero and the mask is confined to lora_B ; “within-pool grad-vs-random” marks a method compared against a matched random baseline drawn from the same pool. Attribution-Guided CL computes its attribution from a separately trained model (

B\neq 0

at scoring time) and gates gradients continuously rather than selecting a binary mask, so it is outside the first axis by construction and unmatched on the second.

Rule	What it scores, and what for	Run
EWC [ Kirkpatrick et al., 2017 ] ; MAS [ Aljundi et al., 2018 ]	Fisher information, output sensitivity; a penalty, not a mask	no
SNIP [ Lee et al., 2019 ] ; GraSP [ Wang et al., 2020 ]	gradient at initialisation; one-shot pruning mask	no
Wanda [ Sun et al., 2024 ]	weight $\times$ activation magnitude; one-shot pruning mask	no
LoRA-FA [ Zhang et al., 2023 ]	none, structural; freezes lora_A , trains lora_B	no
Grad-Full, Grad-B (§ 3.3 )	calibration gradient magnitude; static CL mask, full pool (Grad-Full) or lora_B at 10% (Grad-B)	yes
Random-B; Random A+B [ Xu and Zhang, 2024 ]	none, uniform; static CL mask, lora_B at 10% (Random-B) or full pool (A+B)	yes

Appendix

Table 18: Selection rules this paper is positioned against. The last column marks the four rules we train and evaluate under one protocol (§ 3.3 ); the rest are compared by design only, and none of their published numbers is cited as a result of ours.

Seed	Code	Medical	Legal	Math	Aggregate
42	$-0.040$	$-0.409$	$-0.002$	$-0.249$	$-0.175$
123	$-0.002$	$-0.698$	$-0.002$	$-0.052$	$-0.188$
456	$-0.074$	$-0.381$	$-0.004$	$-0.033$	$-0.123$
789	$-0.063$	$-0.054$	$+0.001$	$-0.023$	$-0.035$
1024	$-0.044$	$-0.410$	$-0.011$	$-0.021$	$-0.121$
2048	$-0.041$	$-0.950$	$+0.010$	$-0.062$	$-0.261$

Appendix

Table 19: Per-seed, per-task D1 decomposition (

\Delta=\text{random-B}-\text{gradient-B}

for each prior task). Negative favours random-B. Legal is the only task whose sign is not consistent across seeds.

Task	random_B_only	gradient_B	full_alora
code	$0.952\pm 0.111$	$\mathbf{0.901\pm 0.100}$	$0.898\pm 0.099$
medical	$1.190\pm 0.040$	$\mathbf{1.152\pm 0.040}$	$1.144\pm 0.035$
legal	$0.388\pm 0.035$	$\mathbf{0.368\pm 0.038}$	$0.365\pm 0.037$
math	$3.838\pm 0.103$	$\mathbf{3.652\pm 0.096}$	$3.642\pm 0.100$
creative	$2.104\pm 0.053$	$\mathbf{2.067\pm 0.053}$	$2.056\pm 0.054$

Appendix

Table 20: Diagonal loss (mean

\pm

std over the seven D1 seeds), by task. Lower is better. gradient-B is lower than random-B-only in every cell (Table 20 ); Grad-Full is the uncontrolled reference. Loss diagonals only; the D1 run records contain no accuracy matrix (accuracy is measured separately in the accuracy branch, § 5.4 ).

Seed	rB start	rB end	gB start	gB end (diff)
42	1.543	1.821	1.478	1.931 ( $+0.110$ )
123	1.632	1.896	1.553	2.006 ( $+0.110$ )
456	1.575	1.858	1.494	1.900 ( $+0.042$ )
789	1.548	1.900	1.475	1.862 ( $-0.038$ )
1024	1.566	1.918	1.489	1.962 ( $+0.045$ )
2048	1.665	1.896	1.587	2.078 ( $+0.182$ )

Appendix

Table 21: Mean final held-out loss over the four prior tasks, per seed, read from the final row of the stored loss matrix. Start is the same quantity at each task’s own training step; end is at the end of the sequence. gradient-B’s endpoint is higher in 6 of 7 seeds (seed 789 is the exception); paired

t(6)=3.03

p=0.023

, mean

+0.081

on the V100, and

+0.078

(

p=0.039

) on the A800. Three of the seven single-seed deletions fall above

p=0.05

(seeds 42, 123, 4096); see text.

Seed	Random-B BWT $\downarrow$	Grad-B BWT $\downarrow$	$\Delta$
42	$\mathbf{0.0397}$	0.1661	$-0.1264$
123	$\mathbf{0.0363}$	0.1273	$-0.0910$
456	$\mathbf{0.0461}$	0.1749	$-0.1289$
1024	$\mathbf{0.0480}$	0.1624	$-0.1144$
Mean	—	—	$-0.1152$ (SD $0.0173$ )

Appendix

Table 22: Density-matched D1 replication on Phi-2, four seeds. Both arms select the same number of coordinates from the same lora_B pool;

\Delta=\text{Random-B}-\text{Grad-B}

in loss-based BWT, so negative values mean the random mask forgets less.

Architecture	$n$	Mean $\Delta$	SD	$t$	$p$
Qwen2.5-3B	7	$-0.1552$	0.0707	$-5.80$	0.0011
Phi-2	4	$-0.1152$	0.0173	$-13.32$	0.0009

Appendix

Table 23: Density-matched D1 contrast on both architectures under the same sign convention,

\Delta=\text{Random-B}-\text{Grad-B}

(negative means Random-B forgets less). Phi-2 is an architecturally distinct four-seed replication of the seven-seed Qwen2.5-3B result.

Storage-Efficient Continual Learning

See all 11 papers

Jul 26, 2026

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.

Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2

College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1

Continual Learning for LLMs Continual LoRA Incremental Learning Parameter-Efficient Fine-Tuning

Ask PDF

May 12, 2026

Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning

LoRA is widely adopted for continual fine-tuning of Large Language Models due to its parameter efficiency, modularity across tasks, and compatibility with replay strategies. However, LoRA-based continual learning remains vulnerable to catastrophic forgetting, whose severity depends on how successive task gradients interact: when consecutive task gradients conflict, standard adapter initializations channel updates into subspaces that overwrite previously learned directions. We propose SLICE, a gradient-surgery-based initialization for LoRA adapters in continual learning. SLICE accumulates gradients from both the current task and a replay buffer of prior tasks, reconciles them through a projection operator, and decomposes the result via truncated SVD to initialize the adapter weights. We evaluate SLICE on the TRACE benchmark and sequences of Super-NI tasks, including a set of adversarial Super-NI sequences that we construct by mining task pairs with maximally opposing gradients. Compared to vanilla LoRA, LoRA-GA, and LoRAM, SLICE consistently achieves a better stability-plasticity trade-off, improving Average Performance, Final Performance and Forgetting metrics while preserving General Performance and In Context Performance across both standard and adversarial continual learning sequences.

Joana Pasquali, Ramiro N. Barros, Arthur S. Bianchessi +7

MALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, Brazil · Kunumi Institute, Brazil

Continual Learning for LLMs Gradient Interference Low-Rank Adaptation Incremental Learning Catastrophic Forgetting Continual LoRA

Ask PDF

May 6, 2026

Attribution-Guided Continual Learning for Large Language Models

Large language models (LLMs) often suffer from catastrophic forgetting in continual learning: after learning new tasks sequentially, they perform worse on earlier tasks. Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization. However, these methods lack understanding of LLM mechanisms and cannot distinguish which parameters store important knowledge from previous tasks and which parameters can be updated for new tasks. To address this, we propose the attribution-guided continual fine-tuning framework that leverages Layer-wise Relevance Propagation (LRP) to estimate parameter importance based on the internal computational process of LLMs. During continual learning, parameters critical to previous tasks are constrained to receive smaller updates, while less relevant parameters remain available for learning new tasks. Extensive experiments show that, compared with baseline methods, our approach reduces catastrophic forgetting while preserving adaptability to new tasks, highlighting the value of mechanistic attribution for continual fine-tuning of LLMs.

Yazheng Liu, Yuxuan Wan, Rui Xu +3

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Beijing University of Posts and Telecommunications, Beijing, China

Continual Learning for LLMs LLM Fine-Tuning Catastrophic Forgetting

Ask PDF

Abstract

Storage-Efficient Continual Learning

Explore similar work

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning

Attribution-Guided Continual Learning for Large Language Models