Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix

Yu Sun; Junwei Zhou; Zuodong Xiang; Yike Zhang; Pengcheng Xu; Xinyu Guan; Ruoyun Ma; Hailu Xu

cs.LGOct 8, 2026

Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix

Authors: Yu Sun, Junwei Zhou, Zuodong Xiang, Yike Zhang, Pengcheng Xu, Xinyu Guan, Ruoyun Ma, Hailu Xu

Organizations: University of Pennsylvania · Matter Innovation Inc. · University of California, Davis · Johns Hopkins University ·

Abstract

Parameter-efficient continual learning updates a restricted subset of adapter parameters to incorporate sequential tasks while limiting catastrophic forgetting. Gradient attribution is a common rule for selecting that subset, but it requires an additional backward pass over calibration data. We identify a parameter-pool confound in comparisons between initialization-time gradient masks and unrestricted random masks under standard zero-initialized LoRA ( $B=0$ ): the gradient with respect to $A$ is identically zero ( $\partial\mathcal{L}/\partial A=0$ ), so gradient masks select strictly from the $B$ matrix whereas unrestricted random masks draw from both $A$ and $B$ . Such a comparison changes both the selection rule and the candidate parameter pool. Isolating the selection criterion by restricting both approaches to the same parameter pool, we find a negative result: a uniform random mask consistently achieves lower loss-based backward transfer than gradient attribution across seven seeds on Qwen2.5-3B. The effect is concentrated on the medical task, which contributes 80% of the aggregate; code and math move in the same direction, and legal shows no reliable difference. This advantage is consistent with a stability--plasticity trade-off under the tested task sequence, in which random masking slightly reduces single-task fitting. On the two discrete-option tasks (Medical and Legal), the mean accuracy-BWT difference is not statistically distinguishable from zero.

Figures & tables

Figure 1: The density-matched comparison. (a) Both arms draw 10% of the rank-16 lora_B pool, so they train exactly the same 1.6M parameters and differ only in how those coordinates are chosen; because

B=0

at initialization, the gradient score is zero for every lora_A coordinate, and any baseline that draws from both pools changes the pool as well as the criterion. (b) Under that matched density the random mask achieves lower loss-based BWT than the gradient mask in all seven seeds.

Symbol	Meaning
$f_{\theta}$ , $W_{0}$	base model and its frozen pretrained weight matrix
$B$ , $A$	low-rank LoRA factors, $B\in\mathbb{R}^{d_{\text{out}}\times r}$ , $A\in\mathbb{R}^{r\times d_{\text{in}}}$
$r$ , $\alpha$	LoRA rank ( $r=16$ ) and scaling ( $\alpha=32$ )
$W$	adapted weight, $W=W_{0}+\alpha BA$
$t$ , $T$	task index and number of tasks ( $T=5$ )
$\mathcal{D}_{t}$	training split of task $t$

Table 1: Notation. Indices run over tasks

t=1,\dots,T

with

T=5

, and over prior tasks

j\in\mathcal{P}=\{1,\dots,T-1\}

Figure 2: Per-seed within- lora_B differences (

\Delta=\text{Random-B}-\text{Grad-Full}

, loss-based BWT, 7 seeds). All seven are negative (Random-B forgets less), mean

\Delta=-0.072

(orange line); seed 42 is the largest effect (

-0.165

). Leave-one-out excluding seed 42 remains significant (paired

t(5)

p=0.015

Seed	Random-B BWT $\downarrow$	Gradient-B BWT $\downarrow$	$\Delta$ (matched)	$\Delta$ (unmatched)
42	$\mathbf{0.278}$	0.453	$-0.175$	$-0.220$
123	$\mathbf{0.264}$	0.453	$-0.188$	$-0.119$
456	$\mathbf{0.283}$	0.406	$-0.123$	$-0.131$
789	$\mathbf{0.352}$	0.387	$-0.035$	$-0.074$
1024	$\mathbf{0.352}$	0.474	$-0.121$	$-0.109$
2048	$\mathbf{0.231}$	0.491	$-0.261$	$-0.187$

Table 2: Headline result (D1, density-matched). Both arms select the same 1,651,508 parameters (10% of lora_B ); only the criterion differs. Per-seed loss-based BWT and

\Delta=\text{Random-B}-\text{Grad-B}

(negative = random-B forgets less), seven sessions, one seed each. The last column is the density- unmatched contrast from the same sessions, that is, Random-B against the unrestricted gradient mask (

\Delta=\text{Random-B}-\text{Grad-Full}

) rather than against Grad-B.

Figure 3: The density-matched D1 per-task decomposition, averaged over seven seeds. For each prior task, the bars show the loss-based BWT contribution of Random-B and Grad-B, with the corresponding values reported above each pair. Because the task-level contributions span roughly two orders of magnitude, the left panel uses a broken

y

-axis: the slashes mark the omitted interval from 0.52 to 0.92, and the labels above the Medical bars report their full values (

+0.998

and

+1.493

) rather than truncated heights. Creative is the final task and therefore has no backward-transfer contribution. The right panel shows the aggregate loss-based BWT on a separate scale. Random-B exhibits less loss-based forgetting on Code, Medical, and Math; the Legal difference is negligible and changes sign across seeds. Most of the aggregate difference is concentrated on Medical. Per-seed results are reported in Appendix Q .

Figure 4: Accuracy-based validation. Blue points are the seven per-seed accuracy-BWT differences (Grad-B minus Random-B, percentage points) on the two discrete-option tasks, spanning

-6.98

+6.00

pp; green is the seven-seed mean with its 95% interval

[-2.91,+4.80]

pp. The plotted quantity is a difference of accuracy-BWT (mean final-minus-learned accuracy over prior tasks), a retention measure, not a difference of final accuracies. The interval spans zero, so no statistically reliable accuracy difference is detected, and it reaches past the 4 pp non-inferiority margin, so that criterion is not met. Per-seed values are in Appendix N .

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Peak GPU (GB)	Time (s)	Backward?
Gradient mask	$13.8^{{\dagger}}$	584.9	Yes
Wanda proxy	7.4	12.1	No
Shared random / RRM	0 ‡	0	No
No mask	0 ‡	0	No

Appendix

Table 3: Attribution cost on Qwen2.5-3B (BF16). Both columns are from a single full-parameter measurement run ( --include-frozen , --full-precision ); the exact correspondence and its limits are described below. Peak GPU is torch.cuda.max_memory_allocated converted to decimal GB (profiling fields *_peak_alloc_mb = 13168 / 7102 MiB, i.e. 13.8 / 7.4 GB); Time is the per-task attribution wall-clock recorded by the run (584.9 s gradient / 12.1 s Wanda).

Figure 5: Spearman $\rho=0.086$ between Wanda and gradient importance (LoRA parameters) on Qwen2.5-3B. These metrics rank parameters nearly independently.

Seed	Gradient BWT $\downarrow$	Random BWT $\downarrow$	No Mask BWT $\downarrow$
42	0.491	0.435	0.507
123	0.463	0.417	$\mathbf{0.409}$
456	$\mathbf{0.388}$	0.414	0.539
789	$\mathbf{0.357}$	0.417	0.484
1024	0.486	0.458	0.479
Mean	0.437	$\mathbf{0.428}$	0.484

Appendix

Table 4: Per-seed BWT breakdown (Qwen2.5-3B, rank-16 LoRA, 5 tasks).

Method	BWT $\downarrow$	Loss $\downarrow$	Calibration pass?
Shared random	$\mathbf{0.428}\pm 0.019$	$\mathbf{1.940}\pm 0.021$	No
Gradient mask	$0.437\pm 0.061$	$1.966\pm 0.037$	Yes
No mask	$0.484\pm 0.047$	$1.970\pm 0.033$	No

Appendix

Table 5: Confounded comparison (Qwen2.5-3B, rank-16 LoRA, 5 tasks,

n{=}5

seeds). The last column records whether an additional calibration forward–backward pass is required; its time and data costs are described in Appendix B . Across all tables, bold marks the better value in its column.

Seed	random_B_only BWT $\downarrow$	full_alora BWT $\downarrow$	$\Delta$
42	$\mathbf{0.326}$	0.491	$-0.165$
123	$\mathbf{0.320}$	0.413	$-0.093$
456	$\mathbf{0.337}$	0.356	$-0.019$
789	$\mathbf{0.346}$	0.364	$-0.018$
1024	$\mathbf{0.336}$	0.442	$-0.106$
2048	$\mathbf{0.367}$	0.402	$-0.035$

Appendix

Table 6: Within- lora_B comparison (Qwen2.5-3B, rank-16 LoRA, 5 tasks, 7 seeds). All comparisons are within-run (same method ordering and RNG state).

\Delta=\text{random\_B\_only}-\text{Grad-Full}

; negative

\Delta

indicates random-B achieves lower (better) BWT.

Level	What varies	Magnitude	What it measures
(a) protocol-equivalence check	execution path only	$9.6\times 10^{-5}$	numerical agreement between two code paths; not an inferential error term
(b) across-seed paired contrast	independent seed	SD $(\Delta_{s})=0.071$	variability used by the paired significance test
(c) position (ordering) sensitivity	position of the two arms	$0.064$ (seed 42, gradient_B )	method-position interaction in a three-seed check
(d) cross-session variation	session, hardware, and method	up to $0.120$ ( full_alora , seed 42)	method-specific offset of unknown origin

Appendix

Table 7: Distinct quantities that are sometimes all called run-to-run noise . The paired tests use the across-seed SD of the within-session contrasts; the protocol-equivalence discrepancy is an implementation check and does not enter inference. Position and cross-session effects are reported as sensitivities rather than as interchangeable error estimates.

Seed	$\|\Delta_{\text{pos}}(\text{rB})\|$	$\|\Delta_{\text{pos}}(\text{gB})\|$	rB / main	gB / main
42	$1.5\times 10^{-3}$	$6.4\times 10^{-2}$	0.01	0.41
789	$3.1\times 10^{-5}$	$6.2\times 10^{-3}$	0.00	0.04
2048	$8.4\times 10^{-6}$	$1.3\times 10^{-2}$	0.00	0.08

Appendix

Table 8: Method-ordering sensitivity (D1 density-matched, three seeds).

\Delta_{\text{pos}}

is the BWT difference between running at position 1 and position 2. The last two columns express each arm’s positional shift as a fraction of the main D1 contrast (

|\Delta|=0.155

Method	Run A	Run B	Run C	Run D / spread
Gradient mask ( full_alora )	0.491	0.378	0.498	— (0.120)
Gradient-B@10% ( gradient_B )	—	—	0.453	0.423 (0.030)
Random mask ( random_A+B )	0.435	0.349	—	—
Random B-only ( random_B_only )	0.326	0.324	0.278	0.278 (0.048)

Appendix

Table 9: Absolute loss-based BWT for the same seed 42 under identical method definitions, across sessions and hardware. Run C is the D1 density-matched session and Run D the uniform-B suppression control (the M2 controller) session, both on a separate GPU and software stack; random_A+B was not run in either. A Run-D entry exists only for methods Run D ran, and it did not train a plain full_alora arm, so that cell is --- and its gradient-side arm is gradient_B , entered on its own row. Spread is

\max-\min

across the runs in which the method appears.

Condition	RRM BWT	Random BWT	$\Delta$	Direction
Rank 4	0.464	0.423	$+$ 0.041	Random better
Rank 8	0.427	0.373	$+$ 0.055	Random better
Rank 16 (ref.)	0.401	0.435	$-$ 0.034	RRM better
Rank 32	0.414	0.390	$+$ 0.024	Random better
Reverse order	0.115	0.109	$+$ 0.006	Random better

Appendix

Table 10: RRM boundary analysis (Qwen2.5-3B, single seed per condition except rank-16 reference).

Figure 6: Sparsity sweep (Qwen2.5-3B, single seed). Shared random achieves numerically lower BWT than gradient attribution at all four tested levels.

Sparsity	Random BWT $\downarrow$	Gradient BWT $\downarrow$
5%	0.327	0.422
10%	0.435	0.491
20%	0.444	0.474
30%	0.361	0.467

Appendix

Table 11: Sparsity sweep numerical values (Qwen2.5-3B, single seed). Random BWT

\leq

Gradient BWT at all tested levels.

Figure 7: The suppression control’s outcome is seed-dependent . All three arms train 1,651,508 parameters, which is 5.5% of the full LoRA pool (each arm draws 10% of the lora_B sub-pool, and

B

holds 55% of the pool’s parameters), so this is the density-matched setting throughout. The uniform-B controller redistributes the gradient mask’s budget uniformly across LoRA modules. At seed 42 it sits in the gradient band (0.440 above gradient-B’s 0.423 and well above random-B-only’s 0.278); seeds 123 and 456 also land on the gradient side; at seed 789 it falls to the random band (0.355 below gradient-B’s 0.416, alongside random-B-only’s 0.352). Three of four seeds fall on the gradient side. M2 is an independent session from D1, so the same-seed gradient-B values differ between them (e.g. seed 42: 0.423 here vs 0.453 in the D1 table; seed 789: 0.416 vs 0.387; § F ). We therefore do not treat this ablation as establishing the o_proj -concentration mechanism (§ 6 ).

Figure 8: Gradient-mask concentration by LoRA module (Qwen2.5-3B). Gradient attribution selects 29% of o_proj parameters (2.9

\times

the 10% uniform rate) and is under-represented in down_proj (4.1%); the dashed line marks uniform 10% selection.

Module	Gradient density	Random density	Ratio
o_proj	29.0%	10.0%	2.9 $\times$
up_proj	12.8%	10.0%	1.3 $\times$
v_proj	10.3%	10.0%	1.0 $\times$
q_proj	8.1%	10.0%	0.8 $\times$
k_proj	8.0%	10.0%	0.8 $\times$
gate_proj	8.0%	10.0%	0.8 $\times$

Appendix

Table 12: Gradient mask concentration by LoRA module (Qwen2.5-3B). Random masking distributes uniformly at 10% per module. Gradient attribution selects exclusively lora_B parameters (all lora_A receive zero importance due to

B=0

initialization).

Scale	Random BWT $\downarrow$	Gradient BWT $\downarrow$	No Mask BWT $\downarrow$
0.5B	0.343	—	0.431
1.5B	0.309	—	0.356
7B	0.447	0.441	0.514

Appendix

Table 13: BWT at additional scales (single seed, observational).

Architecture	Comparison	Grad. BWT $\downarrow$	Rand. BWT $\downarrow$	$\Delta$
Qwen2.5-3B ( $n{=}5$ )	Confounded	$0.437\pm 0.061$	$0.428\pm 0.019$	$-0.009$
Phi-2 ( $n{=}5$ )	Confounded	$0.158\pm 0.019$	$0.183\pm 0.016$	$+0.025$
Qwen2.5-3B ( $n{=}7$ )	Within-B	$0.409\pm 0.046$	$0.338\pm 0.015$	$-0.072$
Phi-2 ( $n{=}4$ )	Within-B	$0.149\pm 0.036$	$0.065\pm 0.017$	$-0.084$

Appendix

Table 14: Cross-architecture comparison on Qwen2.5-3B and Phi-2 (2.78B, GPT-NeoX-style), with both arms within a row sharing one target set. Qwen2.5-3B targets q/k/v/o/gate/up/down_proj (pool

29{,}933{,}568

), whereas Phi-2 uses GPT-NeoX-style q,k,v,fc1,fc2,dense projections (pool

23{,}592{,}960

) in the confounded design. In the within- lora_B design, the Phi-2 rows omit dense (pool

20{,}971{,}520

, 12.5% smaller), so the two Phi-2 designs are not comparable row-for-row. The within- lora_B rows are matched on pool but not density (the gradient mask activates approximately

2\times

the B parameters of Random-B), as on Qwen2.5-3B.

Method	Unrestricted BWT	B-only BWT	Gap
Gradient	0.378	0.378	$+0.000$
Random	0.349	0.324	$-0.025$

Appendix

Table 15: Effect of restricting the mask to lora_B (seed 42). For gradient attribution, the restriction is a no-op by construction (B=0 collapses every lora_A importance score to zero); for random, restricting to B removes lora_A dilution and improves BWT by 0.025. This asymmetry demonstrates that the within-B comparison varies the selection criterion, not the parameter pool.

Seed	Accuracy-BWT $\Delta$ [Grad $-$ Rand], pp	Loss BWT $\Delta$ [Grad $-$ Rand]
42	$+2.00$	$+0.2320$
123	$0.00$	$+0.1647$
456	$+0.48$	$+0.1300$
789	$-6.98$	$+0.0786$
1024	$+4.60$	$+0.0514$
2048	$+0.51$	$+0.2572$

Appendix

Table 16: Per-seed results, both columns in the direction [Grad

-

Random] (positive = gradient forgets more). Col. 2 is the accuracy-BWT difference: the difference in mean final-minus-learned accuracy over the prior discrete-option tasks, not a difference of final accuracies. It is therefore a retention quantity on the same footing as the loss column and does not mix learning with retention. Col. 2 is reported in percentage points and is measured in the A800 accuracy runs; Col. 3 is the loss-based BWT difference from the A800 loss runs. Together they form a second environment relative to the original V100 loss runs. Col. 2 carries the 95%

t

-interval on the seven-seed mean. Col. 3 has mean

+0.152

(SD

0.0749

t=5.37

, CI

[+0.083,+0.221]

, 7/7 same direction). The V100 loss contrast is

+0.155

under the same [Grad

-

Random] convention. Across matched seeds, the A800-minus-V100 loss-contrast difference has mean

-0.0033

and SD

0.0441

, so individual seeds move between environments even though the average does not.

Method	B=0 pool	Within-pool grad-vs-random
EWC / SI / MAS	No	No
PiSSA ( $B\neq 0$ )	N/A by init	No
Attribution-Guided CL (post-training attribution)	N/A by design	No
Ours (within- lora_B )	Yes	Yes

Appendix

Table 17: Positioning against gradient-attribution families. “B=0 pool” marks a method whose attribution is computed at a zero-initialized LoRA, where the gradient with respect to lora_A is zero and the mask is confined to lora_B ; “within-pool grad-vs-random” marks a method compared against a matched random baseline drawn from the same pool. Attribution-Guided CL computes its attribution from a separately trained model (

B\neq 0

at scoring time) and gates gradients continuously rather than selecting a binary mask, so it is outside the first axis by construction and unmatched on the second.

Rule	What it scores, and what for	Run
EWC [ Kirkpatrick et al., 2017 ] ; MAS [ Aljundi et al., 2018 ]	Fisher information, output sensitivity; a penalty, not a mask	no
SNIP [ Lee et al., 2019 ] ; GraSP [ Wang et al., 2020 ]	gradient at initialisation; one-shot pruning mask	no
Wanda [ Sun et al., 2024 ]	weight $\times$ activation magnitude; one-shot pruning mask	no
LoRA-FA [ Zhang et al., 2023 ]	none, structural; freezes lora_A , trains lora_B	no
Grad-Full, Grad-B (§ 3.3 )	calibration gradient magnitude; static CL mask, full pool (Grad-Full) or lora_B at 10% (Grad-B)	yes
Random-B; Random A+B [ Xu and Zhang, 2024 ]	none, uniform; static CL mask, lora_B at 10% (Random-B) or full pool (A+B)	yes

Appendix

Table 18: Selection rules this paper is positioned against. The last column marks the four rules we train and evaluate under one protocol (§ 3.3 ); the rest are compared by design only, and none of their published numbers is cited as a result of ours.

Seed	Code	Medical	Legal	Math	Aggregate
42	$-0.040$	$-0.409$	$-0.002$	$-0.249$	$-0.175$
123	$-0.002$	$-0.698$	$-0.002$	$-0.052$	$-0.188$
456	$-0.074$	$-0.381$	$-0.004$	$-0.033$	$-0.123$
789	$-0.063$	$-0.054$	$+0.001$	$-0.023$	$-0.035$
1024	$-0.044$	$-0.410$	$-0.011$	$-0.021$	$-0.121$
2048	$-0.041$	$-0.950$	$+0.010$	$-0.062$	$-0.261$

Appendix

Table 19: Per-seed, per-task D1 decomposition (

\Delta=\text{random-B}-\text{gradient-B}

for each prior task). Negative favours random-B. Legal is the only task whose sign is not consistent across seeds.

Task	random_B_only	gradient_B	full_alora
code	$0.952\pm 0.111$	$\mathbf{0.901\pm 0.100}$	$0.898\pm 0.099$
medical	$1.190\pm 0.040$	$\mathbf{1.152\pm 0.040}$	$1.144\pm 0.035$
legal	$0.388\pm 0.035$	$\mathbf{0.368\pm 0.038}$	$0.365\pm 0.037$
math	$3.838\pm 0.103$	$\mathbf{3.652\pm 0.096}$	$3.642\pm 0.100$
creative	$2.104\pm 0.053$	$\mathbf{2.067\pm 0.053}$	$2.056\pm 0.054$

Appendix

Table 20: Diagonal loss (mean

\pm

std over the seven D1 seeds), by task. Lower is better. gradient-B is lower than random-B-only in every cell (Table 20 ); Grad-Full is the uncontrolled reference. Loss diagonals only; the D1 run records contain no accuracy matrix (accuracy is measured separately in the accuracy branch, § 5.4 ).

Seed	rB start	rB end	gB start	gB end (diff)
42	1.543	1.821	1.478	1.931 ( $+0.110$ )
123	1.632	1.896	1.553	2.006 ( $+0.110$ )
456	1.575	1.858	1.494	1.900 ( $+0.042$ )
789	1.548	1.900	1.475	1.862 ( $-0.038$ )
1024	1.566	1.918	1.489	1.962 ( $+0.045$ )
2048	1.665	1.896	1.587	2.078 ( $+0.182$ )

Appendix

Table 21: Mean final held-out loss over the four prior tasks, per seed, read from the final row of the stored loss matrix. Start is the same quantity at each task’s own training step; end is at the end of the sequence. gradient-B’s endpoint is higher in 6 of 7 seeds (seed 789 is the exception); paired

t(6)=3.03

p=0.023

, mean

+0.081

on the V100, and

+0.078

(

p=0.039

) on the A800. Three of the seven single-seed deletions fall above

p=0.05

(seeds 42, 123, 4096); see text.

Seed	Random-B BWT $\downarrow$	Grad-B BWT $\downarrow$	$\Delta$
42	$\mathbf{0.0397}$	0.1661	$-0.1264$
123	$\mathbf{0.0363}$	0.1273	$-0.0910$
456	$\mathbf{0.0461}$	0.1749	$-0.1289$
1024	$\mathbf{0.0480}$	0.1624	$-0.1144$
Mean	—	—	$-0.1152$ (SD $0.0173$ )

Appendix

Table 22: Density-matched D1 replication on Phi-2, four seeds. Both arms select the same number of coordinates from the same lora_B pool;

\Delta=\text{Random-B}-\text{Grad-B}

in loss-based BWT, so negative values mean the random mask forgets less.

Architecture	$n$	Mean $\Delta$	SD	$t$	$p$
Qwen2.5-3B	7	$-0.1552$	0.0707	$-5.80$	0.0011
Phi-2	4	$-0.1152$	0.0173	$-13.32$	0.0009

Appendix

Table 23: Density-matched D1 contrast on both architectures under the same sign convention,

\Delta=\text{Random-B}-\text{Grad-B}

(negative means Random-B forgets less). Phi-2 is an architecturally distinct four-seed replication of the seven-seed Qwen2.5-3B result.

Storage-Efficient Continual Learning

See all 11 papers

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Continual Learning for LLMs Continual LoRA
College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1
Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning
May 12, 2026Joana Pasquali, Ramiro N. Barros, Arthur S. Bianchessi +7Continual Learning for LLMs Gradient Interference
MALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, Brazil · Kunumi Institute, Brazil
Attribution-Guided Continual Learning for Large Language Models
May 6, 2026Yazheng Liu, Yuxuan Wan, Rui Xu +3Continual Learning for LLMs LLM Fine-Tuning
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Beijing University of Posts and Telecommunications, Beijing, China

Jul 26, 2026J/K move · Enter open · S save

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli, Pavan Turaga+1

College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA+1

Continual Learning for LLMs Continual LoRA Incremental Learning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting. To mitigate this, LoRA-based continual learning methods allocate a separate low-rank adapter per task, yet existing approaches either require task identity at inference or sum all adapters indiscriminately, letting irrelevant branches distort the output. Recent gating-based solutions route inputs to the correct adapter but introduce trainable parameters that themselves need protection against forgetting. In this work, we observe that pooled token embeddings from a frozen LLM embedding layer already separate task distributions throughout the learning sequence. A Gaussian mixture model fitted on these embeddings, without any gradient-based training, is sufficient for task-agnostic adapter selection at test time. This eliminates the need for a learned gating module. On the adapter side, constraining each task's parameters to the principal subspace of the pretrained weights via SVD yields a compact latent-space parameterization. Within this subspace, orthogonal regularization directly controls inter-task interference. The resulting system, Latent-LoRA, is replay-free, requires no trainable routing component, and uses substantially fewer parameters per task. Experiments across five model scales and two established continual learning benchmarks show state-of-the-art performance with near-zero forgetting.

Abstract

Storage-Efficient Continual Learning

Explore similar work

Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning

Attribution-Guided Continual Learning for Large Language Models