BISECT: Bilevel Spectral Decomposition for Low-Rank Continual Learning

Tom Saliencro; Wen-Hsuan Liu; Rohan Desai; Amara Okonkwo; Maya Lindqvist

cs.LGNov 2, 2026

BISECT: Bilevel Spectral Decomposition for Low-Rank Continual Learning

Authors: Tom Saliencro, Wen-Hsuan Liu, Rohan Desai, Amara Okonkwo, Maya Lindqvist

Organizations: University of California, Irvine · University of Washington

Abstract

Exemplar-free continual learning on pre-trained Vision Transformers increasingly relies on Low-Rank Adaptation (LoRA), yet prevailing methods govern the low-rank update space through strict subspace isolation. By projecting each incoming task into the approximate null space of historical token activations, isolation-based learners equate any cross-task feature overlap with catastrophic interference. We show that this orthogonality premise induces a structural bottleneck we term spectral starvation: when down-projection bases are locked before task optimization, the gradient norm reaching the trainable up-projection is upper-bounded by the energy of current-task features captured by the chosen basis. Because natural visual tasks share low- and mid-level covariance structure, historical null tails contain barely $5.5\%$ of incoming task variance, starving new-class plasticity while blocking positive consolidation along shared directions. We propose BISECT (Bilevel Spectral Decomposition), a continual LoRA framework that partitions a fixed rank- $r$ budget into two orthogonal functional blocks: an invariant carrier spanned by the dominant modes of the joint historical-and-current Gram matrix, and a novelty complement extracted via generalized Rayleigh whitening followed by orthogonal complement projection. After single-task training, BISECT folds both blocks back into the frozen backbone using a closed-form precision gate that scales each rank-1 invariant atom by its current-to-joint Rayleigh ratio while preserving novelty atoms at unit gain. Across eight incremental settings on ImageNet-R, ImageNet-A, CIFAR-100, CUB-200, and DomainNet, BISECT outperforms prompt, adapter, and LoRA baselines with zero extra inference parameters or FLOPs, and remains complementary to sparse analytic expansion.

Figures & tables

Figure 1: Diagnosing the orthogonality trap in LoRA continual learning. (a) Final accuracy

A_{\mathrm{Final}}

across four challenging splits: unconstrained LoRA overwrites prior sessions, whereas null-space projection (InfLoRA) collapses under severe distribution shift on 10S-ImageNet-A. (b) Distribution of task-

t

signal across quartiles of the historical eigenspectrum

G_{<t}

on ViT-B/16: the bottom

30\%

null tail holds only

5.5\%

of raw task energy, starving gradient flow, whereas generalized Rayleigh whitening shifts selection to the mid-spectrum where unexplained variance peaks (

51.6\%

). (c) Historical retention vs. emerging-class accuracy on 10S-ImageNet-R; varying the spectral threshold

\eta

traces a Pareto frontier above isolation and gradient-splitting baselines.

Figure 2: Overview of BISECT at session

t

. (I) Using the historical Gram state

G_{<t}

and current-task Gram

G_{t}

, BISECT determines the layerwise rank split

(k_{\mathrm{inv}},k_{\mathrm{nov}})

from the top-

r

spectral energy threshold

\eta

, extracts the invariant carrier

P_{\mathrm{inv}}

from the joint Gram

G_{1:t}

via randomized SVD, and synthesizes the orthogonal novelty complement

P_{\mathrm{nov}}

via Cholesky-whitened generalized Rayleigh extraction. (II) Both basis blocks

P_{\mathrm{inv}},P_{\mathrm{nov}}

remain locked while the up-maps

Q_{\mathrm{inv}},Q_{\mathrm{nov}}

are optimized by SGD on

\mathcal{D}_{t}

. (III) Each rank-1 invariant atom is folded into

W^{(t-1)}

with its closed-form posterior precision weight

\omega_{j}^{\star}\in[0,1]

, while novelty atoms fold at unit weight, yielding zero inference overhead.

Method	Venue	10S-ImageNet-R		10S-ImageNet-A		10S-CIFAR-100		10S-CUB-200
		$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$
L2P [ 41 ]	CVPR’22	72.28	77.31	44.12	51.30	84.10	88.24	67.08	79.58
DualPrompt [ 40 ]	ECCV’22	69.15	74.32	53.14	64.52	86.88	91.09	68.52	80.64
CODA-Prompt [ 30 ]	CVPR’23	73.26	78.42	52.15	63.88	83.29	87.76	77.18	81.86
EASE [ 46 ]	CVPR’24	76.18	81.74	55.38	65.12	87.84	92.18	76.85	84.92
InfLoRA [ 21 ]	CVPR’24	74.68	80.62	49.35	60.86	86.72	91.68	70.88	81.42

Table 1: Comparison with state-of-the-art rehearsal-free continual learners on four 10-session (10S) benchmarks using ViT-B/16. Every method is evaluated across all four datasets (

A_{\mathrm{Final}}

\bar{A}

in %, averaged over 3 seeds). Best results are in bold , second best underlined .

Method	Venue	5S-ImageNet-R		20S-ImageNet-R		20S-ImageNet-A		10S-DomainNet
		$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$
L2P [ 41 ]	CVPR’22	70.88	78.39	69.58	75.22	40.54	49.68	81.12	87.38
DualPrompt [ 40 ]	ECCV’22	73.10	79.52	66.68	72.51	42.34	53.44	81.66	87.76
CODA-Prompt [ 30 ]	CVPR’23	74.86	79.21	69.92	75.30	44.58	54.82	80.09	86.31
EASE [ 46 ]	CVPR’24	78.64	83.12	73.84	79.46	48.26	58.54	82.18	87.92
InfLoRA [ 21 ]	CVPR’24	77.58	82.06	71.14	77.34	46.95	57.21	81.42	87.51

Table 2: Evaluation across variable task horizons (5S and 20S on ImageNet-R, 20S on ImageNet-A) and multi-domain class-incremental streams (10S-DomainNet). All baseline cells are evaluated under the unified ViT-B/16 protocol (

A_{\mathrm{Final}}

\bar{A}

in %).

Variant	$P_{\mathrm{inv}}$	$P_{\mathrm{nov}}$	$\Omega_{\mathrm{inv}}^{\star}$	Adapt. $k$	10S-IN-R	10S-IN-A	20S-IN-R	20S-IN-A
1. Seq-LoRA	–	–	–	–	71.88	52.96	68.04	44.92
2. Invariant only	✓	–	–	–	79.14	59.64	76.92	51.84
3. Novelty only	–	✓	–	–	79.32	60.02	76.16	51.60
4. Bilevel (fixed $k$ )	✓	✓	–	–	81.36	61.78	79.02	54.18
5. + Precision gate	✓	✓	✓	–	81.84	62.46	79.31	55.54
6. BISECT (Full)	✓	✓	✓	✓	82.18	62.84	79.48	56.12

Table 3: Component ablation study of BISECT reporting

A_{\mathrm{Final}}

(%) across 10- and 20-session ImageNet-R (IN-R) and ImageNet-A (IN-A).

Figure 3: Mechanistic verification on ViT-B/16. (a) Layerwise generalized Rayleigh selectivity

\mathcal{R}(P)

across blocks

1\dots 12

. (b) Layer-adaptive rank split

(k_{\mathrm{inv}},k_{\mathrm{nov}})

\eta=0.7

. (c) Cumulative old-class representation drift

\|X_{<t}\Delta W_{t}^{\top}\|_{F}

on 10S-ImageNet-R; precision folding (

\Omega_{\mathrm{inv}}^{\star}

) removes

83.4\%

of unscaled drift.

Figure 4: Stream dynamics, sensitivity, and setup efficiency. (a) Session-wise seen-class accuracy on 10S-ImageNet-A. (b) Sensitivity to

\eta\in[0.3,0.9]

across three benchmarks (10S-IN-A shifted by

+19

). (c) Final accuracy vs. per-session setup time on 20S-ImageNet-R (

10.2

s/task,

12.4\times

faster than LoDA).

Benchmark	Subspace Constructor	$\mathcal{R}(P)\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$
10S-IN-R	Random Orthogonal	$1.01\times$	68.34	79.12
	Adam-NSCL [ 37 ]	$1.46\times$	74.58	82.34
	InfLoRA [ 21 ]	$1.74\times$	75.08	82.52
	Unwhitened Task PCA	$1.36\times$	76.84	83.68
	BISECT ( $P_{\mathrm{nov}}$ )	$\mathbf{3.95\times}$	79.32	85.38
10S-IN-A	Random Orthogonal	$1.02\times$	53.56	64.41

Table 4: Controlled comparison of rank-

32

plastic subspace constructors (without invariant carrier). We report mean selectivity ratio

\mathcal{R}(P)

and

A_{\mathrm{Final}}

\bar{A}

(%).

Method	Gram Mem.	Train	Setup	Extra Test	Test	$A_{\mathrm{Final}}\uparrow$
	(MB)	GFLOPs	(s/task)	Params (M)	GFLOPs	(%)
InfLoRA [ 21 ]	11.3	17.67	51.8	0.00	17.60	71.14
SD-LoRA [ 44 ]	14.1	20.51	15.2	0.00	17.60	75.18
VPT-NSP [ 23 ]	28.4	17.72	55.8	0.09	17.72	76.08
CPG [ 8 ]	28.4	17.72	51.4	1.66	17.72	75.28
SplitLoRA [ 28 ]	27.0	18.42	33.6	0.00	17.60	78.68

Table 5: Computational, memory, and inference cost audit on 20S-ImageNet-R (single RTX 4090 GPU). Setup latency measures per-session subspace construction.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Oversampling $q$	$\sin\Theta_{\max}\downarrow$	Setup Time (s/task) $\downarrow$	Speedup vs. Exact	10S-IN-R $A_{\mathrm{Final}}$ (%)	10S-IN-A $A_{\mathrm{Final}}$ (%)
$q=0$ (no oversampling)	0.138	6.2	$8.2\times$	81.48	62.04
$q=2$	0.054	6.9	$7.4\times$	81.91	62.48
$q=4$	0.022	8.1	$6.3\times$	82.09	62.71
$q=8$ (BISECT default)	0.008	10.2	$\mathbf{5.0\times}$	82.18	62.84
$q=12$	0.005	12.4	$4.1\times$	82.19	62.85
$q=16$	0.004	14.8	$3.4\times$	82.19	62.85

Appendix

Table 6: Sensitivity to the RSVD oversampling parameter

q

across all

12

ViT-B/16 blocks on 10S-ImageNet-R and 10S-ImageNet-A.

\sin\Theta_{\max}

measures the worst-case principal angle deviation from exact full SVD.

Benchmark Split	Total Classes	Sessions $T$	Classes / Session	Train Images	Base LR	Batch Size / Epochs
5S-ImageNet-R [ 12 ]	200	5	40	24,000	0.020	48 / 20
10S-ImageNet-R [ 12 ]	200	10	20	24,000	0.020	48 / 20
20S-ImageNet-R [ 12 ]	200	20	10	24,000	0.020	48 / 20
10S-ImageNet-A [ 13 ]	200	10	20	5,981	0.010	48 / 20
20S-ImageNet-A [ 13 ]	200	20	10	5,981	0.010	48 / 20
10S-CIFAR-100 [ 18 ]	100	10	10	50,000	0.010	48 / 20

Appendix

Table 7: Summary of benchmarks, class-incremental splits, and session training hyperparameters.

Method	10S-ImageNet-R		10S-ImageNet-A		10S-CIFAR-100		10S-CUB-200
	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$
L2P [ 41 ]	$72.28_{\pm 0.19}$	$77.31_{\pm 0.61}$	$44.12_{\pm 0.89}$	$51.30_{\pm 2.18}$	$84.10_{\pm 0.84}$	$88.24_{\pm 1.29}$	$67.08_{\pm 1.84}$	$79.58_{\pm 1.54}$
DualPrompt [ 40 ]	$69.15_{\pm 0.58}$	$74.32_{\pm 0.64}$	$53.14_{\pm 0.71}$	$64.52_{\pm 0.14}$	$86.88_{\pm 0.26}$	$91.09_{\pm 0.34}$	$68.52_{\pm 0.49}$	$80.64_{\pm 1.46}$
CODA-Prompt [ 30 ]	$73.26_{\pm 0.48}$	$78.42_{\pm 0.52}$	$52.15_{\pm 0.18}$	$63.88_{\pm 0.16}$	$83.29_{\pm 3.24}$	$87.76_{\pm 3.08}$	$77.18_{\pm 1.08}$	$81.86_{\pm 0.82}$
EASE [ 46 ]	$76.18_{\pm 0.34}$	$81.74_{\pm 0.41}$	$55.38_{\pm 0.68}$	$65.12_{\pm 0.84}$	$87.84_{\pm 0.29}$	$92.18_{\pm 0.31}$	$76.85_{\pm 0.44}$	$84.92_{\pm 0.52}$
InfLoRA [ 21 ]	$74.68_{\pm 0.61}$	$80.62_{\pm 0.53}$	$49.35_{\pm 1.08}$	$60.86_{\pm 0.59}$	$86.72_{\pm 0.33}$	$91.68_{\pm 0.18}$	$70.88_{\pm 0.25}$	$81.42_{\pm 0.17}$

Appendix

Table 8: Full 10-session benchmark results (mean

\pm

std over 3 class-order seeds), corresponding to Table 1 .

Method	5S-ImageNet-R		20S-ImageNet-R		20S-ImageNet-A		10S-DomainNet
	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$
L2P [ 41 ]	$70.88_{\pm 0.56}$	$78.39_{\pm 0.45}$	$69.58_{\pm 0.44}$	$75.22_{\pm 0.55}$	$40.54_{\pm 1.72}$	$49.68_{\pm 1.42}$	$81.12_{\pm 0.81}$	$87.38_{\pm 0.92}$
DualPrompt [ 40 ]	$73.10_{\pm 0.48}$	$79.52_{\pm 0.39}$	$66.68_{\pm 0.56}$	$72.51_{\pm 0.38}$	$42.34_{\pm 1.88}$	$53.44_{\pm 1.58}$	$81.66_{\pm 0.75}$	$87.76_{\pm 0.94}$
CODA-Prompt [ 30 ]	$74.86_{\pm 0.35}$	$79.21_{\pm 0.51}$	$69.92_{\pm 0.48}$	$75.30_{\pm 0.82}$	$44.58_{\pm 1.86}$	$54.82_{\pm 0.52}$	$80.09_{\pm 0.76}$	$86.31_{\pm 0.79}$
EASE [ 46 ]	$78.64_{\pm 0.31}$	$83.12_{\pm 0.36}$	$73.84_{\pm 0.42}$	$79.46_{\pm 0.51}$	$48.26_{\pm 1.44}$	$58.54_{\pm 1.18}$	$82.18_{\pm 0.45}$	$87.92_{\pm 0.48}$
InfLoRA [ 21 ]	$77.58_{\pm 0.36}$	$82.06_{\pm 0.15}$	$71.14_{\pm 0.43}$	$77.34_{\pm 0.44}$	$46.95_{\pm 2.18}$	$57.21_{\pm 0.92}$	$81.42_{\pm 0.65}$	$87.51_{\pm 0.54}$

Appendix

Table 9: Full variable-horizon and DomainNet results (mean

\pm

std over 3 seeds), corresponding to Table 2 .

Benchmark	Method	$t=1$	$t=2$	$t=3$	$t=4$	$t=5$	$t=6$	$t=7$	$t=8$	$t=9$	$t=10$ ( $A_{\mathrm{Final}}$ )
10S-IN-R	Seq-LoRA	92.04	88.78	86.18	83.76	81.48	79.28	77.14	75.06	73.02	71.88
	InfLoRA [ 21 ]	89.28	86.71	84.64	82.72	80.92	79.17	77.48	75.84	74.76	74.68
	LoDA [ 11 ]	93.72	91.70	90.08	88.58	87.16	85.79	84.46	83.16	81.89	81.86
	BISECT (Ours)	93.96	91.95	90.34	88.83	87.39	86.01	84.68	83.38	82.18	82.18
10S-IN-A	InfLoRA [ 21 ]	76.10	71.65	68.08	64.78	61.64	58.63	55.70	52.84	49.83	49.35
	SD-LoRA [ 44 ]	79.42	75.18	71.62	68.40	65.42	62.64	60.12	58.68	56.80	56.12

Appendix

Table 10: Session-by-session seen-class accuracy

A_{t}

(%) across

t=1,\ldots,10

on 10S-ImageNet-R and 10S-ImageNet-A. The final column (

t=10

) equals

A_{\mathrm{Final}}

, and the row mean equals

\bar{A}

Method	10S-IN-R	10S-IN-A	20S-IN-R	20S-IN-A	Consolidation Rule ( $P_{\mathrm{inv}}$ )	10S-IN-A $A_{\mathrm{Final}}$	20S-IN-A $A_{\mathrm{Final}}$
Seq-LoRA	18.74	24.92	22.58	29.24	Direct Add ( $\Omega_{\mathrm{inv}}=I$ )	61.78	54.18
InfLoRA [ 21 ]	9.48	16.82	11.76	19.32	Uniform Scale ( $\omega=1/t$ )	61.98	54.62
SD-LoRA [ 44 ]	7.56	13.14	8.94	15.68	EMA Decay ( $\alpha=0.7$ )	62.22	55.08
SplitLoRA [ 28 ]	6.52	9.84	7.12	11.16	Diagonal Fisher Weighting	62.41	55.42
LoDA [ 11 ]	6.38	9.24	6.88	10.52	Coupled Dense Solve ( $r_{s}\times r_{s}$ )	62.86	56.15
BISECT	5.84	8.46	6.32	9.58	BISECT Closed-Form $\Omega_{\mathrm{inv}}^{\star}$	62.84	56.12

Appendix

Table 11: Average forgetting

\mathcal{F}_{T}

(%, lower is better) and consolidation rule comparison across 10S and 20S splits.

Rank $r$	10S-IN-R	10S-CUB	10S-IN-A	Damping $\rho$	10S-IN-R	10S-IN-A	Threshold $\eta$	10S-IN-R	10S-CUB	10S-IN-A
$r=8$	79.96	79.68	59.62	$\rho=10^{-6}$	81.92	62.51	$\eta=0.3$	80.24	80.15	60.32
$r=16$	81.28	81.12	61.48	$\rho=10^{-4}$	82.18	62.84	$\eta=0.5$	81.42	81.28	61.65
$r=32$	82.18	82.06	62.84	$\rho=10^{-2}$	82.01	62.58	$\eta=0.7$	82.18	82.06	62.84
$r=64$	82.31	82.15	62.96	$\rho=10^{0}$	80.56	60.88	$\eta=0.9$	80.72	80.84	61.04

Appendix

Table 12: Sensitivity to total rank budget

r

, Cholesky damping

\rho

, and spectral threshold

\eta

(

A_{\mathrm{Final}}

in %).

Configuration	Train GFLOPs	Setup (s/task)	10S-IN-R $A_{\mathrm{Final}}$	10S-IN-A $A_{\mathrm{Final}}$	10S-CUB $A_{\mathrm{Final}}$
Attach to deeper blocks only ( $\ell=9\dots 12$ , QKV)	17.64	3.4	78.82	58.94	78.64
Attach to Query ( $W_{Q}$ ) only ( $\ell=1\dots 12$ )	17.64	10.2	80.34	60.76	80.42
BISECT Default (All blocks $\ell=1\dots 12$ , QKV)	17.71	10.2	82.18	62.84	82.06
Attach to QKV + Output Projection ( $W_{O}$ )	17.75	13.8	82.26	62.91	82.12
Attach to QKV + MLP ( $W_{\mathrm{fc1}},W_{\mathrm{fc2}}$ )	18.04	24.6	82.31	62.95	82.18
BISECT + Analytic Ridge Head ( $m_{\mathrm{RP}}=2048$ )	17.71	11.9	82.94	63.98	83.12

Appendix

Table 13: Ablation of transformer module attachment and hybridization with an analytic ridge readout head (

A_{\mathrm{Final}}

\bar{A}

in %).

Figure 5: Supplementary sensitivity and consolidation studies. (a) Randomized rangefinder oversampling

q

vs. exact SVD on 10S-ImageNet-R. (b) Post-task consolidation rules on 10S- and 20S-ImageNet-A. (c) Total rank budget scaling

r\in\{8,16,32,64\}

across three benchmarks (10S-ImageNet-A shifted by

+18

for visualization).

Figure 6: Subspace persistence and session-by-task accuracy heatmaps on 10S-ImageNet-R. (a) Cross-session Grassmann affinity $\|P_{t}P_{t-\Delta}^{\top}\|_{F}/\sqrt{k}$ as a function of session lag $\Delta$ : the invariant carrier $P_{\mathrm{inv}}$ persists across time while the novelty complement $P_{\mathrm{nov}}$ rotates rapidly to capture emerging task variations. (b, c) Per-task accuracy matrices after each training session $t$ for InfLoRA vs. BISECT.

Exemplar-Free Class-Incremental Learning

See all 9 papers

RASP: Read-All, Sparse-Plasticity Continual Learning in a Stationary-Size LoRA
Nov 2, 2026Wen-Hsuan Liu, Amara Okonkwo, Tom Saliencro +2Continual LoRA Storage-Efficient Continual Learning
University of California, Irvine · University of Washington
MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning
Oct 1, 2026Junkang LiuContinual Learning Low-Rank Adaptation
Tianjin University
Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
Jan 31, 2026Hao Gu, Mao-Lin Luo, Zi-Hao Zhou +3VLM Adaptation Continual Learning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China

Nov 2, 2026J/K move · Enter open · S save

RASP: Read-All, Sparse-Plasticity Continual Learning in a Stationary-Size LoRA

Wen-Hsuan Liu, Amara Okonkwo, Tom Saliencro, Rohan Desai+1

University of California, Irvine · University of Washington

Continual LoRA Storage-Efficient Continual Learning Sparse Fine-Tuning

Open paper AskPDF arXiv

Parameter-efficient continual learning on pre-trained Vision Transformers largely relies on a costly premise: preventing catastrophic forgetting requires either spawning a dedicated LoRA branch per session---causing adapter parameters to scale linearly with task horizon---or applying sparse Mixture-of-Experts gating symmetrically to both forward and backward passes, which sacrifices representational rank at test time. We argue that reading consolidated knowledge and writing new task credit are fundamentally asymmetric operations. Because visual features in a shared low-rank space distribute across rank-1 atoms rather than localizing in disjoint task silos, forward inference should read all atoms at full rank capacity, whereas backward optimization should restrict plasticity to a sparse subset of input-aligned channels. We introduce RASP (Read-All, Sparse-Plasticity LoRA), which adapts a single stationary-size adapter across sequential tasks. At each step, a scene-modulated cosine router selects the top-

q

unlocked atoms to receive gradients, blocks updates on the remaining

r-q

channels, and penalizes Frobenius Gram coherence between active and non-updated output directions. Across sessions, the router's empirical activation trace and each atom's exact spectral norm form a zero-overhead geometric consolidation index

ω_j = \sqrt{\tilde{c}_j \tildeσ_j}

that drives Cauchy step attenuation and boundary anchor locking. Across CIFAR-100, CUB-200, ImageNet-R, and ImageNet-A, RASP matches or surpasses branch-expanding LoRA learners under a strictly constant parameter footprint.

Abstract

Exemplar-Free Class-Incremental Learning

Explore similar work

RASP: Read-All, Sparse-Plasticity Continual Learning in a Stationary-Size LoRA

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation