BISECT: Bilevel Spectral Decomposition for Low-Rank Continual Learning

Tom Saliencro; Wen-Hsuan Liu; Rohan Desai; Amara Okonkwo; Maya Lindqvist

cs.LGNov 2, 2026

BISECT: Bilevel Spectral Decomposition for Low-Rank Continual Learning

Authors: Tom Saliencro, Wen-Hsuan Liu, Rohan Desai, Amara Okonkwo, Maya Lindqvist

Organizations: University of California, Irvine · University of Washington

Abstract

Exemplar-free continual learning on pre-trained Vision Transformers increasingly relies on Low-Rank Adaptation (LoRA), yet prevailing methods govern the low-rank update space through strict subspace isolation. By projecting each incoming task into the approximate null space of historical token activations, isolation-based learners equate any cross-task feature overlap with catastrophic interference. We show that this orthogonality premise induces a structural bottleneck we term spectral starvation: when down-projection bases are locked before task optimization, the gradient norm reaching the trainable up-projection is upper-bounded by the energy of current-task features captured by the chosen basis. Because natural visual tasks share low- and mid-level covariance structure, historical null tails contain barely $5.5\%$ of incoming task variance, starving new-class plasticity while blocking positive consolidation along shared directions. We propose BISECT (Bilevel Spectral Decomposition), a continual LoRA framework that partitions a fixed rank- $r$ budget into two orthogonal functional blocks: an invariant carrier spanned by the dominant modes of the joint historical-and-current Gram matrix, and a novelty complement extracted via generalized Rayleigh whitening followed by orthogonal complement projection. After single-task training, BISECT folds both blocks back into the frozen backbone using a closed-form precision gate that scales each rank-1 invariant atom by its current-to-joint Rayleigh ratio while preserving novelty atoms at unit gain. Across eight incremental settings on ImageNet-R, ImageNet-A, CIFAR-100, CUB-200, and DomainNet, BISECT outperforms prompt, adapter, and LoRA baselines with zero extra inference parameters or FLOPs, and remains complementary to sparse analytic expansion.

Figures & tables

Figure 1: Diagnosing the orthogonality trap in LoRA continual learning. (a) Final accuracy

A_{\mathrm{Final}}

across four challenging splits: unconstrained LoRA overwrites prior sessions, whereas null-space projection (InfLoRA) collapses under severe distribution shift on 10S-ImageNet-A. (b) Distribution of task-

t

signal across quartiles of the historical eigenspectrum

G_{<t}

on ViT-B/16: the bottom

30\%

null tail holds only

5.5\%

of raw task energy, starving gradient flow, whereas generalized Rayleigh whitening shifts selection to the mid-spectrum where unexplained variance peaks (

51.6\%

). (c) Historical retention vs. emerging-class accuracy on 10S-ImageNet-R; varying the spectral threshold

\eta

traces a Pareto frontier above isolation and gradient-splitting baselines.

Figure 2: Overview of BISECT at session

t

. (I) Using the historical Gram state

G_{<t}

and current-task Gram

G_{t}

, BISECT determines the layerwise rank split

(k_{\mathrm{inv}},k_{\mathrm{nov}})

from the top-

r

spectral energy threshold

\eta

, extracts the invariant carrier

P_{\mathrm{inv}}

from the joint Gram

G_{1:t}

via randomized SVD, and synthesizes the orthogonal novelty complement

P_{\mathrm{nov}}

via Cholesky-whitened generalized Rayleigh extraction. (II) Both basis blocks

P_{\mathrm{inv}},P_{\mathrm{nov}}

remain locked while the up-maps

Q_{\mathrm{inv}},Q_{\mathrm{nov}}

are optimized by SGD on

\mathcal{D}_{t}

. (III) Each rank-1 invariant atom is folded into

W^{(t-1)}

with its closed-form posterior precision weight

\omega_{j}^{\star}\in[0,1]

, while novelty atoms fold at unit weight, yielding zero inference overhead.

Method	Venue	10S-ImageNet-R		10S-ImageNet-A		10S-CIFAR-100		10S-CUB-200
		$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$
L2P [ 41 ]	CVPR’22	72.28	77.31	44.12	51.30	84.10	88.24	67.08	79.58
DualPrompt [ 40 ]	ECCV’22	69.15	74.32	53.14	64.52	86.88	91.09	68.52	80.64
CODA-Prompt [ 30 ]	CVPR’23	73.26	78.42	52.15	63.88	83.29	87.76	77.18	81.86
EASE [ 46 ]	CVPR’24	76.18	81.74	55.38	65.12	87.84	92.18	76.85	84.92
InfLoRA [ 21 ]	CVPR’24	74.68	80.62	49.35	60.86	86.72	91.68	70.88	81.42

Table 1: Comparison with state-of-the-art rehearsal-free continual learners on four 10-session (10S) benchmarks using ViT-B/16. Every method is evaluated across all four datasets (

A_{\mathrm{Final}}

\bar{A}

in %, averaged over 3 seeds). Best results are in bold , second best underlined .

Method	Venue	5S-ImageNet-R		20S-ImageNet-R		20S-ImageNet-A		10S-DomainNet
		$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$
L2P [ 41 ]	CVPR’22	70.88	78.39	69.58	75.22	40.54	49.68	81.12	87.38
DualPrompt [ 40 ]	ECCV’22	73.10	79.52	66.68	72.51	42.34	53.44	81.66	87.76
CODA-Prompt [ 30 ]	CVPR’23	74.86	79.21	69.92	75.30	44.58	54.82	80.09	86.31
EASE [ 46 ]	CVPR’24	78.64	83.12	73.84	79.46	48.26	58.54	82.18	87.92
InfLoRA [ 21 ]	CVPR’24	77.58	82.06	71.14	77.34	46.95	57.21	81.42	87.51

Table 2: Evaluation across variable task horizons (5S and 20S on ImageNet-R, 20S on ImageNet-A) and multi-domain class-incremental streams (10S-DomainNet). All baseline cells are evaluated under the unified ViT-B/16 protocol (

A_{\mathrm{Final}}

\bar{A}

in %).

Variant	$P_{\mathrm{inv}}$	$P_{\mathrm{nov}}$	$\Omega_{\mathrm{inv}}^{\star}$	Adapt. $k$	10S-IN-R	10S-IN-A	20S-IN-R	20S-IN-A
1. Seq-LoRA	–	–	–	–	71.88	52.96	68.04	44.92
2. Invariant only	✓	–	–	–	79.14	59.64	76.92	51.84
3. Novelty only	–	✓	–	–	79.32	60.02	76.16	51.60
4. Bilevel (fixed $k$ )	✓	✓	–	–	81.36	61.78	79.02	54.18
5. + Precision gate	✓	✓	✓	–	81.84	62.46	79.31	55.54
6. BISECT (Full)	✓	✓	✓	✓	82.18	62.84	79.48	56.12

Table 3: Component ablation study of BISECT reporting

A_{\mathrm{Final}}

(%) across 10- and 20-session ImageNet-R (IN-R) and ImageNet-A (IN-A).

Figure 3: Mechanistic verification on ViT-B/16. (a) Layerwise generalized Rayleigh selectivity

\mathcal{R}(P)

across blocks

1\dots 12

. (b) Layer-adaptive rank split

(k_{\mathrm{inv}},k_{\mathrm{nov}})

\eta=0.7

. (c) Cumulative old-class representation drift

\|X_{<t}\Delta W_{t}^{\top}\|_{F}

on 10S-ImageNet-R; precision folding (

\Omega_{\mathrm{inv}}^{\star}

) removes

83.4\%

of unscaled drift.

Figure 4: Stream dynamics, sensitivity, and setup efficiency. (a) Session-wise seen-class accuracy on 10S-ImageNet-A. (b) Sensitivity to

\eta\in[0.3,0.9]

across three benchmarks (10S-IN-A shifted by

+19

). (c) Final accuracy vs. per-session setup time on 20S-ImageNet-R (

10.2

s/task,

12.4\times

faster than LoDA).

Benchmark	Subspace Constructor	$\mathcal{R}(P)\uparrow$	$A_{\mathrm{Final}}\uparrow$	$\bar{A}\uparrow$
10S-IN-R	Random Orthogonal	$1.01\times$	68.34	79.12
	Adam-NSCL [ 37 ]	$1.46\times$	74.58	82.34
	InfLoRA [ 21 ]	$1.74\times$	75.08	82.52
	Unwhitened Task PCA	$1.36\times$	76.84	83.68
	BISECT ( $P_{\mathrm{nov}}$ )	$\mathbf{3.95\times}$	79.32	85.38
10S-IN-A	Random Orthogonal	$1.02\times$	53.56	64.41

Table 4: Controlled comparison of rank-

32

plastic subspace constructors (without invariant carrier). We report mean selectivity ratio

\mathcal{R}(P)

and

A_{\mathrm{Final}}

\bar{A}

(%).

Method	Gram Mem.	Train	Setup	Extra Test	Test	$A_{\mathrm{Final}}\uparrow$
	(MB)	GFLOPs	(s/task)	Params (M)	GFLOPs	(%)
InfLoRA [ 21 ]	11.3	17.67	51.8	0.00	17.60	71.14
SD-LoRA [ 44 ]	14.1	20.51	15.2	0.00	17.60	75.18
VPT-NSP [ 23 ]	28.4	17.72	55.8	0.09	17.72	76.08
CPG [ 8 ]	28.4	17.72	51.4	1.66	17.72	75.28
SplitLoRA [ 28 ]	27.0	18.42	33.6	0.00	17.60	78.68

Table 5: Computational, memory, and inference cost audit on 20S-ImageNet-R (single RTX 4090 GPU). Setup latency measures per-session subspace construction.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Oversampling $q$	$\sin\Theta_{\max}\downarrow$	Setup Time (s/task) $\downarrow$	Speedup vs. Exact	10S-IN-R $A_{\mathrm{Final}}$ (%)	10S-IN-A $A_{\mathrm{Final}}$ (%)
$q=0$ (no oversampling)	0.138	6.2	$8.2\times$	81.48	62.04
$q=2$	0.054	6.9	$7.4\times$	81.91	62.48
$q=4$	0.022	8.1	$6.3\times$	82.09	62.71
$q=8$ (BISECT default)	0.008	10.2	$\mathbf{5.0\times}$	82.18	62.84
$q=12$	0.005	12.4	$4.1\times$	82.19	62.85
$q=16$	0.004	14.8	$3.4\times$	82.19	62.85

Appendix

Table 6: Sensitivity to the RSVD oversampling parameter

q

across all

12

ViT-B/16 blocks on 10S-ImageNet-R and 10S-ImageNet-A.

\sin\Theta_{\max}

measures the worst-case principal angle deviation from exact full SVD.

Benchmark Split	Total Classes	Sessions $T$	Classes / Session	Train Images	Base LR	Batch Size / Epochs
5S-ImageNet-R [ 12 ]	200	5	40	24,000	0.020	48 / 20
10S-ImageNet-R [ 12 ]	200	10	20	24,000	0.020	48 / 20
20S-ImageNet-R [ 12 ]	200	20	10	24,000	0.020	48 / 20
10S-ImageNet-A [ 13 ]	200	10	20	5,981	0.010	48 / 20
20S-ImageNet-A [ 13 ]	200	20	10	5,981	0.010	48 / 20
10S-CIFAR-100 [ 18 ]	100	10	10	50,000	0.010	48 / 20

Appendix

Table 7: Summary of benchmarks, class-incremental splits, and session training hyperparameters.

Method	10S-ImageNet-R		10S-ImageNet-A		10S-CIFAR-100		10S-CUB-200
	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$
L2P [ 41 ]	$72.28_{\pm 0.19}$	$77.31_{\pm 0.61}$	$44.12_{\pm 0.89}$	$51.30_{\pm 2.18}$	$84.10_{\pm 0.84}$	$88.24_{\pm 1.29}$	$67.08_{\pm 1.84}$	$79.58_{\pm 1.54}$
DualPrompt [ 40 ]	$69.15_{\pm 0.58}$	$74.32_{\pm 0.64}$	$53.14_{\pm 0.71}$	$64.52_{\pm 0.14}$	$86.88_{\pm 0.26}$	$91.09_{\pm 0.34}$	$68.52_{\pm 0.49}$	$80.64_{\pm 1.46}$
CODA-Prompt [ 30 ]	$73.26_{\pm 0.48}$	$78.42_{\pm 0.52}$	$52.15_{\pm 0.18}$	$63.88_{\pm 0.16}$	$83.29_{\pm 3.24}$	$87.76_{\pm 3.08}$	$77.18_{\pm 1.08}$	$81.86_{\pm 0.82}$
EASE [ 46 ]	$76.18_{\pm 0.34}$	$81.74_{\pm 0.41}$	$55.38_{\pm 0.68}$	$65.12_{\pm 0.84}$	$87.84_{\pm 0.29}$	$92.18_{\pm 0.31}$	$76.85_{\pm 0.44}$	$84.92_{\pm 0.52}$
InfLoRA [ 21 ]	$74.68_{\pm 0.61}$	$80.62_{\pm 0.53}$	$49.35_{\pm 1.08}$	$60.86_{\pm 0.59}$	$86.72_{\pm 0.33}$	$91.68_{\pm 0.18}$	$70.88_{\pm 0.25}$	$81.42_{\pm 0.17}$

Appendix

Table 8: Full 10-session benchmark results (mean

\pm

std over 3 class-order seeds), corresponding to Table 1 .

Method	5S-ImageNet-R		20S-ImageNet-R		20S-ImageNet-A		10S-DomainNet
	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$	$A_{\mathrm{Final}}$	$\bar{A}$
L2P [ 41 ]	$70.88_{\pm 0.56}$	$78.39_{\pm 0.45}$	$69.58_{\pm 0.44}$	$75.22_{\pm 0.55}$	$40.54_{\pm 1.72}$	$49.68_{\pm 1.42}$	$81.12_{\pm 0.81}$	$87.38_{\pm 0.92}$
DualPrompt [ 40 ]	$73.10_{\pm 0.48}$	$79.52_{\pm 0.39}$	$66.68_{\pm 0.56}$	$72.51_{\pm 0.38}$	$42.34_{\pm 1.88}$	$53.44_{\pm 1.58}$	$81.66_{\pm 0.75}$	$87.76_{\pm 0.94}$
CODA-Prompt [ 30 ]	$74.86_{\pm 0.35}$	$79.21_{\pm 0.51}$	$69.92_{\pm 0.48}$	$75.30_{\pm 0.82}$	$44.58_{\pm 1.86}$	$54.82_{\pm 0.52}$	$80.09_{\pm 0.76}$	$86.31_{\pm 0.79}$
EASE [ 46 ]	$78.64_{\pm 0.31}$	$83.12_{\pm 0.36}$	$73.84_{\pm 0.42}$	$79.46_{\pm 0.51}$	$48.26_{\pm 1.44}$	$58.54_{\pm 1.18}$	$82.18_{\pm 0.45}$	$87.92_{\pm 0.48}$
InfLoRA [ 21 ]	$77.58_{\pm 0.36}$	$82.06_{\pm 0.15}$	$71.14_{\pm 0.43}$	$77.34_{\pm 0.44}$	$46.95_{\pm 2.18}$	$57.21_{\pm 0.92}$	$81.42_{\pm 0.65}$	$87.51_{\pm 0.54}$

Appendix

Table 9: Full variable-horizon and DomainNet results (mean

\pm

std over 3 seeds), corresponding to Table 2 .

Benchmark	Method	$t=1$	$t=2$	$t=3$	$t=4$	$t=5$	$t=6$	$t=7$	$t=8$	$t=9$	$t=10$ ( $A_{\mathrm{Final}}$ )
10S-IN-R	Seq-LoRA	92.04	88.78	86.18	83.76	81.48	79.28	77.14	75.06	73.02	71.88
	InfLoRA [ 21 ]	89.28	86.71	84.64	82.72	80.92	79.17	77.48	75.84	74.76	74.68
	LoDA [ 11 ]	93.72	91.70	90.08	88.58	87.16	85.79	84.46	83.16	81.89	81.86
	BISECT (Ours)	93.96	91.95	90.34	88.83	87.39	86.01	84.68	83.38	82.18	82.18
10S-IN-A	InfLoRA [ 21 ]	76.10	71.65	68.08	64.78	61.64	58.63	55.70	52.84	49.83	49.35
	SD-LoRA [ 44 ]	79.42	75.18	71.62	68.40	65.42	62.64	60.12	58.68	56.80	56.12

Appendix

Table 10: Session-by-session seen-class accuracy

A_{t}

(%) across

t=1,\ldots,10

on 10S-ImageNet-R and 10S-ImageNet-A. The final column (

t=10

) equals

A_{\mathrm{Final}}

, and the row mean equals

\bar{A}

Method	10S-IN-R	10S-IN-A	20S-IN-R	20S-IN-A	Consolidation Rule ( $P_{\mathrm{inv}}$ )	10S-IN-A $A_{\mathrm{Final}}$	20S-IN-A $A_{\mathrm{Final}}$
Seq-LoRA	18.74	24.92	22.58	29.24	Direct Add ( $\Omega_{\mathrm{inv}}=I$ )	61.78	54.18
InfLoRA [ 21 ]	9.48	16.82	11.76	19.32	Uniform Scale ( $\omega=1/t$ )	61.98	54.62
SD-LoRA [ 44 ]	7.56	13.14	8.94	15.68	EMA Decay ( $\alpha=0.7$ )	62.22	55.08
SplitLoRA [ 28 ]	6.52	9.84	7.12	11.16	Diagonal Fisher Weighting	62.41	55.42
LoDA [ 11 ]	6.38	9.24	6.88	10.52	Coupled Dense Solve ( $r_{s}\times r_{s}$ )	62.86	56.15
BISECT	5.84	8.46	6.32	9.58	BISECT Closed-Form $\Omega_{\mathrm{inv}}^{\star}$	62.84	56.12

Appendix

Table 11: Average forgetting

\mathcal{F}_{T}

(%, lower is better) and consolidation rule comparison across 10S and 20S splits.

Rank $r$	10S-IN-R	10S-CUB	10S-IN-A	Damping $\rho$	10S-IN-R	10S-IN-A	Threshold $\eta$	10S-IN-R	10S-CUB	10S-IN-A
$r=8$	79.96	79.68	59.62	$\rho=10^{-6}$	81.92	62.51	$\eta=0.3$	80.24	80.15	60.32
$r=16$	81.28	81.12	61.48	$\rho=10^{-4}$	82.18	62.84	$\eta=0.5$	81.42	81.28	61.65
$r=32$	82.18	82.06	62.84	$\rho=10^{-2}$	82.01	62.58	$\eta=0.7$	82.18	82.06	62.84
$r=64$	82.31	82.15	62.96	$\rho=10^{0}$	80.56	60.88	$\eta=0.9$	80.72	80.84	61.04

Appendix

Table 12: Sensitivity to total rank budget

r

, Cholesky damping

\rho

, and spectral threshold

\eta

(

A_{\mathrm{Final}}

in %).

Configuration	Train GFLOPs	Setup (s/task)	10S-IN-R $A_{\mathrm{Final}}$	10S-IN-A $A_{\mathrm{Final}}$	10S-CUB $A_{\mathrm{Final}}$
Attach to deeper blocks only ( $\ell=9\dots 12$ , QKV)	17.64	3.4	78.82	58.94	78.64
Attach to Query ( $W_{Q}$ ) only ( $\ell=1\dots 12$ )	17.64	10.2	80.34	60.76	80.42
BISECT Default (All blocks $\ell=1\dots 12$ , QKV)	17.71	10.2	82.18	62.84	82.06
Attach to QKV + Output Projection ( $W_{O}$ )	17.75	13.8	82.26	62.91	82.12
Attach to QKV + MLP ( $W_{\mathrm{fc1}},W_{\mathrm{fc2}}$ )	18.04	24.6	82.31	62.95	82.18
BISECT + Analytic Ridge Head ( $m_{\mathrm{RP}}=2048$ )	17.71	11.9	82.94	63.98	83.12

Appendix

Table 13: Ablation of transformer module attachment and hybridization with an analytic ridge readout head (

A_{\mathrm{Final}}

\bar{A}

in %).

Figure 5: Supplementary sensitivity and consolidation studies. (a) Randomized rangefinder oversampling

q

vs. exact SVD on 10S-ImageNet-R. (b) Post-task consolidation rules on 10S- and 20S-ImageNet-A. (c) Total rank budget scaling

r\in\{8,16,32,64\}

across three benchmarks (10S-ImageNet-A shifted by

+18

for visualization).

Figure 6: Subspace persistence and session-by-task accuracy heatmaps on 10S-ImageNet-R. (a) Cross-session Grassmann affinity $\|P_{t}P_{t-\Delta}^{\top}\|_{F}/\sqrt{k}$ as a function of session lag $\Delta$ : the invariant carrier $P_{\mathrm{inv}}$ persists across time while the novelty complement $P_{\mathrm{nov}}$ rotates rapidly to capture emerging task variations. (b, c) Per-task accuracy matrices after each training session $t$ for InfLoRA vs. BISECT.

Exemplar-Free Class-Incremental Learning

See all 9 papers

Nov 2, 2026

RASP: Read-All, Sparse-Plasticity Continual Learning in a Stationary-Size LoRA

Parameter-efficient continual learning on pre-trained Vision Transformers largely relies on a costly premise: preventing catastrophic forgetting requires either spawning a dedicated LoRA branch per session---causing adapter parameters to scale linearly with task horizon---or applying sparse Mixture-of-Experts gating symmetrically to both forward and backward passes, which sacrifices representational rank at test time. We argue that reading consolidated knowledge and writing new task credit are fundamentally asymmetric operations. Because visual features in a shared low-rank space distribute across rank-1 atoms rather than localizing in disjoint task silos, forward inference should read all atoms at full rank capacity, whereas backward optimization should restrict plasticity to a sparse subset of input-aligned channels. We introduce RASP (Read-All, Sparse-Plasticity LoRA), which adapts a single stationary-size adapter across sequential tasks. At each step, a scene-modulated cosine router selects the top-

q

unlocked atoms to receive gradients, blocks updates on the remaining

r-q

channels, and penalizes Frobenius Gram coherence between active and non-updated output directions. Across sessions, the router's empirical activation trace and each atom's exact spectral norm form a zero-overhead geometric consolidation index

ω_j = \sqrt{\tilde{c}_j \tildeσ_j}

that drives Cauchy step attenuation and boundary anchor locking. Across CIFAR-100, CUB-200, ImageNet-R, and ImageNet-A, RASP matches or surpasses branch-expanding LoRA learners under a strictly constant parameter footprint.

Wen-Hsuan Liu, Amara Okonkwo, Tom Saliencro +2

University of California, Irvine · University of Washington

Continual LoRA Storage-Efficient Continual Learning Sparse Fine-Tuning

Ask PDF

Oct 1, 2026

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Low-rank adaptation (LoRA) provides a parameter-efficient approach to continual learning, but its nominal rank can conceal a loss of effective adaptation capacity. We identify spectral plasticity collapse: during sequential adaptation, update energy becomes concentrated in a small subset of singular modes, leaving much of the available low-rank space underutilized. This exposes a limitation of interference avoidance alone: protecting historical representations does not ensure that the remaining adaptation capacity is responsive to new tasks or effectively utilized. To address this problem, we propose MuLoRA, which jointly controls capacity allocation and utilization. First, historical whitening identifies input directions with strong current-task response relative to accumulated historical response, yielding a task-adaptive basis that remains fixed during training. Second, approximate polar orthogonalization of momentum updates reduces spectral concentration within theselected space. An orthonormal basis connects these mechanisms by transferring the factor-update spectrum exactly tothe induced weight update. We establish a max--min characterization of exact subspace selection and derive cumulative spectral bounds under controlled cross-step anisotropy. Across five class-incremental benchmarks and eight incremental settings, MuLoRA achieves the highest mean accuracy in 15 of 16 reported metrics.

Junkang Liu

Tianjin University

Continual Learning Low-Rank Adaptation Continual LoRA Adapter-Based Continual Learning Class-Incremental Learning

Ask PDF

Jan 31, 2026

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge. From a knowledge-decomposition perspective, we observe that low-rank adaptations exhibit highly imbalanced singular value spectra: a few dominant components absorb most of the adaptation energy, thereby (i) more likely to disrupt previously acquired knowledge and (ii) making the update more vulnerable to interference from subsequent tasks. To enable explicit balance among components, we decouple the magnitude of the task update from its directional structure and formulate it as a constrained optimization problem on a restricted Stiefel manifold. We address this problem using a projected first-order method compatible with standard deep-learning optimizers used in vision-language models. Our method mitigates both backward and forward forgetting, consistently outperforming continual learning baselines. The implementation code is available at https://github.com/haodotgu/EBLoRA.

Hao Gu, Mao-Lin Luo, Zi-Hao Zhou +3

School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China

VLM Adaptation Continual Learning Low-Rank Adaptation Catastrophic Forgetting Continual LoRA

Ask PDF

Abstract

Exemplar-Free Class-Incremental Learning

Explore similar work

RASP: Read-All, Sparse-Plasticity Continual Learning in a Stationary-Size LoRA

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation