MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Junkang Liu

cs.LGOct 1, 2026

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Authors: Junkang Liu

Organizations: Tianjin University

Abstract

Low-rank adaptation (LoRA) provides a parameter-efficient approach to continual learning, but its nominal rank can conceal a loss of effective adaptation capacity. We identify spectral plasticity collapse: during sequential adaptation, update energy becomes concentrated in a small subset of singular modes, leaving much of the available low-rank space underutilized. This exposes a limitation of interference avoidance alone: protecting historical representations does not ensure that the remaining adaptation capacity is responsive to new tasks or effectively utilized. To address this problem, we propose MuLoRA, which jointly controls capacity allocation and utilization. First, historical whitening identifies input directions with strong current-task response relative to accumulated historical response, yielding a task-adaptive basis that remains fixed during training. Second, approximate polar orthogonalization of momentum updates reduces spectral concentration within theselected space. An orthonormal basis connects these mechanisms by transferring the factor-update spectrum exactly tothe induced weight update. We establish a max--min characterization of exact subspace selection and derive cumulative spectral bounds under controlled cross-step anisotropy. Across five class-incremental benchmarks and eight incremental settings, MuLoRA achieves the highest mean accuracy in 15 of 16 reported metrics.

Figures & tables

Panel of Figure 1 — Figure 1: Spectral concentration limits effective low-rank capacity. (a,b) Singular-value distributions (Frequency vs. $\sigma$ ) of learned LoRA updates $\Delta W$ at Tasks 1 and 20, with and without spectral balancing. More balanced spectra distribute update energy across a larger number of modes. (c) Conceptual illustration of concentrated versus balanced spectra and their relationship to effective rank. A fixed nominal rank can correspond to markedly different levels of spectral utilization.

Figure 2: Comparison of different LoRA-based CL paradigms. (a) Seq-LoRA employs independent adapters, (b) InfLoRA preserve previous knowledge by restricting updates to isolated directions, (c) MuLoRA fixes a task-responsive orthonormal basis and applies Muon to the trainable factor, transferring spectral balancing directly to weight steps.

Figure 3: Relative-response subspace selection and spectrally controlled optimization. (A) Historical whitening identifies directions with strong current-task response relative to accumulated historical response, from which we construct a row-orthonormal basis $A_{t}$ . (B1) We freeze the backbone and $A_{t}$ , and train only $B_{t}$ . (B2) Approximate polar orthogonalization reshapes the momentum spectrum, balancing individual optimization steps within the selected subspace. (C) After each task, the learned update is merged into the backbone as $W_{t}=W_{t-1}+B_{t}A_{t}$ .

Figure 4: Allocating LoRA capacity using current-task response relative to historical feature statistics.

Figure 5: Spectral control of LoRA updates. An orthonormal input basis transfers the singular spectrum of each factor update to the induced weight update without distortion.

Panel of Figure 6 — Figure 6: Spectral balancing prevents rank collapse. We track effective rank, $\kappa_{+}$ , stable rank of $\Delta W$ , and overall test accuracy across tasks, comparing MuLoRA with and without spectral balancing. SSB maintains high effective rank and stable rank while keeping the nonzero spectrum well conditioned throughout sequential adaptation. Importantly, this geometric advantage persists over the same tasks for which overall class-incremental accuracy is better preserved, suggesting that spectral utilization is closely associated with retained adaptation capacity.

Method	Venue	10S-ImageNetR		10S-ImageNetA		10S-CIFAR100		10S-CUB
Method	Venue	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
L2P	CVPR’22	$72.34_{\pm 0.17}$	$77.36_{\pm 0.64}$	$44.04_{\pm 0.93}$	$51.24_{\pm 2.26}$	$84.06_{\pm 0.88}$	$88.26_{\pm 1.34}$	$67.02_{\pm 1.90}$	$79.62_{\pm 1.60}$
DualPrompt	ECCV’22	$69.10_{\pm 0.62}$	$74.28_{\pm 0.66}$	$53.19_{\pm 0.74}$	$64.59_{\pm 0.08}$	$86.93_{\pm 0.24}$	$91.13_{\pm 0.32}$	$68.48_{\pm 0.47}$	$80.59_{\pm 1.50}$
CODAPrompt	CVPR’23	$73.31_{\pm 0.50}$	$78.47_{\pm 0.53}$	$52.08_{\pm 0.12}$	$63.92_{\pm 0.12}$	$83.21_{\pm 3.39}$	$87.71_{\pm 3.17}$	$77.23_{\pm 1.12}$	$81.90_{\pm 0.85}$
InfLoRA	CVPR’24	$74.75_{\pm 0.64}$	$80.67_{\pm 0.55}$	$49.20_{\pm 1.12}$	$60.92_{\pm 0.61}$	$86.75_{\pm 0.35}$	$91.72_{\pm 0.15}$	$70.82_{\pm 0.23}$	$81.39_{\pm 0.14}$
SD-LoRA	ICLR’25	$77.34_{\pm 0.35}$	$82.04_{\pm 0.24}$	$55.96_{\pm 0.73}$	$64.95_{\pm 1.63}$	$88.01_{\pm 0.31}$	$92.54_{\pm 0.18}$	$77.48_{\pm 0.20}$	$85.59_{\pm 0.44}$

Table 1: Experimental results for 10 incremental tasks on ImageNetR, ImageNetA, CIFAR100, and CUB. We report the averaged results over 3 random seeds. Best and second-best results are bold and underlined .

Method	Venue	5S-ImageNetR		20S-ImageNetR		20S-ImageNetA		10S-DomainNet
Method	Venue	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
L2P	CVPR’22	$70.83_{\pm 0.58}$	$78.34_{\pm 0.47}$	$69.64_{\pm 0.42}$	$75.28_{\pm 0.57}$	$40.48_{\pm 1.78}$	$49.62_{\pm 1.46}$	$81.17_{\pm 0.83}$	$87.43_{\pm 0.95}$
DualPrompt	ECCV’22	$73.05_{\pm 0.50}$	$79.47_{\pm 0.40}$	$66.61_{\pm 0.58}$	$72.45_{\pm 0.37}$	$42.28_{\pm 1.94}$	$53.39_{\pm 1.64}$	$81.70_{\pm 0.78}$	$87.80_{\pm 0.99}$
CODAPrompt	CVPR’23	$74.91_{\pm 0.33}$	$79.25_{\pm 0.53}$	$69.96_{\pm 0.50}$	$75.34_{\pm 0.85}$	$44.62_{\pm 1.92}$	$54.86_{\pm 0.50}$	$80.04_{\pm 0.79}$	$86.27_{\pm 0.82}$
InfLoRA	CVPR’24	$77.52_{\pm 0.37}$	$82.01_{\pm 0.12}$	$71.01_{\pm 0.45}$	$77.28_{\pm 0.45}$	$46.81_{\pm 2.30}$	$57.13_{\pm 0.95}$	$81.45_{\pm 0.68}$	$87.55_{\pm 0.57}$
SD-LoRA	ICLR’25	$79.15_{\pm 0.20}$	$83.01_{\pm 0.42}$	$75.26_{\pm 0.37}$	$80.22_{\pm 0.72}$	$49.12_{\pm 1.78}$	$59.63_{\pm 1.57}$	$81.01_{\pm 0.42}$	$86.85_{\pm 0.40}$

Table 2: Experimental results for 5 and 20 incremental sessions on ImageNetR, 20 sessions on ImageNetA, and 10 sessions on DomainNet. The highest results are in bold , and the second highest results are underlined .

Idx	RRS	SSB	10S-ImageNetR		10S-ImageNetA		20S-ImageNetR		20S-ImageNetA
Idx	RRS	SSB	$A_{Last}\uparrow$	$A_{Avg}\uparrow$	$A_{Last}\uparrow$	$A_{Avg}\uparrow$	$A_{Last}\uparrow$	$A_{Avg}\uparrow$	$A_{Last}\uparrow$	$A_{Avg}\uparrow$
1	–	–	$76.23_{\pm 0.18}$	$82.05_{\pm 0.62}$	$57.48_{\pm 1.47}$	$67.34_{\pm 1.86}$	$76.21_{\pm 0.18}$	$80.26_{\pm 0.51}$	$48.62_{\pm 1.28}$	$60.52_{\pm 1.26}$
2	✓	–	$79.23_{\pm 0.28}$	$85.34_{\pm 0.36}$	$59.77_{\pm 0.56}$	$68.91_{\pm 1.08}$	$77.86_{\pm 0.20}$	$83.06_{\pm 0.32}$	$51.92_{\pm 0.66}$	$62.42_{\pm 0.51}$
3	–	✓	$80.74_{\pm 0.28}$	$86.31_{\pm 0.35}$	$61.86_{\pm 0.55}$	$70.26_{\pm 0.98}$	$78.50_{\pm 0.26}$	$84.25_{\pm 0.36}$	$54.32_{\pm 0.64}$	$64.53_{\pm 0.51}$
4	✓	✓	$\mathbf{81.77}_{\pm 0.25}$	$\mathbf{87.28}_{\pm 0.28}$	$\mathbf{62.23}_{\pm 0.52}$	$\mathbf{71.27}_{\pm 1.02}$	$\mathbf{78.93}_{\pm 0.18}$	$\mathbf{85.02}_{\pm 0.21}$	$\mathbf{55.86}_{\pm 0.61}$	$\mathbf{65.61}_{\pm 0.64}$

Table 3: Ablation studies on ImageNetR and ImageNetA under 10 and 20 incremental sessions, evaluating the individual and combined contributions of Relative-Response Spectral Allocation (RRS) and Subspace Spectral Balancing (SSB). Idx 1 uses a fixed random orthonormal

A

and trains only

B

Panel of Figure 7 — Figure 7: Stability–plasticity dynamics across incremental tasks. Overall and new-task accuracy on 20S-ImageNetA and 20S-ImageNetR. MuLoRA better maintains overall performance while preserving strong adaptation to newly introduced classes.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Method	$A_{\mathrm{Last}}\uparrow$	$A_{\mathrm{Avg}}\uparrow$
Trainable $A,B$ + Muon	$51.34\pm 0.35$	$62.46\pm 0.33$
Fixed random orthonormal $A$ + Muon	$54.32\pm 0.64$	$64.53\pm 0.51$
Fixed current-task PCA $A$ + Muon	$52.45\pm 0.34$	$63.56\pm 0.67$
Fixed historical-isolation $A$ + Muon	$54.67\pm 0.25$	$64.76\pm 0.47$
EBLoRA	$52.52\pm 1.01$	$62.16\pm 0.82$
MuLoRA (RRS + SSB)	$\mathbf{55.86}\pm 0.61$	$\mathbf{65.61}\pm 0.64$

Appendix

Table 4: Comparison of subspace construction and spectral control on ImageNet-A with 20 incremental sessions.

A_{\mathrm{Last}}

and

A_{\mathrm{Avg}}

are reported in %.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 289 papers

Energy-Structured Low-Rank Adaptation for Continual Learning
May 26, 2026Longhua Li, Lei Qi, Qi Tian +1Continual Learning Low-Rank Adaptation
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China
When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
Jun 26, 2026Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2Continual Learning Low-Rank Adaptation
Department of Computer Science, ETH Zurich, Zurich, Switzerland.
Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
Jan 31, 2026Hao Gu, Mao-Lin Luo, Zi-Hao Zhou +3VLM Adaptation Continual Learning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China

May 26, 2026J/K move · Enter open · S save

Energy-Structured Low-Rank Adaptation for Continual Learning

Longhua Li, Lei Qi, Qi Tian, Xin Geng

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China

Continual Learning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Continual LoRA

Open paper AskPDF arXiv

While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose Energy-Concentrated and Energy-Ordered Low-Rank Adaptation (E

^2

-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E

^2

-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E

^2

-LoRA achieves state-of-the-art performance. Code is available at https://github.com/kiddo127/E2-LoRA.

Abstract

Continual Learning

Explore similar work

Energy-Structured Low-Rank Adaptation for Continual Learning

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation