MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Junkang Liu

cs.LGOct 1, 2026

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Authors: Junkang Liu

Organizations: Tianjin University

Abstract

Low-rank adaptation (LoRA) provides a parameter-efficient approach to continual learning, but its nominal rank can conceal a loss of effective adaptation capacity. We identify spectral plasticity collapse: during sequential adaptation, update energy becomes concentrated in a small subset of singular modes, leaving much of the available low-rank space underutilized. This exposes a limitation of interference avoidance alone: protecting historical representations does not ensure that the remaining adaptation capacity is responsive to new tasks or effectively utilized. To address this problem, we propose MuLoRA, which jointly controls capacity allocation and utilization. First, historical whitening identifies input directions with strong current-task response relative to accumulated historical response, yielding a task-adaptive basis that remains fixed during training. Second, approximate polar orthogonalization of momentum updates reduces spectral concentration within theselected space. An orthonormal basis connects these mechanisms by transferring the factor-update spectrum exactly tothe induced weight update. We establish a max--min characterization of exact subspace selection and derive cumulative spectral bounds under controlled cross-step anisotropy. Across five class-incremental benchmarks and eight incremental settings, MuLoRA achieves the highest mean accuracy in 15 of 16 reported metrics.

Figures & tables

Panel of Figure 1 — Figure 1: Spectral concentration limits effective low-rank capacity. (a,b) Singular-value distributions (Frequency vs. $\sigma$ ) of learned LoRA updates $\Delta W$ at Tasks 1 and 20, with and without spectral balancing. More balanced spectra distribute update energy across a larger number of modes. (c) Conceptual illustration of concentrated versus balanced spectra and their relationship to effective rank. A fixed nominal rank can correspond to markedly different levels of spectral utilization.

Figure 2: Comparison of different LoRA-based CL paradigms. (a) Seq-LoRA employs independent adapters, (b) InfLoRA preserve previous knowledge by restricting updates to isolated directions, (c) MuLoRA fixes a task-responsive orthonormal basis and applies Muon to the trainable factor, transferring spectral balancing directly to weight steps.

Figure 3: Relative-response subspace selection and spectrally controlled optimization. (A) Historical whitening identifies directions with strong current-task response relative to accumulated historical response, from which we construct a row-orthonormal basis $A_{t}$ . (B1) We freeze the backbone and $A_{t}$ , and train only $B_{t}$ . (B2) Approximate polar orthogonalization reshapes the momentum spectrum, balancing individual optimization steps within the selected subspace. (C) After each task, the learned update is merged into the backbone as $W_{t}=W_{t-1}+B_{t}A_{t}$ .

Figure 4: Allocating LoRA capacity using current-task response relative to historical feature statistics.

Figure 5: Spectral control of LoRA updates. An orthonormal input basis transfers the singular spectrum of each factor update to the induced weight update without distortion.

Panel of Figure 6 — Figure 6: Spectral balancing prevents rank collapse. We track effective rank, $\kappa_{+}$ , stable rank of $\Delta W$ , and overall test accuracy across tasks, comparing MuLoRA with and without spectral balancing. SSB maintains high effective rank and stable rank while keeping the nonzero spectrum well conditioned throughout sequential adaptation. Importantly, this geometric advantage persists over the same tasks for which overall class-incremental accuracy is better preserved, suggesting that spectral utilization is closely associated with retained adaptation capacity.

Method	Venue	10S-ImageNetR		10S-ImageNetA		10S-CIFAR100		10S-CUB
Method	Venue	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
L2P	CVPR’22	$72.34_{\pm 0.17}$	$77.36_{\pm 0.64}$	$44.04_{\pm 0.93}$	$51.24_{\pm 2.26}$	$84.06_{\pm 0.88}$	$88.26_{\pm 1.34}$	$67.02_{\pm 1.90}$	$79.62_{\pm 1.60}$
DualPrompt	ECCV’22	$69.10_{\pm 0.62}$	$74.28_{\pm 0.66}$	$53.19_{\pm 0.74}$	$64.59_{\pm 0.08}$	$86.93_{\pm 0.24}$	$91.13_{\pm 0.32}$	$68.48_{\pm 0.47}$	$80.59_{\pm 1.50}$
CODAPrompt	CVPR’23	$73.31_{\pm 0.50}$	$78.47_{\pm 0.53}$	$52.08_{\pm 0.12}$	$63.92_{\pm 0.12}$	$83.21_{\pm 3.39}$	$87.71_{\pm 3.17}$	$77.23_{\pm 1.12}$	$81.90_{\pm 0.85}$
InfLoRA	CVPR’24	$74.75_{\pm 0.64}$	$80.67_{\pm 0.55}$	$49.20_{\pm 1.12}$	$60.92_{\pm 0.61}$	$86.75_{\pm 0.35}$	$91.72_{\pm 0.15}$	$70.82_{\pm 0.23}$	$81.39_{\pm 0.14}$
SD-LoRA	ICLR’25	$77.34_{\pm 0.35}$	$82.04_{\pm 0.24}$	$55.96_{\pm 0.73}$	$64.95_{\pm 1.63}$	$88.01_{\pm 0.31}$	$92.54_{\pm 0.18}$	$77.48_{\pm 0.20}$	$85.59_{\pm 0.44}$

Table 1: Experimental results for 10 incremental tasks on ImageNetR, ImageNetA, CIFAR100, and CUB. We report the averaged results over 3 random seeds. Best and second-best results are bold and underlined .

Method	Venue	5S-ImageNetR		20S-ImageNetR		20S-ImageNetA		10S-DomainNet
Method	Venue	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$	$\mathcal{A}_{Last}\uparrow$	$\mathcal{A}_{Avg}\uparrow$
L2P	CVPR’22	$70.83_{\pm 0.58}$	$78.34_{\pm 0.47}$	$69.64_{\pm 0.42}$	$75.28_{\pm 0.57}$	$40.48_{\pm 1.78}$	$49.62_{\pm 1.46}$	$81.17_{\pm 0.83}$	$87.43_{\pm 0.95}$
DualPrompt	ECCV’22	$73.05_{\pm 0.50}$	$79.47_{\pm 0.40}$	$66.61_{\pm 0.58}$	$72.45_{\pm 0.37}$	$42.28_{\pm 1.94}$	$53.39_{\pm 1.64}$	$81.70_{\pm 0.78}$	$87.80_{\pm 0.99}$
CODAPrompt	CVPR’23	$74.91_{\pm 0.33}$	$79.25_{\pm 0.53}$	$69.96_{\pm 0.50}$	$75.34_{\pm 0.85}$	$44.62_{\pm 1.92}$	$54.86_{\pm 0.50}$	$80.04_{\pm 0.79}$	$86.27_{\pm 0.82}$
InfLoRA	CVPR’24	$77.52_{\pm 0.37}$	$82.01_{\pm 0.12}$	$71.01_{\pm 0.45}$	$77.28_{\pm 0.45}$	$46.81_{\pm 2.30}$	$57.13_{\pm 0.95}$	$81.45_{\pm 0.68}$	$87.55_{\pm 0.57}$
SD-LoRA	ICLR’25	$79.15_{\pm 0.20}$	$83.01_{\pm 0.42}$	$75.26_{\pm 0.37}$	$80.22_{\pm 0.72}$	$49.12_{\pm 1.78}$	$59.63_{\pm 1.57}$	$81.01_{\pm 0.42}$	$86.85_{\pm 0.40}$

Table 2: Experimental results for 5 and 20 incremental sessions on ImageNetR, 20 sessions on ImageNetA, and 10 sessions on DomainNet. The highest results are in bold , and the second highest results are underlined .

Idx	RRS	SSB	10S-ImageNetR		10S-ImageNetA		20S-ImageNetR		20S-ImageNetA
Idx	RRS	SSB	$A_{Last}\uparrow$	$A_{Avg}\uparrow$	$A_{Last}\uparrow$	$A_{Avg}\uparrow$	$A_{Last}\uparrow$	$A_{Avg}\uparrow$	$A_{Last}\uparrow$	$A_{Avg}\uparrow$
1	–	–	$76.23_{\pm 0.18}$	$82.05_{\pm 0.62}$	$57.48_{\pm 1.47}$	$67.34_{\pm 1.86}$	$76.21_{\pm 0.18}$	$80.26_{\pm 0.51}$	$48.62_{\pm 1.28}$	$60.52_{\pm 1.26}$
2	✓	–	$79.23_{\pm 0.28}$	$85.34_{\pm 0.36}$	$59.77_{\pm 0.56}$	$68.91_{\pm 1.08}$	$77.86_{\pm 0.20}$	$83.06_{\pm 0.32}$	$51.92_{\pm 0.66}$	$62.42_{\pm 0.51}$
3	–	✓	$80.74_{\pm 0.28}$	$86.31_{\pm 0.35}$	$61.86_{\pm 0.55}$	$70.26_{\pm 0.98}$	$78.50_{\pm 0.26}$	$84.25_{\pm 0.36}$	$54.32_{\pm 0.64}$	$64.53_{\pm 0.51}$
4	✓	✓	$\mathbf{81.77}_{\pm 0.25}$	$\mathbf{87.28}_{\pm 0.28}$	$\mathbf{62.23}_{\pm 0.52}$	$\mathbf{71.27}_{\pm 1.02}$	$\mathbf{78.93}_{\pm 0.18}$	$\mathbf{85.02}_{\pm 0.21}$	$\mathbf{55.86}_{\pm 0.61}$	$\mathbf{65.61}_{\pm 0.64}$

Table 3: Ablation studies on ImageNetR and ImageNetA under 10 and 20 incremental sessions, evaluating the individual and combined contributions of Relative-Response Spectral Allocation (RRS) and Subspace Spectral Balancing (SSB). Idx 1 uses a fixed random orthonormal

A

and trains only

B

Panel of Figure 7 — Figure 7: Stability–plasticity dynamics across incremental tasks. Overall and new-task accuracy on 20S-ImageNetA and 20S-ImageNetR. MuLoRA better maintains overall performance while preserving strong adaptation to newly introduced classes.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Method	$A_{\mathrm{Last}}\uparrow$	$A_{\mathrm{Avg}}\uparrow$
Trainable $A,B$ + Muon	$51.34\pm 0.35$	$62.46\pm 0.33$
Fixed random orthonormal $A$ + Muon	$54.32\pm 0.64$	$64.53\pm 0.51$
Fixed current-task PCA $A$ + Muon	$52.45\pm 0.34$	$63.56\pm 0.67$
Fixed historical-isolation $A$ + Muon	$54.67\pm 0.25$	$64.76\pm 0.47$
EBLoRA	$52.52\pm 1.01$	$62.16\pm 0.82$
MuLoRA (RRS + SSB)	$\mathbf{55.86}\pm 0.61$	$\mathbf{65.61}\pm 0.64$

Appendix

Table 4: Comparison of subspace construction and spectral control on ImageNet-A with 20 incremental sessions.

A_{\mathrm{Last}}

and

A_{\mathrm{Avg}}

are reported in %.

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 289 papers

May 26, 2026

Energy-Structured Low-Rank Adaptation for Continual Learning

While orthogonal subspace methods try to mitigate task interference in Continual Learning (CL), they often suffer from energy diffusion across the basis, hindering knowledge compaction and exhausting capacity for future tasks. We observe that output feature drift induced by parameter updates is inherently low-rank, and theoretically prove that preserving parameters along the principal directions of this drift minimizes the output reconstruction error. Motivated by this, we propose Energy-Concentrated and Energy-Ordered Low-Rank Adaptation (E

^2

-LoRA). By explicitly ordering and concentrating knowledge into leading ranks, E

^2

-LoRA frees capacity for subsequent tasks. Furthermore, we design a dynamic rank allocation strategy to balance stability and plasticity by jointly optimizing energy retention and model plasticity. Extensive experiments across multiple benchmarks demonstrate that E

^2

-LoRA achieves state-of-the-art performance. Code is available at https://github.com/kiddo127/E2-LoRA.

Longhua Li, Lei Qi, Qi Tian +1

School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China

Continual Learning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning Continual LoRA

Ask PDF

Jun 26, 2026

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20-70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.

Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2

Department of Computer Science, ETH Zurich, Zurich, Switzerland.

Continual Learning Low-Rank Adaptation Incremental Learning Parameter-Efficient Fine-Tuning Continual LoRA

Ask PDF

Jan 31, 2026

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge. Most existing approaches treat continual learning as avoiding interference with past updates, rather than considering what properties make the current task-specific update naturally preserve previously acquired knowledge. From a knowledge-decomposition perspective, we observe that low-rank adaptations exhibit highly imbalanced singular value spectra: a few dominant components absorb most of the adaptation energy, thereby (i) more likely to disrupt previously acquired knowledge and (ii) making the update more vulnerable to interference from subsequent tasks. To enable explicit balance among components, we decouple the magnitude of the task update from its directional structure and formulate it as a constrained optimization problem on a restricted Stiefel manifold. We address this problem using a projected first-order method compatible with standard deep-learning optimizers used in vision-language models. Our method mitigates both backward and forward forgetting, consistently outperforming continual learning baselines. The implementation code is available at https://github.com/haodotgu/EBLoRA.

Hao Gu, Mao-Lin Luo, Zi-Hao Zhou +3

School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China

VLM Adaptation Continual Learning Low-Rank Adaptation Catastrophic Forgetting Continual LoRA

Ask PDF

Abstract

Continual Learning

Explore similar work

Energy-Structured Low-Rank Adaptation for Continual Learning

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation