Low-Rank Adaptation (LoRA) often underperforms full fine-tuning, especially in heterogeneous and continual-learning settings in which substantial forgetting occurs. In general, it suffers from an undesirable learning-forgetting trade-off across diverse fine-tuning settings. We identify a potential contributor: LoRA adapters, especially high-rank ones, frequently fail to realize the available capacity afforded by the configured rank. Specifically, we find that standard LoRA updates have effective rank far below their configured rank, limiting how broadly the update energy is distributed. Therefore, we propose training mechanisms that optimize the original objective while maximizing adapter effective rank. Our main method adds a simple, stable, differentiable, and SVD-free Rényi-2 effective-rank regularizer to the language-modeling loss. We also connect capacity maximization to norm-bounded updates. In mixed-domain rank sweeps on three models evaluated with multiple random seeds, a penalty weight of λ=0.01 lowers mean Wiki perplexity at every tested rank of at least 16 relative to unpenalized LoRA. Mean training perplexity improves or increases by at most 0.02 in all but one model-rank comparison. Single-task and scheduling experiments further show that the balance between learning and retention depends on the configuration. We also observe that several LoRA variants in the literature also suffer from the identified capacity issue, and our approach provides a complementary axis of improvement to LoRA-based parameter-efficient fine-tuning. These results suggest that effective rank is not only a post-hoc diagnostic of adapter quality, but a trainable objective for improving the learning-forgetting frontier. Our code is available at https://github.com/tomohiro-sawada/lora-capacity-utilization
Figures & tables
Figure 1: Effective-rank utilization of standard LoRA adapters across LoRA ranks, decomposed by depth. Each curve reports the mean utilization quotient erank(ΔW)/r averaged over the adapted MLP down_proj layers in that group (all 24 layers; early layers 0–2; middle layers 11–12; late layers 21–23), for GSM8K (left), Hausa (center), and mixed-domain continual-learning (45% GSM8K, 10% WMT19 Czech–English, 45% Aya Hausa; right) training sweeps. Line styles denote depth groups: all layers use solid circles, early layers dashed squares, middle layers dash-dot triangles, and late layers dotted diamonds. The aggregate curve falls at high rank because realized effective rank grows sublinearly in r ; depth groups reveal that underutilization is not uniform—late layers often fall below early and middle layers.
Figure 2: Dense FT denotes dense down-projection fine-tuning. Mixed-domain continual-learning baselines on Qwen2.5-0.5B (base, not Instruct; rank r=896 unless noted). Each point represents a method–penalty configuration; the legend maps dot colors to adapter type. Points show multi-run means or single-run endpoints; sample SDs are reported in Appendix Table 5 . (A) Effective rank erank(ΔW) shown as a vertical strip from 0 to nominal rank 896 (horizontal dashed line), with horizontal offsets only for visibility; higher indicates fuller adapter capacity utilization. (B–D) Learning–forgetting trade-offs: Train PPL (mixture learning) versus Wiki PPL, StarCoder PPL (pretraining proxies), and WMT CS–EN PPL (in-mixture translation retention); lower-left is better. The three Rényi-2 penalty points use different regularization weights λ . Unpenalized PiSSA and LoRA-XS have the highest Wiki perplexities; VeRA, SVF, and TinyLoRA limit forgetting with weaker mixture fit. NB-LoRA approaches the λ=0.01 learning–retention trade-off. Unpenalized PiSSA uses two runs after excluding one resumed execution with apparent data replay; the other multi-run points use three executions. Dense FT, LoRA at λ=1 , and penalized PiSSA at λ=0.1 are single runs. Full numeric values and the PiSSA exclusion are documented in Appendix Table 5 and its accompanying text.
Figure 3: LoRA rank sweep on Qwen2.5-0.5B on the mixed-domain continual-learning proxy setup (aggregated metrics in Appendix Table 9 ). The panels show effective-rank utilization, Wiki perplexity, and training perplexity. Points use the table’s means across three seeds per setting, with repeated executions averaged within each seed first; effective-rank utilization divides the eRank mean by r . Colors indicate penalty weight λ . At r=256,λ=0.1 , the reported means of both Train PPL and Wiki PPL are lower than for standard LoRA at r=128 .
Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module heterogeneity, where the rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget. To address this, we propose an adaptive anisotropic learning-rate model that assigns each rank-one component its own effective learning rate, computed online from training-time signals and mean-normalized per module to preserve the global LR budget. AnLR-LoRA instantiates this model with two signals available during AdamW optimization, namely function-space velocity and Adam SNR, as a lightweight scheme with no extra trainable parameters. Across commonsense reasoning, natural language generation and visual instruction-tuning benchmarks, AnLR-LoRA consistently improves over LoRA while encouraging broader use of rank capacity, with gains that remain robust across a wide range of global learning rates and transfer cleanly to other LoRA variants.
Low-Rank Adaptation (LoRA) has become one of the most widely used fine-tuning mechanisms for adapting large language models to new domains, tasks, and users. Yet adaptation performance alone can obscure an important failure mode: LoRA updates may improve performance on the target distribution while degrading prior capabilities learned during pretraining and alignment. We show that this forgetting becomes especially severe when the adaptation distribution differs substantially from the models original training or alignment distributions. The challenge is amplified in practical settings, where the original training and alignment data are typically unavailable. Motivated by this constraint, we study how LoRA based adaptation balances new learning against forgetting in a replay-free setting, and introduce a simple output space regularizer that can be added directly to existing training pipelines. Our method removes the ground-truth token from both the base and adapted model distributions, renormalizes the remaining probabilities, and applies KL regularization only over the non-target vocabulary. This preserves the base models relative preferences among alternative tokens without directly opposing the cross-entropy signal required for adaptation. As the regularizer acts only at the loss level, it requires no replay data, architectural changes, adapter redesign, or inference-time overhead, and can be applied directly to existing LoRA variants. Across all LoRA variants tested and across various backbones, our method improves the frontier between new learning and forgetting when the adaptation distribution differs substantially from the base models original training or alignment distributions, suggesting a broadly applicable route toward more reliable LLM updating.
Runze Xu, Arpit Garg, Hemanth Saratchandran +1
Australian Institute for Machine Learning Adelaide University Adelaide SA 5000
Low-Rank Adaptation (LoRA) is an effective approach for adapting large pretrained models by learning low-rank weight updates. In practice, the LoRA rank is used to control an adapter's parameter budget and representational capacity. We show that this view is incomplete: while the nominal rank determines the representational capacity, the optimizer shapes how much of that capacity is used in the induced weight-space updates. In a case study of GPT-2 adaptation with LoRA, we observe a strong rank-dependent optimizer effect. Despite using the same nominal rank, AdamW often produces per-step updates with concentrated singular spectra and low effective rank, whereas Muon uses a richer set of directions and benefits more consistently from increasing LoRA rank. These observations motivate ISO-LoRA, an optimizer that couples the LoRA factor updates through spectral descent on the induced tangent perturbation in weight space. ISO-LoRA promotes updates that distribute energy more evenly across singular directions, improving rank utilization while preserving compatibility with the LoRA parameterization. We complement this design with theoretical guarantees showing that ISO-LoRA can achieve higher effective rank than standard factor-wise optimizers through a one-step analysis under a stylized spiked-gradient model. We validate this design on language-model adaptation across 0.1B-7B-parameter models, where ISO-LoRA improves effective rank and downstream performance, with the strongest gains at moderate-to-large LoRA ranks. Our results highlight rank utilization as a key factor in LoRA optimization and suggest that optimizer design offers an important path toward stronger parameter-efficient adaptation.
Zihan Zhu, Zhehang Du, Xuyang Chen +5
University of Pennsylvania · DRW Associates LLC · University of Chicago