cs.LGNov 12, 2026

Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off

Authors: Tomohiro Sawada, Kartik Goyal

Organizations: School of Interactive Computing, Georgia Institute of Technology

Abstract

Low-Rank Adaptation (LoRA) often underperforms full fine-tuning, especially in heterogeneous and continual-learning settings in which substantial forgetting occurs. In general, it suffers from an undesirable learning-forgetting trade-off across diverse fine-tuning settings. We identify a potential contributor: LoRA adapters, especially high-rank ones, frequently fail to realize the available capacity afforded by the configured rank. Specifically, we find that standard LoRA updates have effective rank far below their configured rank, limiting how broadly the update energy is distributed. Therefore, we propose training mechanisms that optimize the original objective while maximizing adapter effective rank. Our main method adds a simple, stable, differentiable, and SVD-free Rényi-2 effective-rank regularizer to the language-modeling loss. We also connect capacity maximization to norm-bounded updates. In mixed-domain rank sweeps on three models evaluated with multiple random seeds, a penalty weight of λ=0.01λ=0.01 lowers mean Wiki perplexity at every tested rank of at least 1616 relative to unpenalized LoRA. Mean training perplexity improves or increases by at most 0.020.02 in all but one model-rank comparison. Single-task and scheduling experiments further show that the balance between learning and retention depends on the configuration. We also observe that several LoRA variants in the literature also suffer from the identified capacity issue, and our approach provides a complementary axis of improvement to LoRA-based parameter-efficient fine-tuning. These results suggest that effective rank is not only a post-hoc diagnostic of adapter quality, but a trainable objective for improving the learning-forgetting frontier. Our code is available at https://github.com/tomohiro-sawada/lora-capacity-utilization

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning

    Sep 5, 2026Huiyi Wang, Daijiao Liu, Lina Yao +1LLM Fine-TuningLow-Rank Adaptation

  2. Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting

    May 28, 2026Runze Xu, Arpit Garg, Hemanth Saratchandran +1Continual Learning for LLMsLLM Fine-Tuning

  3. Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature

    Sep 10, 2026Zihan Zhu, Zhehang Du, Xuyang Chen +5Low-Rank AdaptationParameter-Efficient Fine-Tuning