Continual LoRA

LoRA: Low-Rank Adaptation

Follow this topic Ask about this topic

Latest papers

21 papers5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Making LoRA Identifiable: Orthogonal Alignment for Continual Learning
Oct 12, 2026Pratik Rakesh Singh, Mohammadi Zaki, Akash Saha +2Orthogonal Procrustes Alignment Continual LoRA
Sony Research India
PaLoRA: Paced Low-Rank Adaptation for Continual Learning
Oct 3, 2026Yuxuan Li, Fanhu Zeng, Hao TangContinual LoRA Continual Learning
School of Computer Science Peking University · Institute of Automation Chinese Academy of Sciences
MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning
Oct 1, 2026Junkang LiuContinual Learning Low-Rank Adaptation
Tianjin University
A Dynamical Theory of LoRA in Continual Learning
Sep 30, 2026Théo Marchetta, Filippo Alessandroni, Alessandro Breccia +2Continual Learning Low-Rank Adaptation
Department of Mathematics, Alma Mater Studiorum – Università di Bologna, Piazza di Porta San Donato 5, 40126 Bologna, Italy · Gatsby Computational Neuroscience Unit, University College London · Donders Centre for Neuroscience, Radboud University, Nijmegen, The Netherlands
Muon Can Outperform Dedicated Continual Learning Methods
Sep 21, 2026Sebastian George Sincari, Bogdan Alexandru Gheorghe, Antonio BarbalauContinual Learning Muon Optimizer
Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania · Bitdefender, Bucharest, Romania
Bilinear Optimization Divergence: Diagnosing Factor-Constrained LoRA Continual Learning
Sep 20, 2026YongShun Wang, JianLin Su, Yong MaContinual Learning for LLMs Low-Rank Adaptation
Behavior Quotient Learning for Low-Rank Adaptation of LLM Agents
Sep 11, 2026Pengyang Zhou, Xiaobin Tu, Zhengxi Liu +7Low-Rank Adaptation Continual LoRA
Alibaba Cloud
Geo-LoRA: Geometry-Aware Subspace Evolution for Low-Rank Adaptation in Continual Learning
Aug 27, 2026Yibo FengContinual Learning Continual LoRA
University of Electronic Science and Technology of China Chengdu, China
Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Jul 26, 2026Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli +2Continual Learning for LLMs Continual LoRA
College of Health Solutions, Arizona State University, Phoenix, Arizona, USA · Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, Texas, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA +1
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
Jul 4, 2026Wentao LuContinual Learning for LLMs Low-Rank Adaptation
When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
Jun 26, 2026Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2Continual Learning Low-Rank Adaptation
Department of Computer Science, ETH Zurich, Zurich, Switzerland.
TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning
Jun 4, 2026Marius Dragoi, Ioana Pintilie, Alexandra Dragomir +2Continual Learning Low-Rank Adaptation
Bitdefender, Romania
Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learning
May 27, 2026Cheng Chen, Pengpeng Zeng, Yuyu Guo +3Stability-Plasticity Dilemma Continual Learning
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China · School of Computer Science and Technology, Tongji University, Shanghai, China · Independent Researcher +1
Energy-Structured Low-Rank Adaptation for Continual Learning
May 26, 2026Longhua Li, Lei Qi, Qi Tian +1Continual Learning Low-Rank Adaptation
School of Computer Science and Engineering, Southeast University, Nanjing, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Huawei Technologies, Shenzhen, China
LoRA vs. Full Fine-Tuning: A Theoretical Perspective
May 18, 2026Ali Zindari, Rotem Mulayoff, Sebastian U. StichFine-Tuning Low-Rank Adaptation
Universität des Saarlandes · CISPA Helmholtz Center for Information Security
Continual Fine-Tuning of Large Language Models via Program Memory
May 13, 2026Hung Le, Svetha VenkateshContinual Learning for LLMs Memory-Augmented Language Models
Deakin Applied AI Initiative, Deakin University, Australia
Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning
May 12, 2026Joana Pasquali, Ramiro N. Barros, Arthur S. Bianchessi +7Continual Learning for LLMs Gradient Interference
MALTA, Machine Learning Theory and Applications Lab, PUCRS, Porto Alegre, Brazil · Kunumi Institute, Brazil
SR $^2$ -LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning
May 8, 2026Fengqiang Wan, Yipeng Lin, Kan Lv +1Representation Drift Low-Rank Adaptation
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models
Apr 17, 2026Alexandra Dragomir, Ioana Pintilie, Antonio Barbalau +6Continual Learning for LLMs Low-Rank Adaptation
Bitdefender · ETH Zurich · University of Bucharest
Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation
Jan 31, 2026Hao Gu, Mao-Lin Luo, Zi-Hao Zhou +3VLM Adaptation Continual Learning
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Feb 25, 2025Xin Zhang, Liang Bai, Xian YangContinual LoRA Low-Rank Adaptation
Institute of Intelligent Information Processing, Shanxi University, Taiyuan, 030006, China · Alliance Manchester Business School, The University of Manchester, Manchester, M13 9PL, UK

Oct 12, 2026J/K move · Enter open · S save

Making LoRA Identifiable: Orthogonal Alignment for Continual Learning

Pratik Rakesh Singh, Mohammadi Zaki, Akash Saha, Aneesh Mukkamala+1

Sony Research India

Orthogonal Procrustes Alignment Continual LoRA Continual Learning for LLMs Low-Rank Adaptation Parameter-Efficient Fine-Tuning Orthogonal Representation Learning

Open paper AskPDF arXiv

There is growing interest in applying Parameter-Efficient Fine-Tuning (PEFT) to Continual Learning, as it enables faster training, mitigates catastrophic forgetting, and facilitates better adaptation to new tasks. Prior approaches have explored a range of strategies, including extending new LoRA branches for each incoming task, training under orthogonal loss objectives across all task-specific LoRA modules, and employing gating mechanisms to integrate new and existing LoRA modules. However, these methods incur memory costs that grow linearly with the number of tasks, limiting their scalability. More recent approaches investigate training a single LoRA by leveraging asymmetries in its parameterization; however, they often suffer from insufficient representational alignment across the LoRA parameter space. To address these limitations, we propose Stiefel Optimization and Aligned Rotation (SOAR), a two-stage framework for continual LoRA learning. First, we constrain the A matrix to lie on the Stiefel manifold, thereby reducing coordinate misalignment to the space of orthogonal rotation matrices. Second, we introduce an Orthogonal Procrustes alignment procedure for the B matrix to estimate the optimal rotation, coupled with a per-rank retention merging strategy that quantifies directional agreement between representations of new and previously learned tasks. We conduct extensive experiments across multiple models and benchmarks, demonstrating that SOAR achieves strong continual learning performance without incurring additional memory overhead.