cs.LGMar 10, 2026

On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning

Authors: Muhammad Ahmad, Jingjing Zheng, Yankai Cao

Organizations: Department of Electrical and Computer Engineering, The University of British Columbia · Department of Mathematics, The University of British Columbia · Department of Chemical and Biological Engineering, The University of British Columbia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Parameter-efficient fine-tuning (PEFT) based on low-rank decomposition, such as LoRA, has become a standard for adapting large pretrained models. However, its behavior in sequential learning -- specifically regarding catastrophic forgetting -- remains insufficiently understood. In this work, we present an empirical study showing that forgetting is strongly influenced by the geometry and parameterization of the update subspace. While methods that restrict updates to small, shared matrix subspaces often suffer from task interference, tensor-based decompositions (e.g., LoRETTA) mitigate forgetting by capturing richer structural information within ultra-compact budgets, and structurally aligned parameterizations (e.g., WeGeFT) preserve pretrained representations. Our findings highlight update subspace design as a key factor in continual learning and offer practical guidance for selecting efficient adaptation strategies in sequential settings.

Figures & tables

Table 1: Comparison of performance and parameter efficiency across different fine-tuning methods on ViT-Base and ViT-Large. The configurations for each method are: LoRA ( r=8r{=}8 ), PiSSA ( r=8r{=}8 ), LoRETTA ( R=5R{=}5 ), and WeGeFT ( r=16r{=}16 ).
Figure 1: Performance comparison of different fine-tuning methods on ViT-Base under varying update space dimensions (rank).
Figure 2: Performance comparison of different fine-tuning methods on ViT-Large under varying update space dimensions (rank).
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

HyperparameterFFLoRAPiSSALoRETTAWeGeFT
OptimizerAdam
Number of epochs per task10
Batch size15
Learning rate0.00010.0010.00010.010.001
Appendix
Table 2: Hyperparameters used for full fine-tuning (FF) and parameter-efficient methods.
More in this topic

Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning adapts pretrained models by updating small parameter subsets or lightweight modules instead of all weights.

See all 609 papers

Explore similar work

CardsList

  1. Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

    Jan 31, 2026Hao Gu, Mao-Lin Luo, Zi-Hao Zhou +3VLM AdaptationContinual Learning

    School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China

  2. Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning: Controlled Empirical Evidence and Frozen-Backbone Representation Probes

    Mar 29, 2026Ashish PandeyCatastrophic ForgettingLow-Rank Adaptation

  3. Are Parameter-Efficient Fine-tuning Methods Really Different?

    Oct 6, 2026Yikuan Li, Pinyan Lu, Fanghui LiuParameter-Efficient Fine-TuningLow-Rank Adaptation

    Department of Statistics, The University of Warwick, UK. · Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics, China. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.