cs.LGMar 29, 2026

Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning: Controlled Empirical Evidence and Frozen-Backbone Representation Probes

Authors: Ashish Pandey

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Sequential fine-tuning of pretrained language encoders often overwrites previously acquired capabilities, but the forgetting behavior of parameter-efficient updates remains under-characterized. We present a controlled empirical study of Low-Rank Adaptation (LoRA) in sequential transformer encoder fine-tuning with companion representation probes that test a frozen-backbone explanation of its robustness. In five full-validation BERT-base reruns on an RTE->MRPC->CoLA->SST-2 sequence, full fine-tuning yields 19.9%+/-4.8% average forgetting, whereas standard LoRA (r=8, query/value modules) yields 0.6%+/-1.4% (paired t-test, p=0.002, Cohen's d_s=3.12). Task-level analyses confirm this reduction is not merely an aggregate effect. Secondary experiments on RoBERTa-base show the same pattern, and the strongest EWC baseline remains at 15.5%+/-1.4% forgetting. A six-task extension reveals that low average forgetting can hide strong task-level heterogeneity. Fine-grained freezing ablations show a marked forgetting drop once frozen parameters exceed roughly 95%, with classifier-only and shallow-adapter baselines approaching LoRA. Companion task-similarity probes in GPT-2 and RoBERTa show the same directional story: frozen-backbone regimes preserve higher inter-task similarity than full fine-tuning, gradual unfreezing weakens stability, and full fine-tuning exhibits its clearest divergence at the final transformer layer. These results support a restrained mechanistic interpretation: LoRA helps largely because backbone freezing preserves a more stable shared feature scaffold. We position standard LoRA as both a strong empirical baseline for sequential encoder adaptation and a useful probe of how selective plasticity shapes interference in transformer continual learning.

Figures & tables

StudyRoleValidationTraining capsSeeds
BERT full FT vs. LoRAConfirmatoryFull validationSST-2 capped at 2,00045–49
RoBERTa full FT vs. LoRASecondaryFull validationSST-2 capped at 2,00042–44
EWC baselineSecondaryFull validationSST-2 capped at 2,00042–44
6-task LoRA extensionSecondaryFull validationSST-2, QNLI capped at 2,00042–44
Rank/module/ quant. sweepsExploratoryvalidation [:200]Screening setup42–44
Fine-grained freezing ablationExploratoryFull validationSST-2 capped at 2,00042–44
Table 1: Protocol taxonomy used in this paper. Only the confirmatory BERT row supplies the main headline result.
Figure 1: Paper overview. Left: full fine-tuning leaves the entire backbone trainable and suffers high forgetting under unconstrained drift. Middle: LoRA freezes the backbone and routes task-specific adaptation through small low-rank updates, preserving a more stable shared scaffold. Right: the empirical freezing ablation shows a sharp forgetting drop once the frozen fraction crosses roughly 95%.
Figure 2: Seed-paired confirmatory BERT forgetting comparison from the audited full-validation reruns (seeds 45–49).
MethodMeanStdMinMaxReduction
Full fine-tuning19.9%4.8%12.1%25.0%—
LoRA ( r ⁣= ⁣8r\!=\!8 , qv)0.6%1.4%−2.0%-2.0\%2.1%97.2%
Table 2: Confirmatory BERT-base forgetting statistics from the full-validation reruns (seeds 45–49). Earlier subset-validation runs are intentionally excluded.
Figure 3: Task-level and stage-wise forgetting in the confirmatory BERT reruns (seeds 45–49). (a) LoRA reduces forgetting on every individual task. (b) The gap between methods widens progressively as more tasks are added.
Figure 4: Mean task-by-stage forgetting in the confirmatory BERT reruns. Each cell reports forgetting relative to that task’s post-training peak, averaged across seeds 45–49.
MethodForgettingFinal Avg. AccuracyReduction
Full fine-tuning24.2% ±\pm 2.2%64.8%—
LoRA ( r ⁣= ⁣8r\!=\!8 , qv)1.7% ±\pm 1.0%59.3%92.8%
Table 3: Secondary full-validation RoBERTa-base replication ( n=3n=3 seeds).
MethodnnForgettingFinal Avg. Accuracy
BERT LoRA (confirmatory)50.6% ±\pm 1.4%59.5%
EWC ( λ=100\lambda=100 )616.4% ±\pm 1.2%66.2%
EWC ( λ=1000\lambda=1000 )316.2% ±\pm 1.2%66.2%
EWC ( λ=5000\lambda=5000 )315.5% ±\pm 1.4%67.0%
Table 4: Secondary EWC baseline under the same task order and training setup. The BERT LoRA row is included only as confirmatory context, not as a paired same-seed comparison.
Figure 5: Secondary cross-checks. Left: RoBERTa full fine-tuning versus LoRA. Right: EWC forgetting by λ\lambda with confirmatory BERT reference lines.
SettingnnAvg. ForgettingFinal Avg. AccuracyMRPC Forgetting
4-task BERT LoRA reference50.6% ±\pm 1.4%59.5%−0.3-0.3 to 0.50.5 pts
6-task BERT LoRA35.6% ±\pm 0.8%53.8%35.8–37.5 pts
Table 5: LoRA under 4-task and 6-task full-validation protocols. The 6-task result has low average forgetting but large MRPC forgetting.
Figure 6: Six-task LoRA extension. (a) MRPC dominates task-level forgetting despite low average values. (b) Forgetting stays near zero through SST-2, then jumps sharply once QNLI is introduced.
ConfigurationFrozen %Avg. ForgettingTrainable ParamsRole
Freeze layers 0–9, train 10–1186.5%14.8%14.8MHigh-forgetting reference
Freeze layers 0–10, train 1193.0%13.2%7.7MHigh-forgetting reference
Freeze 0–10 + layer-11 attention95.1%5.2%5.3MMarked drop
Freeze all 12 layers99.5%3.1%592KClassifier-only baseline
LoRA on layer 11 only100.0%2.9%26KAdapter baseline
LoRA on layers 10–11100.0%2.9%51KAdapter baseline
Table 6: Exploratory full-validation freezing ablation ( n=3n=3 seeds). The pattern suggests a marked drop in forgetting once freezing exceeds roughly 95%, but we do not interpret this as identifying a unique causal mechanism.
Figure 7: Fine-grained freezing ablation. Average forgetting falls sharply once freezing exceeds roughly 95% of parameters.
Figure 8: Companion mechanism probes from the verified second repository. Top left: direct task-similarity comparison between a frozen backbone and full fine-tuning in a four-task GPT-2 probe. Top right: task similarity degrades steadily as more layers are unfrozen. Bottom: layer-wise task similarity remains high in both regimes, but full fine-tuning shows its largest drop at the final transformer layer. These probes are exploratory and architecture-mismatched to the main BERT benchmark, so they strengthen interpretation rather than change the confirmatory headline result.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Exploratory configuration sweeps on validation[:200] .
Figure 10: Exploratory quantization and baseline context from validation[:200] .
Figure 11: Companion robustness probes. Left: task similarity across RoBERTa, GPT-2 Medium, and eight-seed GPT-2. Right: orthogonal regularization effect. Bottom: forgetting across task orderings.
More in this topic

Catastrophic Forgetting

Catastrophic forgetting is the loss of previously learned behavior when a model adapts to new tasks or changing data.

See all 351 papers

Explore similar work

CardsList
Mar 10, 2026

On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning (PEFT) based on low-rank decomposition, such as LoRA, has become a standard for adapting large pretrained models. However, its behavior in sequential learning -- specifically regarding catastrophic forgetting -- remains insufficiently understood. In this work, we present an empirical study showing that forgetting is strongly influenced by the geometry and parameterization of the update subspace. While methods that restrict updates to small, shared matrix subspaces often suffer from task interference, tensor-based decompositions (e.g., LoRETTA) mitigate forgetting by capturing richer structural information within ultra-compact budgets, and structurally aligned parameterizations (e.g., WeGeFT) preserve pretrained representations. Our findings highlight update subspace design as a key factor in continual learning and offer practical guidance for selecting efficient adaptation strategies in sequential settings.

Muhammad Ahmad, Jingjing Zheng, Yankai Cao

Department of Electrical and Computer Engineering, The University of British Columbia · Department of Mathematics, The University of British Columbia · Department of Chemical and Biological Engineering, The University of British Columbia

Parameter-Efficient Fine-TuningCatastrophic ForgettingLow-Rank AdaptationContinual Learning for LLMs
AskPDF
Sep 30, 2026

A Dynamical Theory of LoRA in Continual Learning

Despite the widespread use of Low-Rank Adaptation (LoRA), little is known about its dynamics in continual learning and the mechanisms by which low-rank updates affect catastrophic forgetting. We provide an asymptotically exact dynamical characterization of LoRA in a solvable two-task teacher-student model. In the high-dimensional online-learning limit, we derive a closed system of ordinary differential equations for a finite set of macroscopic order parameters, yielding exact expressions for the generalization errors throughout both the initial Task 1 learning phase and the subsequent LoRA fine-tuning on Task 2. The theory quantitatively matches finite-dimensional simulations and exposes two characteristic effects of LoRA: low-rank adaptation reduces interference with features learned on the first task, but its initialization slows adaptation to the second task. Building on this mechanistic picture, we analyze a state-dependent masking strategy that freezes hidden units carrying the strongest first-task representations and restricts adaptation to the complementary subspace. This structural partitioning markedly reduces forgetting, while preserving plasticity on the new task. Our framework further clarifies the role of adapter rank: transfer improves only up to the intrinsic dimensionality of the target task and saturates beyond it, while forgetting continues to grow with rank. These results provide a dynamical and geometric account of how low-rank adaptation organizes information across sequential tasks and are qualitatively reproduced on a sequential MNIST benchmark.

Théo Marchetta, Filippo Alessandroni, Alessandro Breccia +2

Department of Mathematics, Alma Mater Studiorum – Università di Bologna, Piazza di Porta San Donato 5, 40126 Bologna, Italy · Gatsby Computational Neuroscience Unit, University College London · Donders Centre for Neuroscience, Radboud University, Nijmegen, The Netherlands

Continual LearningLow-Rank AdaptationOnline Continual LearningNeural Network Training DynamicsCatastrophic ForgettingContinual LoRA
AskPDF
May 29, 2026

CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMs

Sequential fine-tuning of large language models forces a choice: let the shared substrate keep learning and accept catastrophic forgetting, or freeze it after task one and foreclose cross-task refinement. Per-task adapter methods (LoRAHub, AdapterFusion, PackNet, Progressive Networks) take the second path. We introduce CRMA (Constrained Residual Mixing Adapter), a residual adapter whose internal mixing matrix M is doubly-stochastic at every forward pass via Sinkhorn normalization, so by Birkhoff's theorem ||M||_2 <= 1 holds by construction -- a structural bound, not a penalty. CRMA's spectrally bounded backbone provides a continuously trained shared substrate that earlier modular methods could not, while preserving their forgetting guarantees. On Mistral-7B across 5 sequential domains and 3 seeds, modular per-task LoRA on a CRMA backbone reduces loss-relative drift from +42.96% +/- 5.5 (naive sequential fine-tuning) to -0.17% +/- 0.17, with disjoint per-seed ranges, and improves prior-task holdout loss by 1.99% +/- 0.54 over a matched frozen-substrate baseline. Three independent experimental setups (Mistral-7B 4-domain controlled ablation, TinyLlama 3-domain contamination-controlled replication, Mistral-7B cross-domain probes at 7B) all show positive backward transfer -- without replay buffers, without growing per-task memory, and without distillation. An inference-time ablation on Gemma-2-9B confirms CRMA mediates access to sequentially trained knowledge: 98/100 vs. 38/100 on the same weights and same questions with only CRMA injection toggled. 867 logged training steps verify ||M||_2 = 1.0 within float32 precision (max deviation 1.2 x 10^-7). The forgetting-prevention effect holds across 1.1B-9.2B parameters and four architecture families.

Kiran Nayudu, Aswini Nutakki, Sai Vinay Naidu +1

ModelBrew AI

Continual Learning for LLMsFine-TuningAdapter TuningParameter-Efficient Fine-TuningCatastrophic Forgetting
AskPDF