Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning: Controlled Empirical Evidence and Frozen-Backbone Representation Probes

Ashish Pandey

cs.LGMar 29, 2026

Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning: Controlled Empirical Evidence and Frozen-Backbone Representation Probes

Authors: Ashish Pandey

Abstract

Sequential fine-tuning of pretrained language encoders often overwrites previously acquired capabilities, but the forgetting behavior of parameter-efficient updates remains under-characterized. We present a controlled empirical study of Low-Rank Adaptation (LoRA) in sequential transformer encoder fine-tuning with companion representation probes that test a frozen-backbone explanation of its robustness. In five full-validation BERT-base reruns on an RTE->MRPC->CoLA->SST-2 sequence, full fine-tuning yields 19.9%+/-4.8% average forgetting, whereas standard LoRA (r=8, query/value modules) yields 0.6%+/-1.4% (paired t-test, p=0.002, Cohen's d_s=3.12). Task-level analyses confirm this reduction is not merely an aggregate effect. Secondary experiments on RoBERTa-base show the same pattern, and the strongest EWC baseline remains at 15.5%+/-1.4% forgetting. A six-task extension reveals that low average forgetting can hide strong task-level heterogeneity. Fine-grained freezing ablations show a marked forgetting drop once frozen parameters exceed roughly 95%, with classifier-only and shallow-adapter baselines approaching LoRA. Companion task-similarity probes in GPT-2 and RoBERTa show the same directional story: frozen-backbone regimes preserve higher inter-task similarity than full fine-tuning, gradual unfreezing weakens stability, and full fine-tuning exhibits its clearest divergence at the final transformer layer. These results support a restrained mechanistic interpretation: LoRA helps largely because backbone freezing preserves a more stable shared feature scaffold. We position standard LoRA as both a strong empirical baseline for sequential encoder adaptation and a useful probe of how selective plasticity shapes interference in transformer continual learning.

Figures & tables

Study	Role	Validation	Training caps	Seeds
BERT full FT vs. LoRA	Confirmatory	Full validation	SST-2 capped at 2,000	45–49
RoBERTa full FT vs. LoRA	Secondary	Full validation	SST-2 capped at 2,000	42–44
EWC baseline	Secondary	Full validation	SST-2 capped at 2,000	42–44
6-task LoRA extension	Secondary	Full validation	SST-2, QNLI capped at 2,000	42–44
Rank/module/ quant. sweeps	Exploratory	validation [:200]	Screening setup	42–44
Fine-grained freezing ablation	Exploratory	Full validation	SST-2 capped at 2,000	42–44

Table 1: Protocol taxonomy used in this paper. Only the confirmatory BERT row supplies the main headline result.

Figure 1: Paper overview. Left: full fine-tuning leaves the entire backbone trainable and suffers high forgetting under unconstrained drift. Middle: LoRA freezes the backbone and routes task-specific adaptation through small low-rank updates, preserving a more stable shared scaffold. Right: the empirical freezing ablation shows a sharp forgetting drop once the frozen fraction crosses roughly 95%.

Figure 2: Seed-paired confirmatory BERT forgetting comparison from the audited full-validation reruns (seeds 45–49).

Method	Mean	Std	Min	Max	Reduction
Full fine-tuning	19.9%	4.8%	12.1%	25.0%	—
LoRA ( $r\!=\!8$ , qv)	0.6%	1.4%	$-2.0\%$	2.1%	97.2%

Table 2: Confirmatory BERT-base forgetting statistics from the full-validation reruns (seeds 45–49). Earlier subset-validation runs are intentionally excluded.

Panel of Figure 3 — Figure 3: Task-level and stage-wise forgetting in the confirmatory BERT reruns (seeds 45–49). (a) LoRA reduces forgetting on every individual task. (b) The gap between methods widens progressively as more tasks are added.

Figure 4: Mean task-by-stage forgetting in the confirmatory BERT reruns. Each cell reports forgetting relative to that task’s post-training peak, averaged across seeds 45–49.

Method	Forgetting	Final Avg. Accuracy	Reduction
Full fine-tuning	24.2% $\pm$ 2.2%	64.8%	—
LoRA ( $r\!=\!8$ , qv)	1.7% $\pm$ 1.0%	59.3%	92.8%

Table 3: Secondary full-validation RoBERTa-base replication (

n=3

seeds).

Method	$n$	Forgetting	Final Avg. Accuracy
BERT LoRA (confirmatory)	5	0.6% $\pm$ 1.4%	59.5%
EWC ( $\lambda=100$ )	6	16.4% $\pm$ 1.2%	66.2%
EWC ( $\lambda=1000$ )	3	16.2% $\pm$ 1.2%	66.2%
EWC ( $\lambda=5000$ )	3	15.5% $\pm$ 1.4%	67.0%

Table 4: Secondary EWC baseline under the same task order and training setup. The BERT LoRA row is included only as confirmatory context, not as a paired same-seed comparison.

Figure 5: Secondary cross-checks. Left: RoBERTa full fine-tuning versus LoRA. Right: EWC forgetting by

\lambda

with confirmatory BERT reference lines.

Setting	$n$	Avg. Forgetting	Final Avg. Accuracy	MRPC Forgetting
4-task BERT LoRA reference	5	0.6% $\pm$ 1.4%	59.5%	$-0.3$ to $0.5$ pts
6-task BERT LoRA	3	5.6% $\pm$ 0.8%	53.8%	35.8–37.5 pts

Table 5: LoRA under 4-task and 6-task full-validation protocols. The 6-task result has low average forgetting but large MRPC forgetting.

Panel of Figure 6 — Figure 6: Six-task LoRA extension. (a) MRPC dominates task-level forgetting despite low average values. (b) Forgetting stays near zero through SST-2, then jumps sharply once QNLI is introduced.

Configuration	Frozen %	Avg. Forgetting	Trainable Params	Role
Freeze layers 0–9, train 10–11	86.5%	14.8%	14.8M	High-forgetting reference
Freeze layers 0–10, train 11	93.0%	13.2%	7.7M	High-forgetting reference
Freeze 0–10 + layer-11 attention	95.1%	5.2%	5.3M	Marked drop
Freeze all 12 layers	99.5%	3.1%	592K	Classifier-only baseline
LoRA on layer 11 only	100.0%	2.9%	26K	Adapter baseline
LoRA on layers 10–11	100.0%	2.9%	51K	Adapter baseline

Table 6: Exploratory full-validation freezing ablation (

n=3

seeds). The pattern suggests a marked drop in forgetting once freezing exceeds roughly 95%, but we do not interpret this as identifying a unique causal mechanism.

Figure 7: Fine-grained freezing ablation. Average forgetting falls sharply once freezing exceeds roughly 95% of parameters.

Figure 8: Companion mechanism probes from the verified second repository. Top left: direct task-similarity comparison between a frozen backbone and full fine-tuning in a four-task GPT-2 probe. Top right: task similarity degrades steadily as more layers are unfrozen. Bottom: layer-wise task similarity remains high in both regimes, but full fine-tuning shows its largest drop at the final transformer layer. These probes are exploratory and architecture-mismatched to the main BERT benchmark, so they strengthen interpretation rather than change the confirmatory headline result.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9: Exploratory configuration sweeps on validation[:200] .

Panel of Figure 10 — Figure 10: Exploratory quantization and baseline context from validation[:200] .

Figure 11: Companion robustness probes. Left: task similarity across RoBERTa, GPT-2 Medium, and eight-seed GPT-2. Right: orthogonal regularization effect. Bottom: forgetting across task orderings.

Catastrophic Forgetting

Catastrophic forgetting is the loss of previously learned behavior when a model adapts to new tasks or changing data.

See all 351 papers

On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning
Mar 10, 2026Muhammad Ahmad, Jingjing Zheng, Yankai CaoParameter-Efficient Fine-Tuning Catastrophic Forgetting
Department of Electrical and Computer Engineering, The University of British Columbia · Department of Mathematics, The University of British Columbia · Department of Chemical and Biological Engineering, The University of British Columbia
A Dynamical Theory of LoRA in Continual Learning
Sep 30, 2026Théo Marchetta, Filippo Alessandroni, Alessandro Breccia +2Continual Learning Low-Rank Adaptation
Department of Mathematics, Alma Mater Studiorum – Università di Bologna, Piazza di Porta San Donato 5, 40126 Bologna, Italy · Gatsby Computational Neuroscience Unit, University College London · Donders Centre for Neuroscience, Radboud University, Nijmegen, The Netherlands
CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMs
May 29, 2026Kiran Nayudu, Aswini Nutakki, Sai Vinay Naidu +1Continual Learning for LLMs Fine-Tuning
ModelBrew AI

Mar 10, 2026J/K move · Enter open · S save

On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning

Muhammad Ahmad, Jingjing Zheng, Yankai Cao

Department of Electrical and Computer Engineering, The University of British Columbia · Department of Mathematics, The University of British Columbia · Department of Chemical and Biological Engineering, The University of British Columbia

Parameter-Efficient Fine-Tuning Catastrophic Forgetting Low-Rank Adaptation Continual Learning for LLMs

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT) based on low-rank decomposition, such as LoRA, has become a standard for adapting large pretrained models. However, its behavior in sequential learning -- specifically regarding catastrophic forgetting -- remains insufficiently understood. In this work, we present an empirical study showing that forgetting is strongly influenced by the geometry and parameterization of the update subspace. While methods that restrict updates to small, shared matrix subspaces often suffer from task interference, tensor-based decompositions (e.g., LoRETTA) mitigate forgetting by capturing richer structural information within ultra-compact budgets, and structurally aligned parameterizations (e.g., WeGeFT) preserve pretrained representations. Our findings highlight update subspace design as a key factor in continual learning and offer practical guidance for selecting efficient adaptation strategies in sequential settings.

Abstract

Catastrophic Forgetting

Explore similar work

On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning

A Dynamical Theory of LoRA in Continual Learning

CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMs