Adapting LLMs to new domains causes forgetting because standard methods (e.g., full fine-tuning, LoRA) inject new directions into the weight space. We show that forgetting is governed by one algebraic property: whether the update preserves the column span of the pretrained weight matrix (Proposition 1). We propose GAIN, the simplest multiplicative alternative (W_new = S * W), which satisfies this by construction and can be absorbed into existing weights for zero inference cost. Across five models (774M to 70B) adapted sequentially over eight domains, GAIN improves earlier-domain perplexity by 7-13%, while LoRA degrades it by 18-36%. GAIN matches replay-augmented LoRA without storing prior data and dominates EWC on the forgetting-adaptation Pareto front. While LoRA can only reduce forgetting by sacrificing in-domain adaptation, GAIN achieves both with no domain boundaries and no regularization. The principle generalises: (IA)^3, an independent multiplicative method, also improves earlier domains.
Figures & tables
PPL Δ % ( ↓ better)
Benchmark Δ (accuracy)
Method
LR
Params
PubMed
WT-103
LAMB
ARC-E
HellaS
PIQA
WinoGr
GAIN
3×10−4
46K
−0.7
+0.0
−0.1
+0.0
+0.1
+0.1
−0.1
GAIN
10−3
46K
−2.0
+0.1
−0.4
−0.1
+0.1
+0.1
−0.1
GAIN
3×10−3
46K
−4.2
+0.3
−0.9
−0.3
+0.0
+0.0
−0.1
GAIN-FFN
10−3
230K
−5.1
+0.1
−2.0
−0.7
−0.1
+0.2
+0.2
GAIN-FFN
3×10−3
230K
−8.3
+1.0
−2.9
−1.1
−0.1
+0.1
+0.2
Table 1: GPT-2 (774M) adapted on PubMedQA. PPL columns: % change in perplexity ( ↓ = better). Benchmark columns: accuracy change on general tasks. PubMed is in-domain; all others measure forgetting.
GAIN (lr =10−3 , % PPL change vs. pretrained baseline)
After training
Med
WT
Fin
PG
MdQ
LB
SST
HS
+Med
−2.9
+0.0
−0.0
−1.3
−0.3
−0.6
−0.0
−0.2
+WT
−3.0
−11.0
−0.1
−3.8
−0.2
−1.5
−2.1
−0.3
+Fin
−2.9
−11.1
−5.6
−2.9
+0.0
−2.6
−1.6
−0.4
+PG
−2.8
−11.0
−5.7
−24.0
−0.1
−4.3
−1.7
−0.2
+MdQ
−2.4
−10.6
−5.4
−23.9
−8.0
−5.0
−1.4
−0.3
Table 2: Sequential adaptation on GPT-2 Large (774M, 200K tokens/domain). Entries = % PPL change vs. pretrained. Diagonal = in-domain; lower triangle = preservation of earlier domains; upper triangle = forward interference. See Tables 5 – 6 for other learning rates and Table 4 for Mistral-7B.
Figure 1: Per-token loss change on four unrelated domains after medical adaptation (GPT-2 Large). GAIN ( σ≈0.12 ) is 3× narrower than LoRA ( σ≈0.30 ).
Figure 2: LoRA’s forgetting-adaptation tradeoff. Red points are LoRA with different learning rates (lr) and L2 regularization strengths ( λ , where ℓ=ℓLM+λ∥p−pprev∥2 ). Higher λ reduces forgetting but sacrifices adaptation. GAIN-FFN (blue star) operates outside this tradeoff curve at lr ≤3×10−3 .
GPT-2 (774M)
Mistral-7B
Qwen2.5-7B
Llama-13B
Llama-70B
Base
GAIN
LoRA
Base
GAIN
LoRA
Base
GAIN
LoRA
Base
GAIN
LoRA
Base
GAIN
LoRA
ARC-E
46.6
−1.4
−1.9
79.5
+1.5
−10.7
77.4
+3.8
−0.2
76.4
+0.5
+1.5
79.5
+1.4
+0.2
ARC-C
25.1
−1.0
−0.1
54.3
+0.3
−10.0
51.2
+3.1
−0.1
49.0
+0.0
+2.5
57.0
+1.4
−1.0
BoolQ
60.5
−2.1
−7.8
84.3
−1.2
−9.2
84.3
−0.8
−14.9
82.5
−1.4
−2.3
85.3
−2.5
−14.6
HellaS
45.3
−0.9
−2.1
81.2
+0.1
−4.1
78.9
+0.4
−0.9
79.7
+0.9
−5.3
84.3
+0.7
−7.1
OBQA
31.2
−1.8
−2.2
46.2
+0.0
−3.0
47.4
−1.0
−2.8
45.6
+0.4
−3.2
49.0
−0.2
−10.6
Table 3: Benchmark accuracy change after seven sequential domain adaptations (acc_norm). Same hyperparameters across all models (lr =10−3 , 200 steps, 5 epochs per domain). Benchmarks are not in the training set.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
GAIN (lr =10−3 , % PPL change vs. pretrained baseline)
After training
WT
Med
Fin
PG
MdQ
LB
SST
HS
+WT
−23.1
+Med
−23.2
−1.2
+Fin
−23.1
−1.1
−5.0
+PG
−22.9
−1.1
−5.0
−0.3
+MdQ
−22.7
−1.1
−4.9
−0.2
−4.6
Appendix
Table 4: Sequential adaptation on Mistral-7B (200K tokens per domain). Same format as Table 2 .
GAIN (lr =3×10−3 , % PPL change vs. pretrained baseline)
After training
Med
WT
Fin
PG
MdQ
LB
SST
HS
+Med
−5.7
+0.2
+0.0
−3.5
−0.3
−1.3
+0.1
−0.2
+WT
−4.9
−15.8
+0.5
−6.4
+0.2
−1.6
−2.7
+0.2
+Fin
−3.9
−15.3
−8.6
−4.4
+0.7
−3.6
−1.0
+0.5
+PG
−3.5
−14.5
−8.2
−25.0
+0.5
−5.0
−1.0
+0.8
+MdQ
−2.1
−13.4
−7.2
−24.2
−14.2
−5.5
−0.2
+1.5
Appendix
Table 5: Sequential adaptation on GPT-2 Large at lr =3×10−3 (GAIN). Same format as Table 2 . Stronger adaptation with minor interference on later domains.
LoRA (lr =3×10−4 , % PPL change vs. pretrained baseline)
After training
Med
WT
Fin
PG
MdQ
LB
SST
HS
+Med
−5.6
+0.2
+0.1
−3.5
−0.3
−1.3
+0.2
−0.3
+WT
+3.8
−19.7
+2.4
−5.6
+4.7
+2.8
−2.5
+2.4
+Fin
+3.0
−13.9
−9.1
−12.6
+2.5
−4.9
+0.6
+1.1
+PG
+1.5
−11.7
−6.8
−25.5
+1.9
−5.4
−1.2
+1.4
+MdQ
+9.3
−5.1
−1.0
−21.9
−17.3
−3.8
+1.4
+2.7
Appendix
Table 6: Sequential adaptation: LoRA at safe lr =3×10−4 on GPT-2 Large. Same format as Table 2 . Even at the safe LR, forgetting accumulates to +10.2% .
Method
Targets
LR
Params
In-domain ↓
Final forgetting
GAIN ( WO )
—
10−3
46K
−10.2%
−9.8%
LoRA r=1
WO
10−3
92K
−21.3%
+12.2%
LoRA r=8
WO
10−3
737K
−24.3%
+11.7%
LoRA r=16
WO
10−3
1.47M
−24.5%
+13.3%
LoRA r=8
WO
3×10−4
737K
−21.5%
+14.0%
LoRA r=8
QKVO
10−3
2.21M
−23.8%
+21.6%
Appendix
Table 7: LoRA at standard ranks and QKVO target sets, on GPT-2 Large. Same protocol as Table 2 ( 5 epochs, 200 steps, 200 K tokens/domain, lr =10−3 unless noted). In-domain : mean PPL change on the diagonal ( ↓ = better adaptation). Final forgetting : mean PPL change on the seven earlier-trained domains after the last adaptation (positive = forgetting).
Figure 3: Loss landscape interpolation on Mistral-7B. Left: in-domain PPL decreases for both. Right: GAIN’s cross-domain loss is flat; LoRA’s rises steeply.
Figure 4: Cross-domain loss when WikiText and Medical adaptations are combined. Left : GAIN—uniform blue; any combination preserves LAMBADA. Right : LoRA—red regions appear when adaptations are combined.
PPL Δ %
Benchmark Δ (accuracy)
Model
Method
Params
PubMed
LAMB
ARC-E
HellaS
PIQA
WinoGr
Mistral-7B
GAIN
131K
−3.1
−0.4
−0.3
−0.2
−0.5
+0.0
LoRA (lr =10−3 )
262K
+5.4
+2.7
+0.6
+0.3
−0.3
−0.1
LoRA (lr =3×10−4 )
262K
−3.7
−0.0
+0.4
+0.1
−0.3
−0.1
Llama-2-13B
GAIN
205K
−2.4
−0.3
−0.1
+0.4
+0.0
+0.5
LoRA (lr =10−3 )
410K
+1.4
−0.2
−1.3
+0.9
+0.2
−0.5
Appendix
Table 8: Single-domain adaptation (PubMedQA) across model scales. PPL columns: % change ( ↓ = better). PubMed is in-domain; LAMBADA measures forgetting. Benchmarks: accuracy (4 tasks averaged).
GAIN (lr =10−2 )
LoRA (lr =3×10−3 )
Model
WikiText
LAMBADA
WikiText
LAMBADA
Mistral-7B
−15.9%
+0.2%
+213%
+180%
Llama-2-13B
−25.7%
−0.1%
−28.4%
+10.0%
Llama-2-70B
−18.0%
+0.1%
+12.3%
+35.6%
Appendix
Table 9: GAIN (lr =10−2 ) vs LoRA (lr =3×10−3 ) on WikiText adaptation. GAIN adapts strongly with zero forgetting at every scale. LoRA diverges on Mistral-7B and Llama-2-70B, and causes +10% forgetting on Llama-2-13B.
lr =10−3
lr =3×10−3
lr =10−2
After
smin
smax
% tight
smin
smax
% tight
smin
smax
% tight
+Dom 1
0.94
1.06
99.9
0.84
1.16
67.6
0.52
1.48
29.3
+Dom 2
0.78
1.25
73.3
0.40
1.62
38.1
−0.76
2.36
16.6
+Dom 4
0.61
1.40
55.8
0.07
1.94
27.8
−0.83
2.87
12.9
+Dom 6
0.34
1.78
42.7
−0.38
2.15
22.3
−1.25
3.06
11.5
+Dom 8
0.24
1.77
36.8
−0.51
2.28
19.3
−1.33
3.11
10.4
Appendix
Table 10: Scaling factor statistics after each sequential domain (GPT-2 Large). At lr =10−3 , factors stay positive and tightly concentrated. At lr =3×10−3 , factors go negative by domain 5. At lr =10−2 , factors go negative by domain 2 and reach −1.33 by domain 8. Negative scaling factors sign-flip hidden dimensions, destroying the multiplicative guarantee.
GAIN-FFN (2M tokens, lr =10−3 , % PPL change vs. pretrained)
After training
Med
WT
Fin
PG
MdQ
LB
SST
HS
+Med
−6.8
+0.5
−0.3
−3.1
−0.1
−2.7
+0.5
−0.4
+WT
−4.1
−17.6
+0.0
−2.8
+0.3
−2.5
−3.7
+0.7
+Fin
−2.9
−15.9
−11.7
−3.0
−0.7
−7.7
−1.0
+0.2
+PG
−3.1
−14.7
−10.9
−12.6
−1.0
−8.3
−1.5
+0.9
+MdQ
−0.7
−12.2
−9.7
−9.9
−23.1
−9.1
+1.4
+2.3
Appendix
Table 12: Sequential adaptation with 2M tokens per domain (GPT-2 Large). Same format as Table 2 . GAIN-FFN maintains positive transfer; LoRA’s forgetting worsens.
Method
Tokens
Params
In-domain ↓
Final forgetting
GAIN ( WO )
200 K
46K
−10.2%
−9.8%
GAIN ( WO )
5 M
46K
−9.3%
−8.5%
LoRA r=8 ( WO )
200 K
737K
−24.3%
+11.7%
LoRA r=8 ( WO )
5 M
737K
−23.6%
+15.7%
Appendix
Table 13: Sequential 8-domain adaptation at 5 M tokens/domain on GPT-2 Large, same hyperparameters as Table 7 ( 5 epochs, 200 steps, lr =10−3 ). Final forgetting : mean PPL change on the seven earlier-trained domains after the last adaptation.
Method
Params
In-domain ↓
Final forgetting
GAIN-FFN
590K
−22.5%
−20.6%
GAIN ( WO )
131K
−17.7%
−16.6%
LoRA r=1 ( WO )
262K
−23.7%
+11.6%
LoRA r=8 ( WO )
2.10M
−6.4%
+56.9%
Appendix
Table 14: Sequential 8-domain adaptation on Llama-3-8B-Instruct. Same protocol and metrics as Table 7 .
Method
Params
In-domain ↓
Final forgetting
Buffer
LoRA r=8 ( WO )
737K
−24.3%
+11.7%
—
Replay-LoRA r=8
737K
−22.4%
−13.7%
128 sequences
GAIN-FFN
230K
−17.3%
−13.4%
—
Replay-GAIN-FFN
230K
−16.1%
−15.1%
128 sequences
Appendix
Table 15: Replay (buffer of 16 sequences per prior domain; every 4th step samples from the buffer) added to LoRA and to GAIN-FFN on GPT-2 Large, same protocol as Table 7 . Buffer lists the extra state replay requires beyond model parameters; a ‘—’ indicates no auxiliary state.
Method
Params
In-domain ↓
Final forgetting
LoRA r=8 ( WO , vanilla)
737K
−24.3%
+11.7%
EWC-LoRA λ=103
737K
−24.1%
+3.3%
EWC-LoRA λ=104
737K
−21.5%
−5.3%
EWC-LoRA λ=105
737K
−14.7%
−11.7%
GAIN-FFN
230K
−17.3%
−13.4%
Appendix
Table 16: EWC-LoRA λ sweep on GPT-2 Large, same protocol as Table 7 . Diagonal Fisher estimated from 64 training sequences after each domain.
Step
Most recent domain
smin
smax
smean
% tight
1
wikitext
0.832
1.178
0.999
73.2
4
pg19
0.588
1.418
0.998
53.4
8
hellaswag
0.480
1.566
0.996
37.7
12
pg19
0.480
1.566
0.994
31.4
16
hellaswag
0.434
1.566
0.993
26.0
20
pg19
0.434
1.566
0.992
23.8
Appendix
Table 17: Scaling-factor distribution for GAIN-FFN through 24 sequential adaptations on GPT-2 Large (lr =10−3 , 5 epochs, 200 steps per adaptation). Domains cycle through the eight-domain order three times. % tight = fraction of scaling factors in [0.95,1.05] . Selected steps shown for compactness; full trajectory in experiments_apr17_round3/exp8/ .
Method
Params
In-domain ↓
Final forgetting
LoRA r=8 ( WO , vanilla)
737K
−24.3%
+11.7%
PackNet-LoRA ( r=1 /domain, 8 domains)
737K
— (reported below)
+4.1%
GAIN-FFN
230K
−17.3%
−13.4%
Appendix
Table 18: PackNet-LoRA on GPT-2 Large, same protocol as Table 7 . Per-domain LoRA rank r=1 ; 8 adapters stacked over the sequence.
Method
In-dom WT
In-dom DE
Forward interference on DE
WT preserved after DE
GAIN-FFN
−21.0%
−7.4%
+0.4%
−19.8%
LoRA r=8
−32.9%
−10.5%
+18.1%
−12.6%
Appendix
Table 19: GPT-2 Large adaptation on a 2-domain OOD protocol: WikiText then German Wikipedia. Forward interference on German is the German-Wikipedia PPL change after training only on WikiText—measuring how much cross-domain spillover each method introduces.
Top-10 right-singular subspace overlap
Method
layer 0
layer 9
layer 18
layer 27
layer 35
mean
GAIN-FFN (multiplicative)
0.99
0.89
0.91
0.82
0.96
0.91
GAIN ( WO ) (multiplicative)
0.98
0.86
0.86
0.76
0.94
0.88
LoRA r=8 (additive)
0.60
0.50
0.38
0.44
0.73
0.53
DoRA r=8 (additive)
0.63
0.44
0.45
0.52
0.78
0.56
Relative Frobenius perturbation ∥Wnew−W∥F/∥W∥F
Appendix
Table 20: Singular-subspace preservation after the eight-domain sequential adaptation protocol on GPT-2 Large. Multiplicative methods (GAIN, GAIN-FFN) preserve ∼88 – 91% of the pretrained top- 10 right-singular subspace; additive methods (LoRA, DoRA) preserve only ∼53 – 56% , meaning roughly half of their top singular directions are “intruder” dimensions absent from W . Magnitude renormalisation (DoRA) does not change the subspace shift relative to vanilla LoRA.
Method
Params
In-domain ↓
Final forgetting
LoRA r=8 ( WO )
737K
−24.3%
+11.7%
DoRA r=8 ( WO )
783K
−24.2%
+14.2%
GAIN-FFN
230K
−17.3%
−13.4%
Appendix
Table 21: DoRA vs LoRA on GPT-2 Large, same protocol as Table 7 .
Method
Params
In-domain ↓
Final forgetting
LoRA r=8 ( WO )
737K
−25.8±0.05%
−2.0±1.0%
GAIN-FFN
230K
−19.2±0.05%
−14.7±0.05%
Appendix
Table 22: 19-distinct-domain protocol on GPT-2 Large (3 seeds each, lr =10−3 , 5 epochs, 200 steps per domain). Mean ± std across seeds. Final forgetting : mean PPL change on the eighteen earlier-trained domains after the last adaptation.
(IA) 3 (lr =10−3 , % PPL change vs. pretrained baseline)
After training
Med
WT
Fin
PG
MdQ
LB
SST
HS
+Med
−5.7
+0.1
+0.2
−3.3
−0.4
−1.4
+0.1
−0.4
+WT
−4.9
−17.7
+0.6
−5.0
−0.0
−1.7
−3.2
+0.2
+Fin
−4.1
−17.3
−9.9
−6.9
−0.1
−4.9
−1.7
+0.2
+PG
−3.8
−16.6
−9.6
−25.6
−0.4
−6.1
−2.0
+0.3
+MdQ
−2.6
−15.5
−8.7
−24.7
−17.5
−7.1
−1.2
+0.9
Appendix
Table 23: (IA) 3 sequential adaptation on GPT-2 Large (lr =10−3 , 200K tokens/domain). (IA) 3 shows positive transfer, confirming that the multiplicative principle—not GAIN’s specific parameterization—prevents forgetting.
Low-Rank Adaptation (LoRA) has become one of the most widely used fine-tuning mechanisms for adapting large language models to new domains, tasks, and users. Yet adaptation performance alone can obscure an important failure mode: LoRA updates may improve performance on the target distribution while degrading prior capabilities learned during pretraining and alignment. We show that this forgetting becomes especially severe when the adaptation distribution differs substantially from the models original training or alignment distributions. The challenge is amplified in practical settings, where the original training and alignment data are typically unavailable. Motivated by this constraint, we study how LoRA based adaptation balances new learning against forgetting in a replay-free setting, and introduce a simple output space regularizer that can be added directly to existing training pipelines. Our method removes the ground-truth token from both the base and adapted model distributions, renormalizes the remaining probabilities, and applies KL regularization only over the non-target vocabulary. This preserves the base models relative preferences among alternative tokens without directly opposing the cross-entropy signal required for adaptation. As the regularizer acts only at the loss level, it requires no replay data, architectural changes, adapter redesign, or inference-time overhead, and can be applied directly to existing LoRA variants. Across all LoRA variants tested and across various backbones, our method improves the frontier between new learning and forgetting when the adaptation distribution differs substantially from the base models original training or alignment distributions, suggesting a broadly applicable route toward more reliable LLM updating.
Runze Xu, Arpit Garg, Hemanth Saratchandran +1
Australian Institute for Machine Learning Adelaide University Adelaide SA 5000
Large language models deployed in specialized domains must improve in-domain performance without sacrificing general capabilities. Existing parameter-efficient fine-tuning methods are typically always on: their learned perturbations are applied to every input, which can degrade out-of-domain (OOD) performance. We propose Engram Adapter, a framework that repurposes pretraining-time conditional memory as a post-hoc adapter for frozen LLMs. It uses multi-channel matching over local n-gram patterns with explicit occupancy tracking as a lightweight selectivity prior, making residual injection more likely on in-domain inputs while a learned scalar gate suppresses incoherent OOD retrievals. We evaluate on Qwen3-4B and Qwen3-8B with AG-News and MedMCQA as adaptation tasks and OOD benchmarks spanning reasoning, translation, code generation, and legal reasoning. Engram Adapter improves in-domain accuracy while preserving 99.4%--100.1% of average OOD performance; on LegalBench it slightly exceeds the frozen base model on average, whereas comparable always-on baselines degrade sharply. Mechanistic analyses show that although OOD activations are non-zero, gate and projection attenuation reduce residuals to approximately 0.08% of hidden-state norm, yielding small KL drift and negligible accuracy change. These results suggest conditional activation is a promising route toward modular, retention-preserving domain specialization over frozen backbones.
Jiayu Hou, Lei Wang
University of Electronic Science and Technology of China, Chengdu, China
Instruction-tuned models are deployed into environments where domains are heterogeneous and evolve, yet adding new domains or data typically requires costly retraining. We present AutoAdapt, a modular framework that incorporates new domains and data via targeted single-adapter training without modifying other adapters. The framework automatically discovers latent domains, uses them to train per-domain Low-Rank Adaptation (LoRA) adapters independently in parallel and performs parameter-free routing. Across 14 domain-specific benchmarks and GPT-4o pairwise judgements, AutoAdapt achieves parity with a LoRA adapter trained on all domains without requiring full-model retraining. We also find evidence of specialisation effect convergence across independent discovery methods. Overall, training each adapter on its own domain prevents domain interference by construction, thus enabling modular, taxonomy-free domain specialisation without aggregate performance loss or full model retraining.