GAIN: Multiplicative Modulation for Domain Adaptation

Hengshuai Yao; Xing Chen; Ahmed Murtadha; Guan Wang

cs.LGApr 6, 2026

GAIN: Multiplicative Modulation for Domain Adaptation

Authors: Hengshuai Yao, Xing Chen, Ahmed Murtadha, Guan Wang

Organizations: Sapient Intelligence · Department of Computing Science, University of Alberta

Abstract

Adapting LLMs to new domains causes forgetting because standard methods (e.g., full fine-tuning, LoRA) inject new directions into the weight space. We show that forgetting is governed by one algebraic property: whether the update preserves the column span of the pretrained weight matrix (Proposition 1). We propose GAIN, the simplest multiplicative alternative (W_new = S * W), which satisfies this by construction and can be absorbed into existing weights for zero inference cost. Across five models (774M to 70B) adapted sequentially over eight domains, GAIN improves earlier-domain perplexity by 7-13%, while LoRA degrades it by 18-36%. GAIN matches replay-augmented LoRA without storing prior data and dominates EWC on the forgetting-adaptation Pareto front. While LoRA can only reduce forgetting by sacrificing in-domain adaptation, GAIN achieves both with no domain boundaries and no regularization. The principle generalises: (IA)^3, an independent multiplicative method, also improves earlier domains.

Figures & tables

			PPL $\Delta$ % ( $\downarrow$ better)			Benchmark $\Delta$ (accuracy)
Method	LR	Params	PubMed	WT-103	LAMB	ARC-E	HellaS	PIQA	WinoGr
GAIN	$3\!\times\!10^{-4}$	46K	$-0.7$	$+0.0$	$-0.1$	$+0.0$	$+0.1$	$+0.1$	$-0.1$
GAIN	$10^{-3}$	46K	$-2.0$	$+0.1$	$-0.4$	$-0.1$	$+0.1$	$+0.1$	$-0.1$
GAIN	$\mathbf{3\!\times\!10^{-3}}$	46K	$\mathbf{-4.2}$	$\mathbf{+0.3}$	$\mathbf{-0.9}$	$-0.3$	$+0.0$	$+0.0$	$-0.1$
GAIN-FFN	$10^{-3}$	230K	$-5.1$	$+0.1$	$-2.0$	$-0.7$	$-0.1$	$+0.2$	$+0.2$
GAIN-FFN	$3\!\times\!10^{-3}$	230K	$-8.3$	$+1.0$	$-2.9$	$-1.1$	$-0.1$	$+0.1$	$+0.2$

Table 1: GPT-2 (774M) adapted on PubMedQA. PPL columns: % change in perplexity (

\downarrow

= better). Benchmark columns: accuracy change on general tasks. PubMed is in-domain; all others measure forgetting.

GAIN (lr $=10^{-3}$ , % PPL change vs. pretrained baseline)
After training	Med	WT	Fin	PG	MdQ	LB	SST	HS
+Med	$-2.9$	$+0.0$	$-0.0$	$-1.3$	$-0.3$	$-0.6$	$-0.0$	$-0.2$
+WT	$-3.0$	$-11.0$	$-0.1$	$-3.8$	$-0.2$	$-1.5$	$-2.1$	$-0.3$
+Fin	$-2.9$	$-11.1$	$-5.6$	$-2.9$	$+0.0$	$-2.6$	$-1.6$	$-0.4$
+PG	$-2.8$	$-11.0$	$-5.7$	$-24.0$	$-0.1$	$-4.3$	$-1.7$	$-0.2$
+MdQ	$-2.4$	$-10.6$	$-5.4$	$-23.9$	$-8.0$	$-5.0$	$-1.4$	$-0.3$

Table 2: Sequential adaptation on GPT-2 Large (774M, 200K tokens/domain). Entries = % PPL change vs. pretrained. Diagonal = in-domain; lower triangle = preservation of earlier domains; upper triangle = forward interference. See Tables 5 – 6 for other learning rates and Table 4 for Mistral-7B.

Figure 1: Per-token loss change on four unrelated domains after medical adaptation (GPT-2 Large). GAIN (

\sigma\approx 0.12

) is

3\times

narrower than LoRA (

\sigma\approx 0.30

Figure 2: LoRA’s forgetting-adaptation tradeoff. Red points are LoRA with different learning rates (lr) and L2 regularization strengths (

\lambda

, where

\ell=\ell_{\text{LM}}+\lambda\|p-p^{\text{prev}}\|^{2}

). Higher

\lambda

reduces forgetting but sacrifices adaptation. GAIN-FFN (blue star) operates outside this tradeoff curve at lr

\leq 3\!\times\!10^{-3}

	GPT-2 (774M)			Mistral-7B			Qwen2.5-7B			Llama-13B			Llama-70B
	Base	GAIN	LoRA	Base	GAIN	LoRA	Base	GAIN	LoRA	Base	GAIN	LoRA	Base	GAIN	LoRA
ARC-E	$46.6$	$-1.4$	$-1.9$	$79.5$	$+1.5$	$\mathbf{-10.7}$	$77.4$	$\mathbf{+3.8}$	$-0.2$	$76.4$	$+0.5$	$+1.5$	$79.5$	$\mathbf{+1.4}$	$+0.2$
ARC-C	$25.1$	$-1.0$	$-0.1$	$54.3$	$+0.3$	$\mathbf{-10.0}$	$51.2$	$\mathbf{+3.1}$	$-0.1$	$49.0$	$+0.0$	$+2.5$	$57.0$	$\mathbf{+1.4}$	$-1.0$
BoolQ	$60.5$	$-2.1$	$\mathbf{-7.8}$	$84.3$	$-1.2$	$\mathbf{-9.2}$	$84.3$	$-0.8$	$\mathbf{-14.9}$	$82.5$	$-1.4$	$-2.3$	$85.3$	$-2.5$	$\mathbf{-14.6}$
HellaS	$45.3$	$-0.9$	$-2.1$	$81.2$	$+0.1$	$\mathbf{-4.1}$	$78.9$	$+0.4$	$-0.9$	$79.7$	$+0.9$	$\mathbf{-5.3}$	$84.3$	$\mathbf{+0.7}$	$\mathbf{-7.1}$
OBQA	$31.2$	$-1.8$	$-2.2$	$46.2$	$+0.0$	$\mathbf{-3.0}$	$47.4$	$-1.0$	$\mathbf{-2.8}$	$45.6$	$+0.4$	$\mathbf{-3.2}$	$49.0$	$-0.2$	$\mathbf{-10.6}$

Table 3: Benchmark accuracy change after seven sequential domain adaptations (acc_norm). Same hyperparameters across all models (lr

=10^{-3}

, 200 steps, 5 epochs per domain). Benchmarks are not in the training set.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

GAIN (lr $=10^{-3}$ , % PPL change vs. pretrained baseline)
After training	WT	Med	Fin	PG	MdQ	LB	SST	HS
+WT	$-23.1$
+Med	$-23.2$	$-1.2$
+Fin	$-23.1$	$-1.1$	$-5.0$
+PG	$-22.9$	$-1.1$	$-5.0$	$-0.3$
+MdQ	$-22.7$	$-1.1$	$-4.9$	$-0.2$	$-4.6$

Appendix

Table 4: Sequential adaptation on Mistral-7B (200K tokens per domain). Same format as Table 2 .

GAIN (lr $=3\!\times\!10^{-3}$ , % PPL change vs. pretrained baseline)
After training	Med	WT	Fin	PG	MdQ	LB	SST	HS
+Med	$-5.7$	$+0.2$	$+0.0$	$-3.5$	$-0.3$	$-1.3$	$+0.1$	$-0.2$
+WT	$-4.9$	$-15.8$	$+0.5$	$-6.4$	$+0.2$	$-1.6$	$-2.7$	$+0.2$
+Fin	$-3.9$	$-15.3$	$-8.6$	$-4.4$	$+0.7$	$-3.6$	$-1.0$	$+0.5$
+PG	$-3.5$	$-14.5$	$-8.2$	$-25.0$	$+0.5$	$-5.0$	$-1.0$	$+0.8$
+MdQ	$-2.1$	$-13.4$	$-7.2$	$-24.2$	$-14.2$	$-5.5$	$-0.2$	$+1.5$

Appendix

Table 5: Sequential adaptation on GPT-2 Large at lr

=3\!\times\!10^{-3}

(GAIN). Same format as Table 2 . Stronger adaptation with minor interference on later domains.

LoRA (lr $=3\!\times\!10^{-4}$ , % PPL change vs. pretrained baseline)
After training	Med	WT	Fin	PG	MdQ	LB	SST	HS
+Med	$-5.6$	$+0.2$	$+0.1$	$-3.5$	$-0.3$	$-1.3$	$+0.2$	$-0.3$
+WT	$+3.8$	$-19.7$	$+2.4$	$-5.6$	$+4.7$	$+2.8$	$-2.5$	$+2.4$
+Fin	$+3.0$	$-13.9$	$-9.1$	$-12.6$	$+2.5$	$-4.9$	$+0.6$	$+1.1$
+PG	$+1.5$	$-11.7$	$-6.8$	$-25.5$	$+1.9$	$-5.4$	$-1.2$	$+1.4$
+MdQ	$+9.3$	$-5.1$	$-1.0$	$-21.9$	$-17.3$	$-3.8$	$+1.4$	$+2.7$

Appendix

Table 6: Sequential adaptation: LoRA at safe lr

=3\!\times\!10^{-4}

on GPT-2 Large. Same format as Table 2 . Even at the safe LR, forgetting accumulates to

+10.2\%

Method	Targets	LR	Params	In-domain $\downarrow$	Final forgetting
GAIN ( $W_{O}$ )	—	$10^{-3}$	46K	$-10.2\%$	$\mathbf{-9.8\%}$
LoRA $r\!=\!1$	$W_{O}$	$10^{-3}$	92K	$-21.3\%$	$+12.2\%$
LoRA $r\!=\!8$	$W_{O}$	$10^{-3}$	737K	$-24.3\%$	$+11.7\%$
LoRA $r\!=\!16$	$W_{O}$	$10^{-3}$	1.47M	$-24.5\%$	$+13.3\%$
LoRA $r\!=\!8$	$W_{O}$	$3\!\times\!10^{-4}$	737K	$-21.5\%$	$+14.0\%$
LoRA $r\!=\!8$	QKVO	$10^{-3}$	2.21M	$-23.8\%$	$+21.6\%$

Appendix

Table 7: LoRA at standard ranks and QKVO target sets, on GPT-2 Large. Same protocol as Table 2 (

5

epochs,

200

steps,

200

K tokens/domain, lr

=10^{-3}

unless noted). In-domain : mean PPL change on the diagonal (

\downarrow

= better adaptation). Final forgetting : mean PPL change on the seven earlier-trained domains after the last adaptation (positive

=

forgetting).

Figure 3: Loss landscape interpolation on Mistral-7B. Left: in-domain PPL decreases for both. Right: GAIN’s cross-domain loss is flat; LoRA’s rises steeply.

Figure 4: Cross-domain loss when WikiText and Medical adaptations are combined. Left : GAIN—uniform blue; any combination preserves LAMBADA. Right : LoRA—red regions appear when adaptations are combined.

			PPL $\Delta$ %		Benchmark $\Delta$ (accuracy)
Model	Method	Params	PubMed	LAMB	ARC-E	HellaS	PIQA	WinoGr
Mistral-7B	GAIN	131K	$-3.1$	$-0.4$	$-0.3$	$-0.2$	$-0.5$	$+0.0$
	LoRA (lr $=10^{-3}$ )	262K	$+5.4$	$+2.7$	$+0.6$	$+0.3$	$-0.3$	$-0.1$
	LoRA (lr $=3\!\times\!10^{-4}$ )	262K	$-3.7$	$-0.0$	$+0.4$	$+0.1$	$-0.3$	$-0.1$
Llama-2-13B	GAIN	205K	$-2.4$	$-0.3$	$-0.1$	$+0.4$	$+0.0$	$+0.5$
	LoRA (lr $=10^{-3}$ )	410K	$+1.4$	$-0.2$	$-1.3$	$+0.9$	$+0.2$	$-0.5$

Appendix

Table 8: Single-domain adaptation (PubMedQA) across model scales. PPL columns: % change (

\downarrow

= better). PubMed is in-domain; LAMBADA measures forgetting. Benchmarks: accuracy (4 tasks averaged).

	GAIN (lr $=10^{-2}$ )		LoRA (lr $=3\!\times\!10^{-3}$ )
Model	WikiText	LAMBADA	WikiText	LAMBADA
Mistral-7B	$\mathbf{-15.9\%}$	$+0.2\%$	$+213\%$	$+180\%$
Llama-2-13B	$\mathbf{-25.7\%}$	$-0.1\%$	$-28.4\%$	$+10.0\%$
Llama-2-70B	$\mathbf{-18.0\%}$	$+0.1\%$	$+12.3\%$	$+35.6\%$

Appendix

Table 9: GAIN (lr

=10^{-2}

) vs LoRA (lr

=3\!\times\!10^{-3}

) on WikiText adaptation. GAIN adapts strongly with zero forgetting at every scale. LoRA diverges on Mistral-7B and Llama-2-70B, and causes

+10\%

forgetting on Llama-2-13B.

	lr $=10^{-3}$			lr $=3\!\times\!10^{-3}$			lr $=10^{-2}$
After	$s_{\min}$	$s_{\max}$	% tight	$s_{\min}$	$s_{\max}$	% tight	$s_{\min}$	$s_{\max}$	% tight
+Dom 1	$0.94$	$1.06$	$99.9$	$0.84$	$1.16$	$67.6$	$0.52$	$1.48$	$29.3$
+Dom 2	$0.78$	$1.25$	$73.3$	$0.40$	$1.62$	$38.1$	$\mathbf{-0.76}$	$2.36$	$16.6$
+Dom 4	$0.61$	$1.40$	$55.8$	$0.07$	$1.94$	$27.8$	$\mathbf{-0.83}$	$2.87$	$12.9$
+Dom 6	$0.34$	$1.78$	$42.7$	$\mathbf{-0.38}$	$2.15$	$22.3$	$\mathbf{-1.25}$	$3.06$	$11.5$
+Dom 8	$0.24$	$1.77$	$36.8$	$\mathbf{-0.51}$	$2.28$	$19.3$	$\mathbf{-1.33}$	$3.11$	$10.4$

Appendix

Table 10: Scaling factor statistics after each sequential domain (GPT-2 Large). At lr

=10^{-3}

, factors stay positive and tightly concentrated. At lr

=3\!\times\!10^{-3}

, factors go negative by domain 5. At lr

=10^{-2}

, factors go negative by domain 2 and reach

-1.33

by domain 8. Negative scaling factors sign-flip hidden dimensions, destroying the multiplicative guarantee.

LR	In-domain $\downarrow$	Earlier domains	Regime
$5\!\times\!10^{-4}$	$-14.6\%$	$\mathbf{-12.8\%}$ better	conservative
$10^{-3}$	$-17.9\%$	$\mathbf{-12.9\%}$ better	default
$2\!\times\!10^{-3}$	$-20.7\%$	$\mathbf{-10.1\%}$ better	aggressive
$3\!\times\!10^{-3}$	$-21.9\%$	$\mathbf{-6.7\%}$ better	matches LoRA in-domain
$10^{-2}$	$-22.5\%$	$+13.5\%$ worse	unstable
LoRA ( $10^{-3}$ )	$-21.4\%$	$+17.9\%$ worse	—

Appendix

Table 11: GAIN-FFN learning rate sweep on GPT-2 Large (sequential 8-domain adaptation, 200K tokens/domain). In-domain: avg diagonal PPL improvement. Earlier domains: avg PPL change on previously trained domains (negative = improvement).

GAIN-FFN (2M tokens, lr $=10^{-3}$ , % PPL change vs. pretrained)
After training	Med	WT	Fin	PG	MdQ	LB	SST	HS
+Med	$-6.8$	$+0.5$	$-0.3$	$-3.1$	$-0.1$	$-2.7$	$+0.5$	$-0.4$
+WT	$-4.1$	$-17.6$	$+0.0$	$-2.8$	$+0.3$	$-2.5$	$-3.7$	$+0.7$
+Fin	$-2.9$	$-15.9$	$-11.7$	$-3.0$	$-0.7$	$-7.7$	$-1.0$	$+0.2$
+PG	$-3.1$	$-14.7$	$-10.9$	$-12.6$	$-1.0$	$-8.3$	$-1.5$	$+0.9$
+MdQ	$-0.7$	$-12.2$	$-9.7$	$-9.9$	$-23.1$	$-9.1$	$+1.4$	$+2.3$

Appendix

Table 12: Sequential adaptation with 2M tokens per domain (GPT-2 Large). Same format as Table 2 . GAIN-FFN maintains positive transfer; LoRA’s forgetting worsens.

Method	Tokens	Params	In-domain $\downarrow$	Final forgetting
GAIN ( $W_{O}$ )	$200$ K	46K	$-10.2\%$	$\mathbf{-9.8\%}$
GAIN ( $W_{O}$ )	$5$ M	46K	$-9.3\%$	$\mathbf{-8.5\%}$
LoRA $r\!=\!8$ ( $W_{O}$ )	$200$ K	737K	$-24.3\%$	$+11.7\%$
LoRA $r\!=\!8$ ( $W_{O}$ )	$5$ M	737K	$-23.6\%$	$+15.7\%$

Appendix

Table 13: Sequential 8-domain adaptation at

5

M tokens/domain on GPT-2 Large, same hyperparameters as Table 7 (

5

epochs,

200

steps, lr

=10^{-3}

). Final forgetting : mean PPL change on the seven earlier-trained domains after the last adaptation.

Method	Params	In-domain $\downarrow$	Final forgetting
GAIN-FFN	590K	$\mathbf{-22.5\%}$	$\mathbf{-20.6\%}$
GAIN ( $W_{O}$ )	131K	$-17.7\%$	$-16.6\%$
LoRA $r\!=\!1$ ( $W_{O}$ )	262K	$-23.7\%$	$+11.6\%$
LoRA $r\!=\!8$ ( $W_{O}$ )	2.10M	$-6.4\%$	$+56.9\%$

Appendix

Table 14: Sequential 8-domain adaptation on Llama-3-8B-Instruct. Same protocol and metrics as Table 7 .

Method	Params	In-domain $\downarrow$	Final forgetting	Buffer
LoRA $r\!=\!8$ ( $W_{O}$ )	737K	$-24.3\%$	$+11.7\%$	—
Replay-LoRA $r\!=\!8$	737K	$-22.4\%$	$-13.7\%$	128 sequences
GAIN-FFN	230K	$-17.3\%$	$\mathbf{-13.4\%}$	—
Replay-GAIN-FFN	230K	$-16.1\%$	$-15.1\%$	128 sequences

Appendix

Table 15: Replay (buffer of

16

sequences per prior domain; every 4th step samples from the buffer) added to LoRA and to GAIN-FFN on GPT-2 Large, same protocol as Table 7 . Buffer lists the extra state replay requires beyond model parameters; a ‘—’ indicates no auxiliary state.

Method	Params	In-domain $\downarrow$	Final forgetting
LoRA $r\!=\!8$ ( $W_{O}$ , vanilla)	737K	$-24.3\%$	$+11.7\%$
EWC-LoRA $\lambda{=}10^{3}$	737K	$\mathbf{-24.1\%}$	$+3.3\%$
EWC-LoRA $\lambda{=}10^{4}$	737K	$-21.5\%$	$-5.3\%$
EWC-LoRA $\lambda{=}10^{5}$	737K	$-14.7\%$	$-11.7\%$
GAIN-FFN	230K	$-17.3\%$	$\mathbf{-13.4\%}$

Appendix

Table 16: EWC-LoRA

\lambda

sweep on GPT-2 Large, same protocol as Table 7 . Diagonal Fisher estimated from

64

training sequences after each domain.

Step	Most recent domain	$s_{\min}$	$s_{\max}$	$s_{\text{mean}}$	% tight
1	wikitext	$0.832$	$1.178$	$0.999$	$73.2$
4	pg19	$0.588$	$1.418$	$0.998$	$53.4$
8	hellaswag	$0.480$	$1.566$	$0.996$	$37.7$
12	pg19	$0.480$	$1.566$	$0.994$	$31.4$
16	hellaswag	$0.434$	$1.566$	$0.993$	$26.0$
20	pg19	$0.434$	$1.566$	$0.992$	$23.8$

Appendix

Table 17: Scaling-factor distribution for GAIN-FFN through 24 sequential adaptations on GPT-2 Large (lr

=10^{-3}

5

epochs,

200

steps per adaptation). Domains cycle through the eight-domain order three times. % tight = fraction of scaling factors in

[0.95,1.05]

. Selected steps shown for compactness; full trajectory in experiments_apr17_round3/exp8/ .

Method	Params	In-domain $\downarrow$	Final forgetting
LoRA $r\!=\!8$ ( $W_{O}$ , vanilla)	737K	$-24.3\%$	$+11.7\%$
PackNet-LoRA ( $r{=}1$ /domain, $8$ domains)	737K	— (reported below)	$+4.1\%$
GAIN-FFN	230K	$-17.3\%$	$\mathbf{-13.4\%}$

Appendix

Table 18: PackNet-LoRA on GPT-2 Large, same protocol as Table 7 . Per-domain LoRA rank

r{=}1

;

8

adapters stacked over the sequence.

Method	In-dom WT	In-dom DE	Forward interference on DE	WT preserved after DE
GAIN-FFN	$-21.0\%$	$-7.4\%$	$\mathbf{+0.4\%}$	$-19.8\%$
LoRA $r\!=\!8$	$-32.9\%$	$-10.5\%$	$+18.1\%$	$-12.6\%$

Appendix

Table 19: GPT-2 Large adaptation on a 2-domain OOD protocol: WikiText then German Wikipedia. Forward interference on German is the German-Wikipedia PPL change after training only on WikiText—measuring how much cross-domain spillover each method introduces.

	Top-10 right-singular subspace overlap
Method	layer 0	layer 9	layer 18	layer 27	layer 35	mean
GAIN-FFN (multiplicative)	$0.99$	$0.89$	$0.91$	$0.82$	$0.96$	$\mathbf{0.91}$
GAIN ( $W_{O}$ ) (multiplicative)	$0.98$	$0.86$	$0.86$	$0.76$	$0.94$	$0.88$
LoRA $r\!=\!8$ (additive)	$0.60$	$0.50$	$0.38$	$0.44$	$0.73$	$0.53$
DoRA $r\!=\!8$ (additive)	$0.63$	$0.44$	$0.45$	$0.52$	$0.78$	$0.56$
	Relative Frobenius perturbation $\\|W_{\text{new}}-W\\|_{F}/\\|W\\|_{F}$

Appendix

Table 20: Singular-subspace preservation after the eight-domain sequential adaptation protocol on GPT-2 Large. Multiplicative methods (GAIN, GAIN-FFN) preserve

\sim\!88

–

91\%

of the pretrained top-

10

right-singular subspace; additive methods (LoRA, DoRA) preserve only

\sim\!53

–

56\%

, meaning roughly half of their top singular directions are “intruder” dimensions absent from

W

. Magnitude renormalisation (DoRA) does not change the subspace shift relative to vanilla LoRA.

Method	Params	In-domain $\downarrow$	Final forgetting
LoRA $r\!=\!8$ ( $W_{O}$ )	737K	$-24.3\%$	$+11.7\%$
DoRA $r\!=\!8$ ( $W_{O}$ )	783K	$-24.2\%$	$+14.2\%$
GAIN-FFN	230K	$-17.3\%$	$\mathbf{-13.4\%}$

Appendix

Table 21: DoRA vs LoRA on GPT-2 Large, same protocol as Table 7 .

Method	Params	In-domain $\downarrow$	Final forgetting
LoRA $r\!=\!8$ ( $W_{O}$ )	737K	$-25.8\pm 0.05\%$	$-2.0\pm 1.0\%$
GAIN-FFN	230K	$-19.2\pm 0.05\%$	$\mathbf{-14.7\pm 0.05\%}$

Appendix

Table 22: 19-distinct-domain protocol on GPT-2 Large (3 seeds each, lr

=10^{-3}

5

epochs,

200

steps per domain). Mean

\pm

std across seeds. Final forgetting : mean PPL change on the eighteen earlier-trained domains after the last adaptation.

(IA) 3 (lr $=10^{-3}$ , % PPL change vs. pretrained baseline)
After training	Med	WT	Fin	PG	MdQ	LB	SST	HS
+Med	$-5.7$	$+0.1$	$+0.2$	$-3.3$	$-0.4$	$-1.4$	$+0.1$	$-0.4$
+WT	$-4.9$	$-17.7$	$+0.6$	$-5.0$	$-0.0$	$-1.7$	$-3.2$	$+0.2$
+Fin	$-4.1$	$-17.3$	$-9.9$	$-6.9$	$-0.1$	$-4.9$	$-1.7$	$+0.2$
+PG	$-3.8$	$-16.6$	$-9.6$	$-25.6$	$-0.4$	$-6.1$	$-2.0$	$+0.3$
+MdQ	$-2.6$	$-15.5$	$-8.7$	$-24.7$	$-17.5$	$-7.1$	$-1.2$	$+0.9$

Appendix

Table 23: (IA) 3 sequential adaptation on GPT-2 Large (lr

=10^{-3}

, 200K tokens/domain). (IA) 3 shows positive transfer, confirming that the multiplicative principle—not GAIN’s specific parameterization—prevents forgetting.

Domain Adaptation for LLMs

Domain adaptation for large language models tailors their knowledge and behavior to specialized fields, languages, and tasks.

See all 93 papers

Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
May 28, 2026Runze Xu, Arpit Garg, Hemanth Saratchandran +1Continual Learning for LLMs LLM Fine-Tuning
Australian Institute for Machine Learning Adelaide University Adelaide SA 5000
When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization
Aug 29, 2026Jiayu Hou, Lei WangDomain Adaptation for LLMs Parameter-Efficient Fine-Tuning
University of Electronic Science and Technology of China, Chengdu, China
AutoAdapt: Automatic Domain Discovery Enables Low-Cost Extensibility
Oct 7, 2026Josh McGiff, Salma Mekaoui, Robert Shanahan +1Domain Adaptation for LLMs
University of Limerick, Ireland

May 28, 2026J/K move · Enter open · S save

Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting

Runze Xu, Arpit Garg, Hemanth Saratchandran, Simon Lucey

Australian Institute for Machine Learning Adelaide University Adelaide SA 5000

Continual Learning for LLMs LLM Fine-Tuning Low-Rank Adaptation Catastrophic Forgetting

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) has become one of the most widely used fine-tuning mechanisms for adapting large language models to new domains, tasks, and users. Yet adaptation performance alone can obscure an important failure mode: LoRA updates may improve performance on the target distribution while degrading prior capabilities learned during pretraining and alignment. We show that this forgetting becomes especially severe when the adaptation distribution differs substantially from the models original training or alignment distributions. The challenge is amplified in practical settings, where the original training and alignment data are typically unavailable. Motivated by this constraint, we study how LoRA based adaptation balances new learning against forgetting in a replay-free setting, and introduce a simple output space regularizer that can be added directly to existing training pipelines. Our method removes the ground-truth token from both the base and adapted model distributions, renormalizes the remaining probabilities, and applies KL regularization only over the non-target vocabulary. This preserves the base models relative preferences among alternative tokens without directly opposing the cross-entropy signal required for adaptation. As the regularizer acts only at the loss level, it requires no replay data, architectural changes, adapter redesign, or inference-time overhead, and can be applied directly to existing LoRA variants. Across all LoRA variants tested and across various backbones, our method improves the frontier between new learning and forgetting when the adaptation distribution differs substantially from the base models original training or alignment distributions, suggesting a broadly applicable route toward more reliable LLM updating.

Abstract

Domain Adaptation for LLMs

Explore similar work

Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting

When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization

AutoAdapt: Automatic Domain Discovery Enables Low-Cost Extensibility