Forget About It: Mitigating Forgetting Without Replay or Auxiliary Objectives

Anat Kleiman; Gintare Karolina Dziugaite; Jonathan Frankle; Sham Kakade; Mansheej Paul

cs.LGJan 9, 2025

Forget About It: Mitigating Forgetting Without Replay or Auxiliary Objectives

Authors: Anat Kleiman, Gintare Karolina Dziugaite, Jonathan Frankle, Sham Kakade, Mansheej Paul

Organizations: Harvard University, Kempner Institute · Google DeepMind · Databricks

Abstract

Have we made mitigating catastrophic forgetting more complicated than it needs to be? Continual adaptation often degrades performance on earlier tasks, motivating rehearsal buffers, auxiliary objectives, and adaptive merging procedures. We investigate how far a simpler alternative-periodic parameter interpolation during fine-tuning-can go on its own. We introduce Sequential Fine-tuning with Averaging (SFA) and evaluate it alongside replay-free Lookahead, originally designed for optimization stability to find compelling results. In Qwen3-14B adapter continual learning, fixed configurations of both methods outperform the tested replay and adaptive-merging configurations in mean final accuracy using up to 36.3% less task training time. Furthermore, across language and vision tasks, our results compete with more complex continual-learning methods. Together, these findings suggest that simple in-training interpolation deserves consideration before introducing more elaborate forgetting-mitigation mechanisms.

Figures & tables

Figure 1: Qwen3-1.7B Math

\to

Law: replay-free comparisons over three seeds (17, 42, 123). Both axes show mean

\pm

sample SD. All underlying Law-training runs use learning rate

10^{-5}

. SFA uses the development-selected

\beta_{\mathrm{end}}=.5

. Appendix B gives the configurations; Appendix B.1 reports the SFA strength sweep. Table 4 additionally includes seeds 7 and 2026 and uses development-selected sequential fine-tuning at

5\cdot 10^{-6}

Figure 2: Llama 2-7B Math

\to

Law experiments. (a) Intermediate SFA merges recover Math accuracy lost during sequential fine-tuning. (b) The SFA interval sweep is compared with endpoint-interpolation and Task Arithmetic weight sweeps, showing favorable intermediate-merge operating points beyond the equal-weight endpoint. Panel (b) sweeps weights of 0.25, 0.50, and 0.75 on the Law-adapted checkpoint for both endpoint-interpolation and Task Arithmetic. Replay samples from full history.

Method	Final average (%) $\uparrow$	Runtime (s) $\downarrow$
SFA	$78.4\pm 0.8$	$\mathbf{117.5}$
Replay (10%)	$78.8\pm 0.3$	$133.0$
SFA-R	$\mathbf{79.8}\pm 0.2$	$130.7$

Table 1: Qwen3-1.7B full-parameter four-task learning at a common total token budget. Final average test accuracy is mean

\pm

sample SD over three seeds; runtime averages two complete-stream training-loop measurements. Replay and SFA-R use approximately 10% replay after the first task. Bold marks the highest accuracy and lowest measured runtime.

Figure 3: Qwen3-14B adapter results under matched current-task exposure. (a) : final four-task accuracy for fixed configurations (three-seed mean and sample SD). Both SFA and Lookahead exceed the tested replay and AIMMerging means. (b) : controlled Amazon-stage training-loop time on one H200 (two repetitions); the measured SFA savings are specific to this stage. Peak GPU memory is comparable.

Appendix figures & tables41 assets

Supplementary material from the paper’s appendix.

Appendix

Method	8B	14B
SFA	243.0	317.6
Replay (buffer)	277.7	389.7
AIMMerging (ported)	350.8	498.4

Appendix

Table 2: Controlled single-stage training-loop time in seconds. Each entry averages two runs on one H200.

Method	Replay-gradient	Monitoring
SFA / Lookahead	0	0
Replay (buffer)	1.20M	0
AIMMerging (ported)	2.51M	4.20M

Appendix

Table 3: Additional training work in the 14B four-task adapter stream. Every method processes 6.01M current-task tokens. Replay and monitoring tokens are distinct from the single-stage timing in Figure 3 .

Method	GSM8K	Law mean
Math-trained start	$74.80\pm 0.65$	$35.63\pm 1.87$
Sequential fine-tuning	$71.18\pm 0.61$	$90.07\pm 0.24$
Lookahead, $\beta=.5$	$71.45\pm 0.95$	$89.74\pm 0.32$
SFA, $\beta_{\rm end}=.5$	$\mathbf{71.83}\pm 1.14$	$\mathbf{90.18}\pm 0.43$

Appendix

Table 4: Five-seed Qwen3-1.7B Math

\to

Law test accuracy (%, mean

\pm

sample SD; seeds 17, 42, 123, 7, 2026). The starting model is evaluated before Law fine-tuning; bold marks the highest adapted mean. Configurations maximize Math development accuracy while allowing at most a one-point Law reduction relative to reference sequential fine-tuning (Appendix B ). Sequential fine-tuning uses learning rate

5\cdot 10^{-6}

Parameter set	Implementation	SFA	Lookahead	SFA less time (%)
1.7B, full parameters	Per-tensor	9.30	13.13	29.2
	Batched	5.09	8.60	40.8
8B, full parameters	Per-tensor	39.91	56.09	28.8
	Batched	24.09	40.76	40.9
14B, full parameters	Per-tensor	70.48	98.98	28.8
	Batched	43.44	73.55	40.9

Appendix

Table 5: Isolated merge cost and recorded cumulative merging time. All times are milliseconds. SFA and Lookahead each use one auxiliary parameter copy; reductions are relative to Lookahead.

Method	Coefficient / LR	Math	Law
SFA	$\beta_{\rm end}=.25$	73.2	91.523
	$\mathbf{.5}$	76.4	90.339
	$.75$	75.4	88.722
Lookahead	$\beta=.25$	72.2	91.730
	$\mathbf{.5}$	76.6	90.471
	$.75$	76.0	88.846

Appendix

Table 6: Development results underlying configuration selection for Table 4 (seed 17; accuracies in %). Bold rows identify the configuration with the highest Math accuracy among those achieving at least 89.843% Law accuracy. These scores determine selection given one seed; the five-seed test results provide the performance comparison.

	SFA		Lookahead		Sequential fine-tuning selected LR
Seed	GSM8K	Law	GSM8K	Law	GSM8K	Law
17	71.42	90.22	71.19	89.59	71.11	89.92
42	71.57	90.49	70.96	90.23	70.20	90.39
123	73.77	90.57	73.09	89.61	71.42	90.23
7 (prospective)	71.65	90.12	70.66	89.90	71.87	90.00
2026 (prospective)	70.74	89.49	71.34	89.40	71.27	89.81

Appendix

Table 7: Math

\to

Law per-seed test accuracy (%). Bold indicates the highest accuracy for each task within each seed. Seeds 7 and 2026 are prospective replications of the frozen middle-requirement configurations.

Seed group	Comparison	GSM8K	Law
Seeds 17, 42, 123	SFA minus Lookahead	$0.51\pm 0.24$	$0.62\pm 0.35$
Seeds 17, 42, 123	SFA minus sequential fine-tuning	$1.34\pm 1.02$	$0.25\pm 0.13$
Seeds 7, 2026	SFA minus Lookahead	$0.19\pm 1.13$	$0.16\pm 0.10$
Seeds 7, 2026	SFA minus sequential fine-tuning	$-0.38\pm 0.21$	$-0.10\pm 0.31$
All five	SFA minus Lookahead	$0.38\pm 0.61$	$0.43\pm 0.36$
All five	SFA minus sequential fine-tuning	$0.65\pm 1.19$	$0.11\pm 0.26$

Appendix

Table 8: Paired Math

\to

Law differences in percentage points (mean

\pm

sample SD). Positive values favor SFA. No significance or equivalence test is implied.

Method	GSM8K	Law mean
SFA, $\beta_{\rm end}=.5$	$72.25\pm 1.32$	$90.43\pm 0.19$
Endpoint interpolation $\beta=.5$	$72.25\pm 1.09$	$88.47\pm 0.09$
EMA, decay .999	$72.48\pm 1.25$	$87.25\pm 0.47$
Lookahead $\beta=.5$	$71.75\pm 1.17$	$89.81\pm 0.36$
Sequential fine-tuning learning rate $10^{-5}$	$67.07\pm 1.60$	$91.30\pm 0.41$
Sequential fine-tuning learning rate $5\cdot 10^{-6}$	$70.91\pm 0.63$	$90.18\pm 0.24$

Appendix

Table 9: Matched Qwen3 Math

\to

Law averaging and fine-tuning comparison (%, seeds 17, 42, 123, mean

\pm

sample SD). These are the non-replay comparison points in Figure 1 ; Table 10 gives the complete SFA sweep.

$\beta_{\mathrm{end}}$	Per-merge $\beta$	GSM8K	Law mean	Math forgetting	Law gain
Math-trained start	—	$74.78\pm 0.86$	$35.15\pm 0.92$	—	—
$.25$	$.069$	$69.93\pm 0.57$	$91.22\pm 0.30$	$4.85\pm 1.42$	$56.06\pm 0.92$
$.50$	$.159$	$72.25\pm 1.32$	$90.43\pm 0.19$	$2.53\pm 0.50$	$55.27\pm 1.01$
$.75$	$.293$	$73.04\pm 1.26$	$88.05\pm 0.29$	$1.74\pm 0.66$	$52.90\pm 0.82$
$.90$	$.438$	$74.37\pm 0.89$	$80.98\pm 2.25$	$0.40\pm 0.04$	$45.82\pm 1.40$
$.99$	$.684$	$74.96\pm 1.15$	$54.96\pm 6.02$	$-0.18\pm 0.31$	$19.81\pm 5.53$

Appendix

Table 10: Qwen3-1.7B SFA strength sweep (three seeds; accuracy in %, changes in percentage points; mean

\pm

sample SD). Positive Math forgetting denotes a decline from the Math-trained start. Law gain is relative to that same starting model.

Input share	Supervised share	GSM8K	Law mean	CaseHOLD	ToS	Overruling
1	$\approx 11$	$74.50\pm 0.97$	$89.74\pm 0.22$	81.02	93.09	95.11
5	$\approx 38$	$74.93\pm 0.82$	$89.01\pm 0.16$	80.12	92.34	94.56
10	$\approx 56$	$75.76\pm 0.85$	$88.76\pm 0.23$	79.67	91.83	94.78

Appendix

Table 11: Qwen3-1.7B bounded-memory replay (three seeds, fixed 500-example reservoir). GSM8K and Law mean show mean

\pm

sample SD; individual Law columns show means. Input and supervised shares refer to different token counts. All accuracies and shares are percentages.

Method	1.7B	8B	14B	32B
Sequential fine-tuning	$70.6\pm 0.8$	$76.2\pm 0.5$	$77.7\pm 1.1$	$78.1\pm 1.2$
Replay (buffer)	$74.6\pm 1.0$	$77.5\pm 0.3$	$78.6\pm 0.8$	$78.7\pm 0.3$
AIMMerging (ported)	$\mathbf{74.8}\pm 0.2$	$\mathbf{77.8}\pm 0.6$	$78.9\pm 0.6$	$79.0\pm 0.3$
Lookahead, fixed coefficient	$72.8\pm 0.6$	$76.7\pm 1.3$	$79.6\pm 0.6$	$79.1\pm 0.4$
SFA , fixed strength	$72.1\pm 1.9$	$77.4\pm 0.9$	$\mathbf{79.9}\pm 0.3$	$\mathbf{79.2}\pm 0.4$

Appendix

Table 12: Final average test accuracy in adapter continual learning (%, three seeds, mean

\pm

sample SD). Configurations fixed at 1.7B and transferred across sizes, with matched current-task exposure. Bold marks the highest fixed-configuration mean at each size. The 32B runs use activation checkpointing.

Method	Final average (%)
Sequential fine-tuning	$79.0\pm 0.2$
Replay (buffer)	$79.6\pm 0.2$
SFA	$\mathbf{79.9}\pm 0.4$
Lookahead	$79.3\pm 0.4$
AIMMerging (ported)	$78.9\pm 1.0$

Appendix

Table 13: Qwen3-14B shared-LoRA continual learning with an alternative task order: DBpedia

\to

Yahoo Answers

\to

Amazon

\to

AG News. Final average test accuracy (%, mean

\pm

sample SD over seeds 17, 42, and 123). Methods retain the fixed configurations used in Figure 3 : SFA uses

\beta_{\mathrm{end}}=.5

and Lookahead uses

\beta=.5

. Bold marks the highest mean.

Metric	Configuration	DBpedia	Amazon	Yahoo	AGNews
Final accuracy (%)	SFA, $\beta_{\rm end}=.5$	$97.04\pm 0.09$	$61.41\pm 0.41$	$71.11\pm 0.96$	$90.00\pm 0.30$
	Lookahead, $\beta=.5$ (fixed)	$96.24\pm 0.95$	$60.27\pm 1.22$	$71.57\pm 0.99$	$90.28\pm 0.54$
	Lookahead, $\beta=.75$ (selected)	$98.06\pm 0.19$	$61.97\pm 0.64$	$72.87\pm 0.48$	$89.90\pm 0.49$
Forgetting (points)	SFA, $\beta_{\rm end}=.5$	$1.37\pm 0.11$	$3.23\pm 0.58$	$1.53\pm 0.48$	—
	Lookahead, $\beta=.5$ (fixed)	$2.45\pm 0.98$	$4.79\pm 1.12$	$1.22\pm 0.68$	—
	Lookahead, $\beta=.75$ (selected)	$0.48\pm 0.21$	$2.36\pm 1.52$	$1.00\pm 0.21$	—

Appendix

Table 14: Task-wise retention after the complete 14B adapter stream, DBpedia

\to

Amazon

\to

Yahoo

\to

AGNews (test, seeds 17/42/123; mean

\pm

sample SD). Accuracy is measured after AGNews, not immediately after learning each task. Signed forgetting is the highest earlier accuracy minus final accuracy; lower is better. The last task has no subsequent stage and hence no forgetting entry. Fixed configurations are those in Figure 3 ; selected Lookahead comes from the equal three-point coefficient study.

Method	$\beta_{\mathrm{end}}$	DBpedia	Amazon	Yahoo Answers	AG News	Final average
Refreshed SFA	$.25$	$96.7\pm 0.6$	$58.0\pm 3.8$	$\mathbf{71.3}\pm 0.9$	$\mathbf{89.9}\pm 0.4$	$79.0\pm 1.0$
SFA-Ref	$.50$	$\mathbf{97.4}\pm 0.2$	$\mathbf{61.3}\pm 1.6$	$69.5\pm 1.0$	$89.6\pm 0.3$	$\mathbf{79.4}\pm 0.7$

Appendix

Table 15: Reference selection in Qwen3-14B LoRA continual learning. Both variants continue from an identical post-Amazon checkpoint within each seed and subsequently learn Yahoo Answers and AG News. Values are final test accuracies (%, mean

\pm

sample SD over seeds 17, 42, and 123). Amazon is the designated protected task; its starting accuracy is

64.4\pm 0.7

%. Strengths are selected on seed-17 development data. Bold marks the higher mean in each column.

$k$ (interval)	$\Delta$ Math	$\Delta$ Law	Merge saving
185	$+0.03\pm 0.46$	$+0.43\pm 0.26$	0.015 s (28.9%)
5	$+0.14\pm 0.68$	$+0.46\pm 0.28$	0.563 s (28.9%)
1	$+0.39\pm 0.51$	$+0.30\pm 0.23$	2.877 s (29.3%)

Appendix

Table 16: Full-parameter Math

\to

Law frequency controls: SFA minus Lookahead in accuracy (points, five-seed mean

\pm

SD) and measured cumulative merge-time savings for SFA on one H200. These pairwise differences diagnose reference-policy and implementation effects; they are not the main comparison to rehearsal-based methods.

		Cumulative merging		Training loop
Method	Interval	Rep. 1	Rep. 2	Rep. 1	Rep. 2
SFA	185	0.037	0.037	585.3	586.1
Lookahead	185	0.052	0.053	585.0	586.1
SFA	5	1.388	1.385	583.9	589.9
Lookahead	5	1.960	1.939	585.2	588.1
SFA	1	6.931	6.925	596.2	596.7

Appendix

Table 17: Qwen3-1.7B full-parameter Math

\to

Law fine-tuning: controlled cumulative merge time and complete Law-stage training-loop time (seconds), with two repetitions on the same H200 per merge interval using seed 17. Training-loop time excludes loading, saving, and evaluation. Merge time uses queued CUDA events; loop time uses synchronized wall-clock timing.

Figure 4: Full-parameter Qwen3-1.7B Math

\to

Law across merge intervals. Top: test accuracy (five-seed mean

\pm

sample SD). Bottom left: paired SFA-minus-Lookahead differences; small marks show individual seeds, with seed 17 hollow because it supplied development selection. Bottom right: controlled cumulative merge time in seconds on a logarithmic vertical scale (two repetitions per method). SFA uses

\beta_{\mathrm{end}}=.5

and Lookahead uses per-sync

\beta=.5

. Timings cover only merging.

Method	GSM8K	Law mean
Sequential fine-tuning	$28.2\pm 4.0$	$91.1\pm 0.5$
Endpoint interpolation, $\beta=.5$	$75.8\pm 1.1$	$87.5\pm 0.7$
Lookahead	$72.9\pm 2.7$	$89.2\pm 0.6$
SFA	$71.7\pm 3.4$	$89.9\pm 0.6$

Appendix

Table 18: Direct pretrained-to-Law full-parameter fine-tuning of Qwen3-1.7B. Test accuracy (%, mean

\pm

sample SD over three seeds) uses the standard evaluation prompt, not the worked-solution diagnostic. The pretrained model scores 79.9% on GSM8K and 51.6% on Law.

Method	DBpedia	Amazon	Yahoo Answers	AG News
SFA	$96.76\pm 0.11$	$\mathbf{60.40}\pm 0.21$	$66.63\pm 3.19$	$89.85\pm 0.22$
Replay	$97.00\pm 1.26$	$57.52\pm 1.60$	$70.15\pm 1.08$	$\mathbf{90.52}\pm 0.39$
SFA-R	$\mathbf{98.07}\pm 0.23$	$59.13\pm 1.32$	$\mathbf{72.37}\pm 0.81$	$89.72\pm 0.23$

Appendix

Table 19: Per-task test accuracy after the complete Qwen3-1.7B full-parameter stream, DBpedia

\to

Amazon

\to

Yahoo Answers

\to

AG News (%, mean

\pm

sample SD over seeds 17, 42, and 123). Every entry measures final accuracy, rather than accuracy immediately after learning that task. Replay and SFA-R use identical schedules with a 500-example reservoir and approximately 10% replay input tokens after the first task, under the same total token budget as SFA. Bold indicates the highest mean in each column.

Figure 5: Qwen2.5-1.5B Math

\to

Law comparison. SFA at

p=.25

retains more Math than the displayed Task Arithmetic and equal-weight endpoint configurations. (Table 24 ).

Figure 6: 20-task vision experiments with ViT-base: Food-101 (top) and CIFAR-100 (bottom). Replay-free SFA improves on the displayed sequential, Task Arithmetic, and endpoint configurations. Dashed lines use replay (full history) at the indicated sampling proportions. SFA-R is a replay-augmented variant of SFA. Multitask reference accuracies are annotated above the plotted range.

Figure 7: A comparison of sequentially fine-tuning ViT (base) on 20 tasks (Food-101) with ( bottom ) and without SFA ( top ). Each new task is introduced with a different colored curve across gradient timesteps (x-axis) resulting in changes to both current and past task accuracies (y-axis). The use of SFA can be seen to improve cumulative past task performance at averaging steps.

Figure 8: A comparison of varying the Task Arithmetic model weights, and

\beta

on SFA (

p

=1), with SFA (varying

p,\beta=0.5

) for Pythia (2.8B). We reproduce the results varying

p

in SFA (orange curve) from Figure 13 and add 2 sweeps showing change in performance on Pythia (2.8B) when the weights for the current and past checkpoints are varied for SFA (

p=1

) (dashed blue) and the domain-specific models are merged in Task Arithmetic (dashed red).

Figure 9: SFA compared against other continual learning methods, where the two tasks (Task A and B) were created by splitting MNIST by label. The accuracy after single-task training, sequential training, and multitask training is also shown. The lines for EWC and L2 are created by varying the coefficient corresponding to each method (and are the same for the top and bottom plots). (Top) visualizes SFA performance under varying

\beta

coefficient, which determines how much weight is being placed on the initial model. (Bottom) visualizes SFA with varying averaging frequency in steps before averaging.

Task/Domain	Eval Metric
Food-101	Food-101 holdout set
CIFAR-100	CIFAR-100 holdout set
Text Simplification (Simpl)	Text Simplification (Simpl) holdout set
Inquisitive Question Generation (InqQG)	Inquisitive Question Generation (InqQG) holdout set
Twitter Stylometry (TwSt)	Twitter Stylometry (TwSt) holdout set
Headline Generation with Constraint (HGen)	Headline Generation with Constraint (HGen) holdout set

Appendix

Table 20: Evaluation metrics for each task and domain used in our work.

Figure 10: An analysis of the negative correlation between accuracy on Math and the

L2

distance of the final model (fine-tuned on Math, then Law) from the original model (fine-tuned on Math only). The fine-tuning on Law is done using SFA with varying values of

p

that determine the merging frequency. For reference we also mark sequential fine-tuning which leads to much higher L2 distance due to no merging, and accuracy just below that achieved with SFA merging once at the end of fine-tuning on law (

p=1

Figure 11: The fine-tuning of T0_3B (3B) and T0pp (11B) on a stream of language generation tasks. Training loss on each subsequent task decreases as the model learns it, while evaluation loss on Simpl continues to increase, indicating that forgetting is present.

Figure 12: The Simpl loss curve of T0_3B (3B) from Figure 11 is compared to a multitask model training on all tasks, and a multitask model training on all tasks except Simpl. As T0_3B (3B) continues to fine-tune on each new task, the loss on Simpl becomes the same order of magnitude as that of a model that is never exposed to Simpl.

Pythia (2.8B)	case_hold	tos	overruling	GSM8K (0-shot)
Pythia (2.8B) Original	0.25	0.85	0.45	0
metamathqa	0.19	0.87	0.48	0.38
law	0.74	0.93	0.97	0
Sequential fine-tuning: metamathqa, Law	0.76	0.95	0.59	0
Endpoint interpolation: metamathqa, Law (p=1)	0.74	0.88	0.49	0.048
Endpoint interpolation: metamathqa, Law (p=1, 0.75 law, 0.25 math)	0.78	0.93	0.52	0

Appendix

Table 21: Results of Pythia (2.8B) models fine-tuning on Math and Law

Pythia (2.8B)	L2-Distance
metamathqa, Law - metamathqa	90.99
metamathqa, Law (p=1) - metamathqa	45.50
metamathqa, Law (p=0.5) - metamathqa	35.38
metamathqa, Law (p=0.25) - metamathqa	24.16
metamathqa, Law (p=0.10) - metamathqa	10.94
Task Arithmetic (0.5 Law, 0.5 math)- metamathqa	101.77

Appendix

Table 22: The L2 distance of Pythia (2.8B) models from previous checkpoints of models fine-tuning on Math and law

Llama 7B	case_hold	tos	overruling	GSM8K (0-shot)
Llama 2 (7B) Original	0.32	0.13	0.49	0
metamathqa	0.21	0.38	0.49	0.42
law	0.81	0.51	0.94	0
Sequential fine-tuning: metamathqa, Law	0.64	0.86	0.93	0
Endpoint interpolation: metamathqa, Law (p=1)	0.61	0.59	0.90	0.0015
Endpoint interpolation: metamathqa, Law (p=1, 0.75 law, 0.25 math)	0.64	0.83	0.94	0

Appendix

Table 23: Results of Llama 2 (7B) models fine-tuning on Math and law

Qwen2.5 1.5B	case_hold	tos	overruling	GSM8K (0-shot)
Qwen2.5 (1.5B) Original	0.47	0.28	0.90	0
metamathqa	0.26	0.19	0.69	0.60
law	0.77	0.91	0.95	0
Sequential fine-tuning: metamathqa, Law	0.78	0.92	0.94	0
Endpoint interpolation: metamathqa, Law (p=1)	0.81	0.88	0.93	0.01
SFA: metamathqa, Law (p=0.5)	0.77	0.89	0.94	0.12

Appendix

Table 24: Qwen2.5-1.5B Math

\to

Law accuracy. These single-run configurations illustrate retention-adaptation trade-offs.

Pythia (2.8B)	HumanEval (5-shot)	GSM8K (0-shot)
original Pythia (2.8B)	0.074	0
metamathqa	0.0	0.38
Magicoder-Evol-Instruct-110K	0.15	0
Sequential fine-tuning: metamathqa, Magicoder-Evol-Instruct-110K	0.13	0.01
Endpoint interpolation: metamathqa, Magicoder-Evol-Instruct-110K (p=1)	0.06	0.33
Endpoint interpolation: metamathqa, Magicoder-Evol-Instruct-110K (p=1, 0.3 math, 0.7 Code)	0.11	0.22

Appendix

Table 25: Results of Pythia (2.8B) models fine-tuning on Math and code

Llama 2 (7B)	HumanEval (5-shot)	GSM8K (0-shot)
Llama 2 (7B) Original	0.15	0
metamathqa	0	0.55
Magicoder-Evol-Instruct-110K	0.35	0
Sequential fine-tuning: Magicoder-Evol-Instruct-110K, metamathqa	0.046	0.54
Endpoint interpolation: Magicoder-Evol-Instruct-110K, metamathqa (p=1)	0.18	0.49
SFA: Magicoder-Evol-Instruct-110K, metamathqa (p=0.75)	0.22	0.41

Appendix

Table 26: Results of Llama 2 (7B) models fine-tuning on Code then Math, as indicated by the row order.

Pythia (2.8B)	case_hold	tos	overruling
Pythia (2.8B) Original	0.25	0.85	0.45
metamathqa	0.19	0.87	0.48
law	0.74	0.93	0.97
Magicoder-Evol-Instruct-110K	0.22	0.28	0.52
Sequential fine-tuning: metamathqa, Law, Magicoder-Evol-Instruct-110K	0.30	0.87	0.51
Endpoint interpolation: metamathqa, Law, Magicoder-Evol-Instruct-110K (p=1)	0.50	0.88	0.59

Appendix

Table 27: Results of Pythia (2.8B) models fine-tuning on Math, Law and Code for 2 orders

Figure 13: A comparison of Pythia (2.8B)’s performance on multiple domains (Math, Law and Math, Code) using various fine-tuning and model merging techniques similar to Figure 2 . On Math to Law, SFA

p=0.25

can be seen as having comparable performance to using a data buffer, while outperforming Task Arithmetic. Likewise, in Math to Code, SFA with varying

p

outperform using a data buffer and Task Arithmetic.

Figure 14: A comparison of Pythia (2.8B)’s performance when training on more than 2 domains (e.g. Math-Law and Code, Math-Code and Law) using various fine-tuning and model merging techniques similar to Figure 13 . On Math-Code to Law, SFA

p=0.25

can be seen as having comparable performance to using a data buffer, while outperforming Task Arithmetic. While, SFA with varying

p

on Math-Law to Code outperforms Task Arithmetic, but performs worse than using a data buffer.

Figure 15: A comparison of varying the Task Arithmetic model weights, and

\beta

on SFA (

p

=1), with SFA (varying

p,\beta=0.5

) for Pythia (2.8B) on math then code, similar to Figure 8 . ( Table 25 ).

Figure 16: SFA’s constant parameter-storage requirement compared with growing replay collections in hypothetical CIFAR-100-sized ( left ) and ImageNet-1k-sized ( right ) task streams. Replay retains 10% of each task without a capacity limit. The SFA curve includes both working and reference parameters, while replay curves show retained data only; adding replay’s working model would increase its total storage. Images are assumed uncompressed, and these analytical examples are distinct from the experimental task streams.

Figure 17: Analytical parameter-plus-buffer storage for SFA and CLS-ER with FP16 parameters and uncompressed

224\times 224

RGB images stored as uint8 pixels. Parenthesized legend values denote the number of retained images. SFA requires one reference alongside the working model, whereas CLS-ER maintains two auxiliary model copies and a replay buffer.

Figure 18: A comparison of SFA with and without using task boundaries to control model averaging and updating frequencies on a sequence of 20 tasks from Food-101 ( left ) and CIFAR-100 ( right ). While SFA is still effective without access to task boundaries, their incorporation enhances performance.

Catastrophic Forgetting

Catastrophic forgetting is the loss of previously learned behavior when a model adapts to new tasks or changing data.

See all 351 papers

Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay
May 25, 2026Martin Marek, Dongkyu Cho, Shikai Qiu +3Continual Learning for LLMs LLM Fine-Tuning
New York University
MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuning
Mar 10, 2026Yiyang Lu, Yu He, Jianlong Chen +1Continual Learning for LLMs Replay-Based Continual Learning
The Chinese University of Hong Kong, Shenzhen · Nanjing University
Interference and Retention in Continual Learning
Jul 10, 2026Julius StörkContinual Learning Online Continual Learning
VARTA Microbattery GmbH, Ellwangen, Germany

May 25, 2026J/K move · Enter open · S save

Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

Martin Marek, Dongkyu Cho, Shikai Qiu, Rumi Chunara+2

New York University

Continual Learning for LLMs LLM Fine-Tuning Replay-Based Continual Learning Catastrophic Forgetting Fine-Tuning Language Models

Open paper AskPDF arXiv

Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting.

Abstract

Catastrophic Forgetting

Explore similar work

Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuning

Interference and Retention in Continual Learning