Continual learning (CL) seeks to adapt pretrained models while preserving prior capabilities. Yet the tasks supplied for preservation cover only part of what a model knows, while forgetting can extend beyond them. Recent work shows that language models can supplement this limited evidence by generating useful replay without access to their original training data. This raises a question about self-preservation: can a model help choose preservation targets that protect its own capabilities more broadly? To study this question, we organize task descriptions and their examples into Task Banks, whose entries can be selected and combined to guide replay generation by a frozen model. Continual instruction tuning with Qwen models of varying sizes yields three findings. First, the new task and learning rate shape which prior capabilities degrade, while tracking previously correct answers reveals losses that average accuracy can conceal. Second, generated replay reduces forgetting on tasks outside the bank. Combining partial banks strengthens this protection, improving mean retention of previously correct answers over either bank alone across the tested sizes. Third, the model can also help choose replay targets: selection informed by the new task improves mean held-out retention over random selection at 4B, with similar new-task gains, although smaller models show no corresponding advantage. Under a fixed example budget, however, replacing new-task examples with replay improves preservation at the cost of smaller new-task gains. Together, these findings show that models can support broader self-preservation by generating replay data and, in some settings, choosing which tasks to replay.
Figures & tables
Figure 1: Generated replay across supplied targets can protect held-out capabilities while retaining new-task gains. Top: prior-task accuracy changes. Bottom: incoming-task gains for Qwen3-0.6B, 1.7B, and 4B after one epoch on 1,024 MedQA examples at 1e-5 . SFT uses incoming data only. Replay uses real prior-task examples and labels. Task-Bank Self-Replay uses model-generated questions and answers. Both replay methods cover MMLU, GSM8K, HellaSwag, and BoolQ. The plot shows seven prior tasks. Stars mark held-out tasks: ARC-C, WinoGrande, PIQA, and SVAMP. Points are five-seed means. Displayed error bars are ±1 SE. The zero ring denotes pre-update accuracy, and all radial panels span −10 to +4 pp.
Figure 2: Nearly unchanged accuracy can conceal substantial loss of previously correct answers. Leftward bars show correct-to-incorrect transitions. Rightward bars show incorrect-to-correct transitions. Diamonds show their net effect on accuracy. The four-task mean loses 3.56% and gains 3.61% of examples, for a net +0.05 pp change. Qwen3-4B learns MedQA using 256 input rows, one epoch, 1e-6 , and two seeds. Each task has 512 paired evaluation items. Bars use all items as the denominator. Retention (correct → correct) conditions on initially correct items.
Figure 3: Broad replay protects held-out tasks better than single-target replay while retaining similar new-task gains. SFT uses incoming data only. Single-Task Replay rehearses one prior target. Replay uses real examples and labels from MMLU, GSM8K, HellaSwag, and BoolQ. Task-Bank Self-Replay generates replay questions and answers for that bank. Rows show incoming gain, changes on three displayed replay tasks, and changes on four held-out tasks. Qwen3-4B learns C-STANCE for one epoch at 1e-5 . Circles are five-seed means with ±1 SE relative to fixed pre-update scores. Diamonds and stars summarize the mean and minimum of the displayed task means. Zero denotes pre-update performance.
Figure 4: Combining partial banks improves mean held-out Retention, although the best bank varies by task. Bank A contains MMLU and GSM8K. Bank B contains HellaSwag and BoolQ. A+B combines all four. Replay questions and answers are model-generated. Top: Retention (correct → correct) on ARC-C, WinoGrande, PIQA, and SVAMP, all held out from replay construction and target selection. The dashed outer ring is the frozen model’s 100% Retention reference, not 100% accuracy. The radial range is 60–100%. Bottom: MedQA accuracy gain. Each model uses the Figure 1 setting: 1,024 incoming examples, one epoch, and 1e-5 . Points and bars are means ±1 SE across five seeds.
Figure 5: Incoming-aware selection improves held-out Retention at 4B and approaches the full-bank reference with two targets. Random, bank-only, and incoming-aware selection each choose two entries from MMLU, GSM8K, HellaSwag, and BoolQ before replay generation. Bank-only selection receives the inventory. Incoming-aware selection also receives incoming-task information. The full bank uses all four entries. No corresponding mean advantage over random selection appears at 0.6B or 1.7B. Top: held-out Retention (correct → correct), with a 60–100% radial range and a dashed 100% pre-update reference. Bottom: MedQA accuracy gain. Training follows Figure 1 : 1,024 incoming examples, one epoch, and 1e-5 . Points and bars are five-seed means ±1 SE. All four evaluation tasks are excluded from selection and replay.
Figure 6: Replay can reduce the worst held-out loss while also reducing new-task gains under a fixed example allocation. Left: incoming accuracy gain. Right: mean seed-wise worst held-out accuracy change. Higher is better. Qwen3-4B learns MedQA at 1e-6 for one epoch and 16 updates. SFT uses 256 incoming rows. Replay arms use 197 incoming and 59 replay rows. Task-Bank Self-Replay generates answers to supplied questions. Replay uses stored examples and labels. Random replay uses one bank task. The post-hoc selector uses observed SFT damage as an analysis reference. Open circles are seeds. Diamonds average two seeds. Input rows and updates are matched, but tokens and generation cost are not. Table 8 gives paired forgetting and complete scores.
Figure 7: Different incoming tasks cause different patterns of forgetting under the same training protocol. Qwen3.5-4B is updated with SFT on MedQA or C-STANCE at 1e-6 , holding the model and optimization protocol fixed. Paired markers show each capability’s score change averaged across two seeds. Connecting segments compare update conditions, not training checkpoints. The two panels group three displayed bank tasks and four held-out tasks on the same horizontal scale. These groups identify evaluation tasks. Neither supplies replay in these SFT runs. The zero line denotes the pre-update model. The respective new-task gains are +4.7pp for MedQA and +7.8pp for C-STANCE.
Figure 8: A learning rate that supports useful new-task learning can still cause uneven losses across prior capabilities. The complete main sweep contains 36 SFT runs: three model rows, two update-task columns, three learning rates, and two seeds. Each run uses 256 selected input rows for one epoch. Cells show seed-mean percentage-point score change from the pre-update model. The top row of each panel is new-task gain. Horizontal rules separate it from three displayed bank tasks and four held-out tasks. Color uses one common scale saturated at ±10 pp. Printed values preserve larger magnitudes. At 1e-5 , several conditions lose new-task accuracy as well as prior-task accuracy, indicating failed adaptation rather than a necessary retention–acquisition trade-off. Bank tasks are replay-eligible evaluation tasks. No replay is used in these SFT runs. These are endpoint comparisons across learning rates, not training trajectories. The full capability profiles appear in Appendix D .
Learning rate
Incoming test Δ accuracy
Held-out mean Δ accuracy
Worst held-out Δ accuracy
Held-out Retention (%)
1e-7
+12.01±0.10
−0.10±0.10
−0.49±0.10
97.02±0.05
3e-7
+3.32±5.08
−0.73±0.34
−2.15±0.39
94.65±0.87
1e-6
+10.45±1.86
−1.98±0.46
−9.57±0.59
89.60±0.38
3e-6
+9.67±0.49
−3.03±0.24
−19.63±0.88
82.22±0.25
1e-5
−2.05±0.88
−28.00±3.00
−63.18±6.35
32.88±4.56
Table 1: A lower learning rate can improve both new-task learning and retention of prior correct answers. Qwen3-1.7B learns C-STANCE from 1,024 examples for one epoch (64 updates), separate from the 256-row study in Figure 8 . Values are means ± SE across seeds 0 and 1 at the FP32 final checkpoint. Accuracy changes compare matched pre/post predictions and use percentage points. ARC-C, WinoGrande, PIQA, and SVAMP each have 512 evaluation items. Incoming test accuracy uses a separate 512-item panel. Worst change is the minimum across held-out tasks within each seed, then averaged. Retention (correct → correct) is averaged equally across held-out tasks.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Role
Task
Evaluation format
Replay data
Update
MedQA
multiple choice
update data only
Update
C-STANCE
three-way choice
update data only
Replay tasks
MMLU
multiple choice
yes
Replay tasks
HellaSwag
multiple choice
yes
Replay tasks
GSM8K
numeric generation
yes
Replay tasks
BoolQ
binary choice
yes
Appendix
Table 2: Evaluation beyond replay targets tests the breadth of preservation. Replay data is available only for the replay tasks. The two update tasks supply training and validation data, while the held-out tasks are evaluated only after the update rule is frozen.
Model
Revision
TP
TPU
Qwen3-0.6B
c1899de289a0
4
v6e-4
Qwen3-1.7B
70d244cc86cc
4
v6e-4
Qwen3-4B
1cfa9a720891
8
v6e-8
Qwen3.5-4B
851bf6e806ef
8
v6e-8
Appendix
Table 3: Fixed model revisions help make preservation comparisons interpretable. Tensor parallelism (TP) and TPU shape describe the main sweep and additional 0.6B sweep. The Qwen3-4B protection replication used TP =4 on v6e-4 with the same pinned weights.
Task
Pre-update
SFT
Single-Task Replay
Replay
Task-Bank Self-Replay
New task
C-STANCE
52.44
80.21±0.44
80.18±0.72
80.30±0.83
80.29±0.75
Displayed replay tasks
MMLU
64.26
63.94±0.68
64.92±0.50
64.85±0.20
64.89±0.30
HellaSwag
73.92
67.30±0.69
69.01±0.53
74.30±0.20
73.50±0.40
BoolQ
85.10
79.00±0.20
80.20±1.00
84.90±0.00
85.10±0.10
Appendix
Table 4: Broad replay can limit prior-capability losses while maintaining new-task gains. Single-Task Replay uses one prior target. Replay and Task-Bank Self-Replay use the four-task inventory: Replay uses stored examples and labels, and Task-Bank Self-Replay uses model-generated questions and answers. Qwen3-4B, learning rate 1e-5 , one epoch, five seeds. Scores are absolute accuracy percentages. Pre-update scores come from the initial checkpoint. Other entries are endpoint means ± SE.
A. Confidence-based competence estimates
Task-level
Item-level
Model
Spearman ↑
Gap ↓
AUROC ↑
ECE ↓
Brier ↓
Qwen3-1.7B
0.679
0.328
0.681
0.331
0.332
Qwen3-4B
0.607
0.235
0.756
0.238
0.239
Qwen3.5-4B
0.857
0.059
0.787
0.069
0.139
Appendix
Table 5: Improved calibration does not guarantee improved task-level competence ranking. Panel A contrasts task-level estimates with item-level confidence metrics across seven multiple-choice capabilities. Panel B reports mean absolute error (MAE) when task accuracy is estimated from n labeled behavioral probes and tested on the disjoint remainder, over 100 resamples. Larger probe budgets reduce this error for every model. Arrows indicate the preferred direction. Calibration gap, ECE, and MAE use the 0–1 accuracy scale, while Spearman, AUROC, and Brier score are dimensionless. Bold identifies the best model within each column, not statistical significance.
Replay interface
Supplied to the model
Model contributes
Replay
Replay questions and labels
No generated replay answer
Supplied-question answers
Demonstrations and unlabeled questions
Answers to supplied questions
Question-and-answer generation
Supplied task entries
Replay questions and answers
Exploratory expanded bank
Incomplete task inventory
Additional task entries, then questions and answers
Appendix
Table 6: Replay designs differ in the preservation information they require. These are alternative interfaces. The endpoint and supplied-bank composition/selection studies generate both questions and answers. The fixed-example-allocation study instead generates answers to supplied questions. Model-proposed additions are examined in the separate exploratory bank-expansion pilot.
Task
Base (%)
Post (%)
Change (pp)
Retention (%)
MedQA test
43.95
39.45
−4.49
47.11 [40.44, 53.34]
ARC-C
74.80
67.77
−7.03
78.59 [74.41, 82.51]
WinoGrande
54.30
51.37
−2.93
49.28 [43.17, 55.76]
PIQA
64.65
61.72
−2.93
45.62 [40.79, 51.06]
SVAMP
78.32
19.53
−58.79
22.69 [18.95, 26.93]
Appendix
Table 7: The expanded-bank pilot loses new-task accuracy and held-out behavior at the tested operating point. The pilot uses Qwen3-1.7B on MedQA with seed 1 and 512 matched examples per endpoint. Retention (correct → correct) conditions on initially correct answers. Brackets are 95% paired-example bootstrap intervals (1,000 resamples), which describe evaluation uncertainty within this run. Accuracy changes are post minus base. Mean held-out change is −17.92 pp. This single-arm result does not measure expansion’s effect relative to a fixed bank.
Replay tasks
Held-out evaluation
Method
New-task
Replay
Worst
MF
Worst
MF
gain (pp)
rows
(pp)
(%)
(pp)
(%)
SFT
+1.17
0
-3.04
3.68
-1.56
5.74
Random replay
+0.98
59
-1.37
3.02
-1.76
5.00
Task-Bank Self-Replay
+0.49
59
-0.39
2.03
-0.68
3.52
Replay
+0.98
59
-1.27
2.26
-0.88
3.82
Appendix
Table 8: Generated replay reduces measured forgetting alongside smaller new-task gains at a fixed example allocation. Qwen3-4B is adapted to MedQA at 1e-6 for one epoch with two seeds per arm. All arms allocate 256 input rows. Replay occupies 59 rows before tokenization, replacing new-task inputs. Worst is the mean of each seed’s minimum capability change within the indicated task group. Retention (correct → correct) equals 100−MF on the same eligible tasks. MF is mean forgetting among initially correct examples over tasks with enough initially correct examples under the rule in Appendix A.1 . It need not cover every task included in Worst. Higher changes and lower MF are better. The post-hoc selector is analysis-only. Bold identifies the best reported retention summary, not statistical significance.
Figure 9: Positive average changes can conceal losses on individual prior tasks even when the incoming task improves. Qwen models after C-STANCE SFT at selected operating points with new-task gains of +7.1 , +6.0 , and +7.8pp , respectively. Each circle is a task’s change averaged across two seeds. Diamonds and stars show the mean and minimum across eight task means. Filled markers identify bank tasks and open markers held-out tasks. These labels identify the evaluation split, not replay in these SFT runs. Blue denotes 0.6B parameters and orange 4B. Horizontal ranges differ by model, with zero marking the pre-update score. Compare magnitudes by tick values, not visual distances. The learning rates are 3e-6 , 1e-6 , and 1e-6 . The 0.6B panel uses the separate 256-row C-STANCE calibration, not the 1,024-row follow-up. This selected comparison does not isolate the effect of model scale from model generation and learning rate. The complete grid follows.
Figure 10: An update can damage prior capabilities without improving performance on the new task. Rows identify models and columns identify update tasks. Color identifies learning rate. Rightward is greater new-task gain, and upward is a more positive prior-task accuracy change. Each circle is one capability’s change averaged across two seeds: filled circles denote bank tasks and open circles denote held-out tasks. Diamonds and stars summarize the mean and minimum of these eight capability means. Circles have small horizontal offsets for visibility. Summary markers retain the exact new-task gain coordinate. Zero denotes no change from the pre-update model. Vertical scales match within model rows but differ across rows. All main sweep learning-rate conditions are retained.
Figure 11: Learning rate changes both the severity and the distribution of forgetting, with different patterns across models and incoming tasks. The top three rows show the complete paired main sweep, with model by row and update task by column. The bottom panel separately shows the additional Qwen3-0.6B C-STANCE sweep. Circles show capability means, with filled markers for bank tasks and open markers for held-out tasks. Diamonds and stars are the mean and minimum of the eight capability means. Labels below each learning rate report new-task gain in percentage points and seed count n . Vertical scales match within model rows but differ across rows. The additional Qwen3-0.6B calibration uses 256 rows and covers C-STANCE only. It is separate from the 1,024-row follow-up. Every measured condition is retained, including high-rate failures as stress tests.
Figure 12: Broad replay reduces held-out accuracy losses across the tested sizes, but an average accuracy gain can still conceal an individual task loss. Qwen3 models are adapted to 1,024 MedQA examples for one epoch at learning rate 1e-5 . Average and worst-task accuracy loss summarize ARC-C, WinoGrande, PIQA, and SVAMP. Each point summarizes five-seed task means. Negative values indicate improvement. Lines connect measured sizes.
Initial → final change (pp)
Stage 1 → 2 (pp)
Method
MedQA
C-STANCE
Worst replay
Worst held-out
MedQA
Sequential SFT
+5.68
+13.00
-12.55
-8.79
+0.39
Replay
+9.39
+13.60
-2.35
+2.54
+0.59
Task-Bank Self-Replay
+3.91
+17.20
-5.49
-12.70
+0.20
Task-Bank Self-Replay + MedQA
+3.52
+16.60
-9.80
-3.71
-0.20
Appendix
Table 9: Adding a replay target can improve held-out accuracy while worsening accuracy on replay tasks. Adding MedQA to Task-Bank Self-Replay improves worst held-out change but worsens worst replay-task change relative to the fixed bank. Qwen3-0.6B learns MedQA then C-STANCE with one seed. The first four score columns compare the final model with the initial model. The last compares MedQA accuracy immediately before and after the C-STANCE update on the same 511 evaluation examples. All changes are in percentage points. The expanded arm remains above the initial MedQA score but slightly below its own first-stage score, illustrating why final gains do not establish retention of every intermediate gain. Stage-relative changes measure net accuracy, not preservation of individual correct answers. No uncertainty or significance is inferred from this one-seed pilot.
Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting.
Adapting pretrained models to new data can cause catastrophic forgetting of previously learned behavior. When only a few past samples remain, they give continual learning methods sparse and narrow evidence about what to preserve. We show that language models can expand this evidence through self-probing, in which the frozen model generates new inputs from the retained samples and records its own predictions on them. Unlike prior work that replays such data as training examples, our method, CPLUS uses self-probe and past-sample gradients to scale down parameter updates that conflict with prior behavior. Experiments with five language models on four benchmarks show three results. First, the same probes preserve more prior behavior as gradient signals than as replay data. Second, CPLUS learns the new data while consistently reducing forgetting more than existing baselines, especially when past data are scarce, and this protection extends to benchmarks not used for training. Third, we observe that CPLUS also becomes more effective as models grow: within the Qwen3 model family, it recovers an increasing share of the forgetting caused by standard fine-tuning.
Continual pre-training of large language models must acquire new information without erasing old knowledge. Existing replay methods often choose a global old/new mixture and sample uniformly, ignoring that examples differ in how quickly they are forgotten. We formulate continual pre-training as adaptive review scheduling: the training loop should decide not only how much history to replay, but which examples should return at each step. We introduce Spaced Repetition Training (SRT), a continual learning framework inspired by cognitive science, which schedules sample-rehearsal using the SuperMemo-2 (SM-2) algorithm. SRT maintains per-example review state, maps per-example perplexity to a recall-quality signal, and schedules historical examples for retention and new examples for consolidation while leaving the model, objective, and optimizer unchanged. On temporally separated Wikipedia and code corpora, SRT improves the stability-plasticity trade-off, recovering 5 to 37 percentage points of old-knowledge accuracy lost by naive continual pre-training across model scales while preserving or improving new-knowledge acquisition. At larger scale, SRT preserves broad benchmark performance that naive continual pre-training and uniform replay substantially degrade. Experiments with vision and tabular data further suggest that the scheduling principle extends beyond language when paired with an appropriate recall signal.