Fine-tuning a language model on new data can make it forget facts that it should keep. We ask whether measurements taken at the start of a fine-tuning run can bound, for each protected fact, the probability that the run makes the model forget it. In LoRA fine-tuning with stochastic gradient descent on models from 0.6B to 14B parameters, a first-order response model estimated by finite-difference probes predicts changes of per-fact margins with correlation 0.974-0.998. Predictions of forgetting built on this model nevertheless failed, because forgetting requires parameter changes far outside the region in which the model was validated. The probes can, however, bound the probability that a margin first falls below a boundary near zero: we derive Freedman and Azuma first-passage bounds for a linear surrogate of the margin and test on new runs whether they hold for the model. The bounds contain a term R that measures how much the response coefficients change during the run. The simplified Freedman bound, which sets R = 0, certified most facts but was violated in 14 of 112 conditions, and every fact on which it was violated had R >= a, where a is the distance of the fact's margin to the boundary. The complete Freedman bound certifies only facts with R < a, and it held in every condition. On the violated facts, the spread of the margin across test runs was a median of 14.6 times the prediction of the response model, so the failures are breakdowns of the model, and in our data they occurred only where R >= a. We found this pattern post hoc and tested it in two preregistered confirmatory studies with 43 new conditions: the complete bound held in all of them, and the simplified bound failed there on only 3 facts, each with R >= a. First-order models of fine-tuning can thus bound forgetting on the facts whose response coefficients change by less than their distance to the boundary.
Figures & tables
Figure 1: The procedure for one condition (schematic). (1) A reference run gives the margins mi(t) , and probes along it give the response coefficients qi,t,s and the gradient covariance Σs . (2) These determine, for each fact, the distance ai to the boundary, the variance estimate Vi , the increment bound ci and the coefficient change Ri , the gap between the coefficients at time t and at the end T . (3) The complete bound uses ai−Ri , so it certifies only facts with Ri<ai (Sections 4 and 5 ). (4) A binomial test compares the forgetting events in the test runs with Ui .
Figure 2: (a) The 3,748 facts that the simplified Freedman bound certifies, by their distance a to the boundary and coefficient change R , both computed before the run. Red: facts on which the simplified bound was violated, in earlier (open) and prospective (filled) conditions; all lie in the shaded region R≥a , which the complete bound excludes. (b) Share of eligible facts forgotten in at least one test run, by R/a , with Wilson 95% intervals (Appendix H ). (c) For the four bounds, the fraction of eligible facts certified against the fraction of conditions with a violation, on the 41 earlier (open, post hoc) and 71 prospective (filled) conditions.
Simplified
Screened
Complete
Azuma
Per-fact
Margin
Freedman
bound
Freedman
bound
rule
threshold
Condition set
n
events
viol.
%
viol.
%
viol.
%
viol.
%
viol.
%
viol.
%
Earlier conditions (post hoc)
41
1,795
7
80
0
64
0
31
0
22
0
37
0
14
Qwen3-0.6B, synthetic
9
603
2
85
2
85
0
29
0
8
0
53
0
6
Llama-3.2-1B, synthetic
6
319
1
71
0
47
0
18
0
18
0
21
0
29
Qwen3-1.7B, synthetic
4
337
1
87
1
49
0
6
0
12
0
11
0
6
Table 1: Main results. For each condition set and rule: the number of violating conditions (viol.) and the percentage of eligible facts that the rule certifies (%). n is the number of evaluable conditions, and events is the number of forgetting events of eligible facts over all test runs. The rows of condition sets contain the main, horizon and 14B studies; the Qwen3-8B row contains the added batch (seeds 79 and 83). The confirmatory set contains the conditions of these studies whose test runs started after the confirmatory test was defined.
Rule
certified (%)
viol. cond.
fact-runs
events
allowed
Simplified Freedman ( R=0 )
81.0
14
118,928
344
467.5
Screened bound
67.2
6
98,480
202
314.8
Per-time Freedman
74.5
5
109,264
188
513.4
Complete Freedman
23.8
0
34,256
0
130.8
Azuma
17.3
0
24,688
0
178.8
Simplified Freedman on facts with R<a
31.0
0
44,912
0
29.8
Table 2: Forgetting events pooled over all 112 conditions (post hoc). For each rule: the percentage of eligible facts certified, the number of conditions with a violation, the pairs of a certified fact and a test run, the forgetting events among them, and the events that the values of the rule allow. The last two rows certify only facts with R<a .
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Checkpoint
Family
n
ρ
slope
Qwen3-0.6B
Qwen
4
0.985–0.994
0.901–1.047
Llama-3.2-1B
Llama
3
0.974–0.998
0.974–1.028
Qwen3-1.7B
Qwen
4
0.982–0.996
0.986–1.075
Qwen3-8B
Qwen
1
0.994
0.987
Gemma-3-12B
Gemma
2
0.992–0.997
1.003–1.010
Qwen3-14B
Qwen
1
0.998
1.014
Appendix
Table 3: First-order response model at task-B learning rate 0.002. A validation run passes a preregistered threshold if the correlation ρ between predicted and measured margin changes under held-out perturbations is at least 0.90 and the slope lies in [0.8,1.2] . Every run passes; n is the number of runs, and ranges are over runs.
Prediction
Independent test
Outcome
Per-fact event probabilities from the response model and a Gaussian noise model
fresh split of facts
mean absolute error 0.060 against a threshold of 0.05; all errors are false alarms
Rotating injected noise of fixed spectrum changes the forgetting rate by 0.145 between the extreme orientations
A drift statistic of the response coefficients ranks facts by prediction error
fresh set of 128 facts
mean rank correlation falls from 0.639 to 0.075
Two histories with the same margin vector have the same future distribution
branching from matched states
energy-distance permutation test, p=0.00035
A rank-2 projection of the protected gradients predicts how futures diverge
runs branched from matched states
predicts how much futures diverge (Spearman 0.658, against 0.050 for a random projection) but misses the direction
Appendix
Table 4: Five predictions built on the response model, each preregistered with a success criterion and tested on data disjoint from the data that suggested it. The outcome column gives the result that missed the criterion: an error above the threshold, an effect outside the predicted interval, a correlation that fell on replication, a rejected sufficiency hypothesis, and a prediction of the size of the divergence that missed its direction.
Threshold on z~
violating cond.
certified
0 (always simplified)
8
79.6%
10
8
79.6%
15
6
77.1%
20
2
67.2%
25
1
66.2%
30
0
62.3%
Appendix
Table 5: Post-hoc sweeps on the earlier conditions. Left: the screening threshold. Right: the simplified Freedman bound with its variance estimate inflated by κ (the default is 2).
Figure 3: Median standardized margin z~ of the facts that the simplified Freedman bound certifies, computed before the run, against the number of facts on which that bound is significantly violated (with small vertical jitter). Hollow: earlier conditions (post hoc); filled: all 71 prospective conditions. The screen selects the Freedman branch to the right of the dashed line.
simplified Freedman bound
screened bound
Set
bound range
facts
mean bound
event freq.
facts
mean bound
event freq.
Earlier
[0,0.001)
1140
0.0001
0.0014
1009
0.0001
0.0000
[0.001,0.01)
270
0.0041
0.0038
151
0.0039
0.0000
[0.01,0.05)
196
0.0253
0.0110
96
0.0240
0.0020
[0.05,0.2)
179
0.1057
0.0239
83
0.1085
0.0041
[0.2,0.5)
91
0.3324
0.0797
67
0.3510
0.0420
Appendix
Table 6: Mean bound and frequency of forgetting events, grouped by the value of the bound, for all 45 earlier conditions and for the main and horizon studies (the added 8B and 14B batches and the complete bound are covered by Table 1 ).
Study
Model
lr
seed
branch
z~
events
simplified
Azuma
screened
S18
Llama-3.2-1B
0.002
11
F
164.9
0
0/60
0/47
0/60
S6
Llama-3.2-1B
0.002
11
F
183.3
0
0/64
0/64
0/64
S13
Llama-3.2-1B
0.006
11
F
57.1
56
0/58
0/28
0/58
S13
Llama-3.2-1B
0.006
11
F
52.0
16
0/56
0/23
0/56
S18
Llama-3.2-1B
0.008
11
F
55.2
0
0/45
0/14
0/45
S18
Llama-3.2-1B
0.008
43
F
87.1
12
0/46
0/16
0/46
Appendix
Table 7: Earlier conditions (post hoc).
Figure 4: Share of eligible facts forgotten in at least one test run, in bins of R/a , over all 112 conditions (facts with a>0 ). Bars show the share, whiskers the Wilson 95% interval, and labels the number of forgotten facts over the number of facts in the bin.
U
min. events
p=0.1
p=0.2
p=0.3
p=0.5
p at power 0.8
0.05
8
0.01
0.30
0.79
1.00
0.304
0.01
4
0.40
0.91
0.99
1.00
0.166
0.001
2
0.84
0.99
1.00
1.00
0.091
Appendix
Table 8: Power of the per-fact test with 32 test runs and a Bonferroni correction over 64 facts: the smallest number of forgetting events that is significant against U , the probability of reaching it for true forgetting probabilities p , and the smallest p at which this probability reaches 0.8.
Figure 5: Fraction of eligible facts certified by the simplified Freedman bound, the complete Freedman bound and the Azuma bound in each condition set of the main, horizon and 14B studies and in the second confirmatory study, with the number of conditions with a violation (right).
Set
lr
seed
branch
z~
events
simplified
Azuma
screened
A
0.012
71
F
173.7
0
0/32
0/13
0/32
A
0.012
73
F
56.4
50
1/33
0/4
1/33
A
0.012
79
F
45.4
242
9/31
0/1
9/31
A
0.018
71
F
122.5
0
0/32
0/2
0/32
A
0.018
73
F
37.8
0
0/32
0/3
0/32
A
0.018
79
F
69.9
267
0/17
0/0
0/17
Appendix
Table 9: Main prospective study, every condition ( T=40 ).
Model
lr
seed
branch 40/80
z~
events
simplified
Azuma
screened
screened at 40
Qwen3-0.6B
0.012
71
F/F
136.6
0
0/32
0/6
0/32
32
Qwen3-0.6B
0.012
73
F/F
51.9
0
0/32
0/4
0/32
33
Qwen3-0.6B
0.018
71
F/F
124.6
0
0/32
0/0
0/32
32
Qwen3-0.6B
0.018
73
F/F
39.6
0
0/32
0/0
0/32
32
Llama-3.2-1B
0.012
71
F/A
26.9
30
0/27
0/6
0/6
30
Llama-3.2-1B
0.012
73
F/F
50.6
0
0/32
0/7
0/32
31
Appendix
Table 10: Horizon study ( T=80 ). The branch is given for the paired condition of the main study ( T=40 ) and for T=80 . Entries are significant violations over certified facts at T=80 ; the last column is the number of facts that the screened bound certifies at T=40 .
Study
lr
seed
branch
z~
events
simplified
Azuma
screened
22
0.008
71
excluded (task A)
22
0.012
71
excluded (task A)
22
0.008
73
F
121.2
14
0/47
0/40
0/47
22
0.012
73
F
32.6
30
0/44
0/16
0/44
23
0.008
79
F
31.8
11
0/54
0/28
0/54
23
0.012
79
A
18.4
65
0/37
0/16
0/16
Appendix
Table 11: Qwen3-14B study ( T=40 ), two preregistered batches (Studies 22 and 23). Conditions below the task-A accuracy threshold are marked as excluded. Entries are violating facts over certified facts.
lr
seed
branch
z~
events
simplified
Azuma
screened
0.01
79
F
47.8
0
0/46
0/20
0/46
0.01
83
F
50.7
8
0/28
0/12
0/28
Appendix
Table 12: Added Qwen3-8B batch ( T=40 , Study 24). Both Qwen3-8B conditions of the main study were below the task-A accuracy threshold (Table 9 ); this batch forms the Qwen3-8B row of Table 1 . Entries are violating facts over certified facts.
model
lr
seed
events
simplified
complete
Azuma
per-time
simplified, R<a
R≥a
Qwen3-0.6B
0.012
89
0
0/32
0/18
0/0
0/29
0/26
0
Qwen3-0.6B
0.012
97
0
0/32
0/5
0/2
0/32
0/8
0
Qwen3-0.6B
0.012
101
0
0/31
0/19
0/0
0/30
0/26
0
Qwen3-0.6B
0.012
103
20
0/35
0/18
0/4
0/31
0/22
0
Qwen3-0.6B
0.012
107
36
0/28
0/8
0/0
0/28
0/15
0
Qwen3-0.6B
0.012
109
65
0/26
0/6
0/7
0/23
0/12
0
Appendix
Table 13: Second confirmatory study ( T=40 , Study 25). Entries are violating facts over certified facts for the simplified, complete and Azuma bounds, the per-time bound, and the simplified bound restricted to facts with R<a . The last column is the number of facts violated by the simplified bound that have R≥a .
Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. We seek to understand when such forgetting is not catastrophic. A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network. Finetuning moves all old representations along a common direction, hiding the old facts while preserving their relative geometry; normalization withdraws this shift once the new facts are learned, whereas fact-specific changes accumulate and cause the erosion. Moreover, subtracting the common shift eliminates the collapse in a Transformer trained on synthetic data, and removing a single direction from each weight update restores old facts in a pretrained language model. Forgetting thus combines a shared, reversible loss of access with a slow erosion of individual facts, and only the second is catastrophic. Which one dominates depends on whether the new data move old memories together or apart.
Vedant Palit, Florent Draye, Nicolas Zucchet +2
MPI for Intelligent Systems, Tübingen · Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI +3
Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting.
Updating a language model's knowledge through fine-tuning is essential for keeping its outputs current, yet can also induce factual forgetting and new hallucinations. Prior work shows that long-tail knowledge is harder to acquire and newly memorized long-tail facts are difficult to retain during later fine-tuning. We study a complementary question: among facts that a model has encoded correctly, which are most vulnerable to collateral corruption during other updates? To investigate this question under a realistic factual distribution, we construct a large-scale graph FACTPROP of verified Wikipedia facts by linking triples that share head or tail entities, thereby preserving connections among factual knowledge. We fine-tune models on factual statements and measure correct-to-incorrect facts after each update. Our results reveal a pattern distinct from prior findings on long-tail vulnerability during acquisition and retention: among facts that models already answer correctly, those associated with highly connected entities are more likely to be corrupted by neighboring updates, and updates to such facts propagate errors more broadly. Structural popularity therefore predicts both vulnerability and downstream damage. Inspired by this finding, we propose Popularity-based Anchoring (PopAnchor), a lightweight rehearsal strategy that preserves a small set of popular facts and reduces forgetting.
Yuji Zhang, Weibing Wang, Cheng Qian +5
University of Illinois Urbana-Champaign · City University of New York · Massachusetts Institute of Technology +1