When Can First-Order Models of Fine-Tuning Bound Forgetting?

Jianchang Su; Wei Zhang

cs.AISep 26, 2026

When Can First-Order Models of Fine-Tuning Bound Forgetting?

Authors: Jianchang Su, Wei Zhang

Organizations: University of Connecticut

Abstract

Fine-tuning a language model on new data can make it forget facts that it should keep. We ask whether measurements taken at the start of a fine-tuning run can bound, for each protected fact, the probability that the run makes the model forget it. In LoRA fine-tuning with stochastic gradient descent on models from 0.6B to 14B parameters, a first-order response model estimated by finite-difference probes predicts changes of per-fact margins with correlation 0.974-0.998. Predictions of forgetting built on this model nevertheless failed, because forgetting requires parameter changes far outside the region in which the model was validated. The probes can, however, bound the probability that a margin first falls below a boundary near zero: we derive Freedman and Azuma first-passage bounds for a linear surrogate of the margin and test on new runs whether they hold for the model. The bounds contain a term R that measures how much the response coefficients change during the run. The simplified Freedman bound, which sets R = 0, certified most facts but was violated in 14 of 112 conditions, and every fact on which it was violated had R >= a, where a is the distance of the fact's margin to the boundary. The complete Freedman bound certifies only facts with R < a, and it held in every condition. On the violated facts, the spread of the margin across test runs was a median of 14.6 times the prediction of the response model, so the failures are breakdowns of the model, and in our data they occurred only where R >= a. We found this pattern post hoc and tested it in two preregistered confirmatory studies with 43 new conditions: the complete bound held in all of them, and the simplified bound failed there on only 3 facts, each with R >= a. First-order models of fine-tuning can thus bound forgetting on the facts whose response coefficients change by less than their distance to the boundary.

Figures & tables

Panel of Figure 1 — Figure 1: The procedure for one condition (schematic). (1) A reference run gives the margins $m_{i}(t)$ , and probes along it give the response coefficients $q_{i,t,s}$ and the gradient covariance $\Sigma_{s}$ . (2) These determine, for each fact, the distance $a_{i}$ to the boundary, the variance estimate $V_{i}$ , the increment bound $c_{i}$ and the coefficient change $R_{i}$ , the gap between the coefficients at time $t$ and at the end $T$ . (3) The complete bound uses $a_{i}-R_{i}$ , so it certifies only facts with $R_{i}<a_{i}$ (Sections 4 and 5 ). (4) A binomial test compares the forgetting events in the test runs with $U_{i}$ .

Figure 2: (a) The 3,748 facts that the simplified Freedman bound certifies, by their distance

a

to the boundary and coefficient change

R

, both computed before the run. Red: facts on which the simplified bound was violated, in earlier (open) and prospective (filled) conditions; all lie in the shaded region

R\geq a

, which the complete bound excludes. (b) Share of eligible facts forgotten in at least one test run, by

R/a

, with Wilson 95% intervals (Appendix H ). (c) For the four bounds, the fraction of eligible facts certified against the fraction of conditions with a violation, on the 41 earlier (open, post hoc) and 71 prospective (filled) conditions.

			Simplified		Screened		Complete		Azuma		Per-fact		Margin
			Freedman		bound		Freedman		bound		rule		threshold
Condition set	$n$	events	viol.	%	viol.	%	viol.	%	viol.	%	viol.	%	viol.	%
Earlier conditions (post hoc)	41	1,795	7	80	0	64	0	31	0	22	0	37	0	14
Qwen3-0.6B, synthetic	9	603	2	85	2	85	0	29	0	8	0	53	0	6
Llama-3.2-1B, synthetic	6	319	1	71	0	47	0	18	0	18	0	21	0	29
Qwen3-1.7B, synthetic	4	337	1	87	1	49	0	6	0	12	0	11	0	6

Table 1: Main results. For each condition set and rule: the number of violating conditions (viol.) and the percentage of eligible facts that the rule certifies (%).

n

is the number of evaluable conditions, and events is the number of forgetting events of eligible facts over all test runs. The rows of condition sets contain the main, horizon and 14B studies; the Qwen3-8B row contains the added batch (seeds 79 and 83). The confirmatory set contains the conditions of these studies whose test runs started after the confirmatory test was defined.

Rule	certified (%)	viol. cond.	fact-runs	events	allowed
Simplified Freedman ( $R=0$ )	81.0	14	118,928	344	467.5
Screened bound	67.2	6	98,480	202	314.8
Per-time Freedman	74.5	5	109,264	188	513.4
Complete Freedman	23.8	0	34,256	0	130.8
Azuma	17.3	0	24,688	0	178.8
Simplified Freedman on facts with $R<a$	31.0	0	44,912	0	29.8

Table 2: Forgetting events pooled over all 112 conditions (post hoc). For each rule: the percentage of eligible facts certified, the number of conditions with a violation, the pairs of a certified fact and a test run, the forgetting events among them, and the events that the values of the rule allow. The last two rows certify only facts with

R<a

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Checkpoint	Family	$n$	$\rho$	slope
Qwen3-0.6B	Qwen	4	0.985–0.994	0.901–1.047
Llama-3.2-1B	Llama	3	0.974–0.998	0.974–1.028
Qwen3-1.7B	Qwen	4	0.982–0.996	0.986–1.075
Qwen3-8B	Qwen	1	0.994	0.987
Gemma-3-12B	Gemma	2	0.992–0.997	1.003–1.010
Qwen3-14B	Qwen	1	0.998	1.014

Appendix

Table 3: First-order response model at task-B learning rate 0.002. A validation run passes a preregistered threshold if the correlation

\rho

between predicted and measured margin changes under held-out perturbations is at least 0.90 and the slope lies in

[0.8,1.2]

. Every run passes;

n

is the number of runs, and ranges are over runs.

Prediction	Independent test	Outcome
Per-fact event probabilities from the response model and a Gaussian noise model	fresh split of facts	mean absolute error 0.060 against a threshold of 0.05; all errors are false alarms
Rotating injected noise of fixed spectrum changes the forgetting rate by 0.145 between the extreme orientations	32 paired runs, spectra matched to $10^{-14}$	difference 0.032, 95% interval $[-0.043,0.105]$ , excludes 0.145
A drift statistic of the response coefficients ranks facts by prediction error	fresh set of 128 facts	mean rank correlation falls from 0.639 to 0.075
Two histories with the same margin vector have the same future distribution	branching from matched states	energy-distance permutation test, $p=0.00035$
A rank-2 projection of the protected gradients predicts how futures diverge	runs branched from matched states	predicts how much futures diverge (Spearman 0.658, against 0.050 for a random projection) but misses the direction

Appendix

Table 4: Five predictions built on the response model, each preregistered with a success criterion and tested on data disjoint from the data that suggested it. The outcome column gives the result that missed the criterion: an error above the threshold, an effect outside the predicted interval, a correlation that fell on replication, a rejected sufficiency hypothesis, and a prediction of the size of the divergence that missed its direction.

Threshold on $\tilde{z}$	violating cond.	certified
0 (always simplified)	8	79.6%
10	8	79.6%
15	6	77.1%
20	2	67.2%
25	1	66.2%
30	0	62.3%

Appendix

Table 5: Post-hoc sweeps on the earlier conditions. Left: the screening threshold. Right: the simplified Freedman bound with its variance estimate inflated by

\kappa

(the default is 2).

Figure 3: Median standardized margin

\tilde{z}

of the facts that the simplified Freedman bound certifies, computed before the run, against the number of facts on which that bound is significantly violated (with small vertical jitter). Hollow: earlier conditions (post hoc); filled: all 71 prospective conditions. The screen selects the Freedman branch to the right of the dashed line.

Set	bound range	facts	mean bound	event freq.	facts	mean bound	event freq.
		simplified Freedman bound			screened bound
Earlier	$[0,0.001)$	1140	0.0001	0.0014	1009	0.0001	0.0000
	$[0.001,0.01)$	270	0.0041	0.0038	151	0.0039	0.0000
	$[0.01,0.05)$	196	0.0253	0.0110	96	0.0240	0.0020
	$[0.05,0.2)$	179	0.1057	0.0239	83	0.1085	0.0041
	$[0.2,0.5)$	91	0.3324	0.0797	67	0.3510	0.0420

Appendix

Table 6: Mean bound and frequency of forgetting events, grouped by the value of the bound, for all 45 earlier conditions and for the main and horizon studies (the added 8B and 14B batches and the complete bound are covered by Table 1 ).

Study	Model	lr	seed	branch	$\tilde{z}$	events	simplified	Azuma	screened
S18	Llama-3.2-1B	0.002	11	F	164.9	0	0/60	0/47	0/60
S6	Llama-3.2-1B	0.002	11	F	183.3	0	0/64	0/64	0/64
S13	Llama-3.2-1B	0.006	11	F	57.1	56	0/58	0/28	0/58
S13	Llama-3.2-1B	0.006	11	F	52.0	16	0/56	0/23	0/56
S18	Llama-3.2-1B	0.008	11	F	55.2	0	0/45	0/14	0/45
S18	Llama-3.2-1B	0.008	43	F	87.1	12	0/46	0/16	0/46

Appendix

Table 7: Earlier conditions (post hoc).

Figure 4: Share of eligible facts forgotten in at least one test run, in bins of

R/a

, over all 112 conditions (facts with

a>0

). Bars show the share, whiskers the Wilson 95% interval, and labels the number of forgotten facts over the number of facts in the bin.

$U$	min. events	$p=0.1$	$p=0.2$	$p=0.3$	$p=0.5$	$p$ at power 0.8
0.05	8	0.01	0.30	0.79	1.00	0.304
0.01	4	0.40	0.91	0.99	1.00	0.166
0.001	2	0.84	0.99	1.00	1.00	0.091

Appendix

Table 8: Power of the per-fact test with 32 test runs and a Bonferroni correction over 64 facts: the smallest number of forgetting events that is significant against

U

, the probability of reaching it for true forgetting probabilities

p

, and the smallest

p

at which this probability reaches 0.8.

Figure 5: Fraction of eligible facts certified by the simplified Freedman bound, the complete Freedman bound and the Azuma bound in each condition set of the main, horizon and 14B studies and in the second confirmatory study, with the number of conditions with a violation (right).

Set	lr	seed	branch	$\tilde{z}$	events	simplified	Azuma	screened
A	0.012	71	F	173.7	0	0/32	0/13	0/32
A	0.012	73	F	56.4	50	1/33	0/4	1/33
A	0.012	79	F	45.4	242	9/31	0/1	9/31
A	0.018	71	F	122.5	0	0/32	0/2	0/32
A	0.018	73	F	37.8	0	0/32	0/3	0/32
A	0.018	79	F	69.9	267	0/17	0/0	0/17

Appendix

Table 9: Main prospective study, every condition (

T=40

Model	lr	seed	branch 40/80	$\tilde{z}$	events	simplified	Azuma	screened	screened at 40
Qwen3-0.6B	0.012	71	F/F	136.6	0	0/32	0/6	0/32	32
Qwen3-0.6B	0.012	73	F/F	51.9	0	0/32	0/4	0/32	33
Qwen3-0.6B	0.018	71	F/F	124.6	0	0/32	0/0	0/32	32
Qwen3-0.6B	0.018	73	F/F	39.6	0	0/32	0/0	0/32	32
Llama-3.2-1B	0.012	71	F/A	26.9	30	0/27	0/6	0/6	30
Llama-3.2-1B	0.012	73	F/F	50.6	0	0/32	0/7	0/32	31

Appendix

Table 10: Horizon study (

T=80

). The branch is given for the paired condition of the main study (

T=40

) and for

T=80

. Entries are significant violations over certified facts at

T=80

; the last column is the number of facts that the screened bound certifies at

T=40

Study	lr	seed	branch	$\tilde{z}$	events	simplified	Azuma	screened
22	0.008	71	excluded (task A)
22	0.012	71	excluded (task A)
22	0.008	73	F	121.2	14	0/47	0/40	0/47
22	0.012	73	F	32.6	30	0/44	0/16	0/44
23	0.008	79	F	31.8	11	0/54	0/28	0/54
23	0.012	79	A	18.4	65	0/37	0/16	0/16

Appendix

Table 11: Qwen3-14B study (

T=40

), two preregistered batches (Studies 22 and 23). Conditions below the task-A accuracy threshold are marked as excluded. Entries are violating facts over certified facts.

lr	seed	branch	$\tilde{z}$	events	simplified	Azuma	screened
0.01	79	F	47.8	0	0/46	0/20	0/46
0.01	83	F	50.7	8	0/28	0/12	0/28

Appendix

Table 12: Added Qwen3-8B batch (

T=40

, Study 24). Both Qwen3-8B conditions of the main study were below the task-A accuracy threshold (Table 9 ); this batch forms the Qwen3-8B row of Table 1 . Entries are violating facts over certified facts.

model	lr	seed	events	simplified	complete	Azuma	per-time	simplified, $R<a$
Qwen3-0.6B	0.012	89	0	0/32	0/18	0/0	0/29	0/26
Qwen3-0.6B	0.012	97	0	0/32	0/5	0/2	0/32	0/8
Qwen3-0.6B	0.012	101	0	0/31	0/19	0/0	0/30	0/26
Qwen3-0.6B	0.012	103	20	0/35	0/18	0/4	0/31	0/22
Qwen3-0.6B	0.012	107	36	0/28	0/8	0/0	0/28	0/15
Qwen3-0.6B	0.012	109	65	0/26	0/6	0/7	0/23	0/12

Appendix

Table 13: Second confirmatory study (

T=40

, Study 25). Entries are violating facts over certified facts for the simplified, complete and Azuma bounds, the per-time bound, and the simplified bound restricted to facts with

R<a

. The last column is the number of facts violated by the simplified bound that have

R\geq a

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 758 papers

When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
Oct 6, 2026Vedant Palit, Florent Draye, Nicolas Zucchet +2Memorization in Language Models LLM Fine-Tuning
MPI for Intelligent Systems, Tübingen · Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI +3
Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay
May 25, 2026Martin Marek, Dongkyu Cho, Shikai Qiu +3Continual Learning for LLMs LLM Fine-Tuning
New York University
Popular Knowledge Propagates More Errors in LLM Knowledge Updating
Sep 8, 2026Yuji Zhang, Weibing Wang, Cheng Qian +5Knowledge Editing Factual Knowledge in Language Models
University of Illinois Urbana-Champaign · City University of New York · Massachusetts Institute of Technology +1

Oct 6, 2026J/K move · Enter open · S save

When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting

Vedant Palit, Florent Draye, Nicolas Zucchet, Zhijing Jin+1

MPI for Intelligent Systems, Tübingen · Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI+3

Memorization in Language Models LLM Fine-Tuning Catastrophic Forgetting

Open paper AskPDF arXiv

Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. We seek to understand when such forgetting is not catastrophic. A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network. Finetuning moves all old representations along a common direction, hiding the old facts while preserving their relative geometry; normalization withdraws this shift once the new facts are learned, whereas fact-specific changes accumulate and cause the erosion. Moreover, subtracting the common shift eliminates the collapse in a Transformer trained on synthetic data, and removing a single direction from each weight update restores old facts in a pretrained language model. Forgetting thus combines a shared, reversible loss of access with a slow erosion of individual facts, and only the second is catastrophic. Which one dominates depends on whether the new data move old memories together or apart.

Abstract

LLM Fine-Tuning

Explore similar work

When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting

Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

Popular Knowledge Propagates More Errors in LLM Knowledge Updating