A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning

Yuyang Zhang; Xiaoyin Chen; Chunlin Ren; Qihuang Zhang

cs.AIOct 3, 2026

A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning

Authors: Yuyang Zhang, Xiaoyin Chen, Chunlin Ren, Qihuang Zhang

Organizations: McGill University Mila – Quebec AI Institute · Université de Montréal · Nanyang Technological University

Abstract

Fine-tuning a language model on new text degrades what it already does. Replay-free projectors such as Adam-NSCL and GPM forbid one shared subspace of a layer's inputs in every row of the update. The tropical geometry of a ReLU layer shows why this is too coarse. In data space, the units' walls are tropical hypersurfaces whose cells are dual to the upper vertices of a zonotope; in weight space, each old token is a hyperplane, and the tokens cut out a polyhedron, the closure of the weights that keep every token on its side. An exact identity joins the two pictures: the squared change of the layer's output under any weight change splits into in-cell, open-to-closed and closed-to-open terms, and the first two live on the tokens each unit fires on. The identity names a gate-aware per-unit projector, and a budget-separation theorem prices exact protection: it costs a unit the rank of its own open tokens, while a shared subspace pays at least the rank of their union in every row. On OPT-1.3b, where 96% of (token, unit) pairs are closed, the projector forgets less than Adam-NSCL at all six matched budgets from 9 to 60 constrained directions per row, the gap widening from $1.1\times$ to $4.3\times$ ; with 1/5.5 of the directions it halves the forgetting of Adam-NSCL at GPM's energy threshold. On OPT-6.7b, it matches Adam-NSCL's forgetting at matched budget while learning more. As the theory predicts, the open/closed partition is the operative variable: open tokens beat random, sign-blind and anti-gate token sets on 18 of 18 seed-pairs. In pruning repair, every derivative-based local model of the output error at the dense weights is blind to pairs that open: the minimisers of the gate-weighted objective can leave the polyhedron, the objective's closed-form solution is 1.94 nats worse than no repair on OPT-1.3b, and a convex one-sided penalty bounds the escape.

Figures & tables

Figure 1: A layer’s weights, its lifted Newton polytope, and its activation pattern , for a toy layer of three ReLU units on two inputs. Pale blue marks weight space and (c), (d) are data space; arrows marked dual join the two. (a) Each colour is a unit and each vertex a set of units that fire, in bits (

101

: units 1 and 3), placed at their summed weights and raised by their summed biases: the Newton polytope of

\sum_{i}\max(0,z_{i})

lifted by the biases ( Zhang et al., 2018 ) (Appendix A ). (b) Its shadow, which subdivides the Newton polytope; the circled vertex is unit 1’s weights

w

. (c) The units’ walls

z_{i}=0

in data space. Regions are labelled by which units fire, each is a vertex of (b), and pattern

110

(hollow in (a)) occurs nowhere. (d) Fine-tuning moves unit 1’s wall from

w

\hat{w}

, and colours say what that does to each old token: green stays open, blue closes, red opens, grey stays closed (the sets

S_{++},S_{+-},S_{-+},S_{--}

of Theorem 5 ). (e) The same move in weight space:

w

is a point and each old token a line, crossing which flips the token (dark: open at

w

; light: closed;

D

’s line is off-panel). The grey region

\mathcal{P}_{\mathcal{X}}

is the closure of the weights that flip no token (Definition 3 ). (f) Our projector (§ 3 ) fixes the pre-activation on the open tokens

A

and

C

, so

w

moves only parallel to their lines (dotted) and the closed token

E

may open; circles are the new-task loss. Biases are fixed in (d)–(f), and

A

C

lie on one ray, so one direction stays free.

Panel of Figure 2 — Figure 2: Forgetting at matched constrained dimensions. Means over seeds 0–2 $\pm$ one standard error, WikiText-103, lr $10^{-4}$ ; dashed: Adam-NSCL at a uniform $259$ dims/row, the mean dimension GPM’s energy criterion selects at $0.95$ .

other selector	$\Delta$ forget	CI	verdict
random tokens (rank-matched)	$-0.0317$	$[-0.0436,-0.0197]$	gate better, $p=0.0005$
sign-blind $\|z-\theta\|$	$-0.0575$	$[-0.0756,-0.0393]$	gate better, $p=0.0002$
anti-gate (most closed)	$-0.0660$	$[-0.0799,-0.0520]$	gate better, $p<10^{-4}$
first-order $\|\partial L/\partial z\|$	$+0.0024$	$[-0.0011,+0.0059]$	equivalent
random within the open set	$+0.0015$	$[-0.0021,+0.0051]$	equivalent

Table 1: Identical per-unit dimension, different tokens. OPT-1.3b, WikiText-103,

\bar{k}=23.5

n=6

(

n=3

for random-within-open). Gate minus the other selector. Superiority: one-sided paired

t

, 95% CI; equivalence: TOST ( Schuirmann, 1987 ) at

\delta=0.0068

nats,

2.5\%

of unconstrained forgetting, 90% CI.

Panel of Figure 3 — Figure 3: Method families on the forgetting–learning plane. OPT-1.3b, WikiText-103, means over seeds 0–2. Axes reversed: up and right is better. Each marker is one setting of its family’s own hyperparameter. Grey steps: the Pareto frontier of the plotted points, whose low-forgetting end is our caps $512$ and $256$ . Settings, and five points off the plotted range, each learning less than unconstrained fine-tuning, are in Appendix F.1 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: A ReLU layer as a tropical polynomial, for one input. The toy layer has three units on a scalar input

x

, with

z_{1}=x-1.2

z_{2}=-0.8x-0.8

and

z_{3}=0.6x-0.12

; each unit keeps its colour from Figure 1 (unit 1 purple, unit 2 amber, unit 3 navy). (a) One unit. The grey line is the monomial

0

and the dashed purple line the monomial

z_{1}

. Their maximum, the thick line, is the ReLU

0\oplus z_{1}=\max(0,z_{1})

. The two monomials tie at

x=1.2

, the unit’s wall: to its left the unit outputs

0

, to its right it outputs

z_{1}

. (b) The whole layer. Adding the three outputs is tropical multiplication,

\bigodot_{i}(0\oplus z_{i})

, and multiplying out gives one line

\sum_{i\in J}z_{i}

for each of the

8

subsets

J

of units (Eq. ( 5 )). The thick envelope is their maximum, which is the layer’s summed output. Each piece of the envelope is labelled by the subset that attains it, written as a bit string in which bit

i

1

when unit

i

is in

J

, that is, open. Reading left to right: at

x=-1.7

only unit 2 fires, so the winning line is

z_{2}

and the output is

0.56

(pattern

010

); between

x=-1.0

and

x=0.2

no unit fires (

000

); between

0.2

and

1.2

only unit 3 fires (

001

); beyond

1.2

units 1 and 3 fire, and at

x=1.8

the output is

z_{1}+z_{3}=1.56

(

101

). The coloured dots are the three walls, where the envelope bends. The four dashed lines never reach the envelope. Pattern

100

, for example, would need unit 1 open and unit 3 closed, that is

x>1.2

and

x<0.2

at once. (c) The same layer in exponent space. Each subset

J

becomes the point

(\sum_{i\in J}w_{i},\sum_{i\in J}b_{i})

000

is the origin,

001=(0.6,-0.12)

101=(1.6,-1.32)

, and so on. The upper hull of the eight points, the roof, has as its vertices exactly the four patterns that win in (b). Each roof edge is one unit’s row. The navy edge from

000

001

is unit 3’s row

(0.6,-0.12)

, and its slope

-0.2

is minus unit 3’s wall at

x=0.2

. In the same way the amber edge has slope

1

, minus unit 2’s wall at

-1.0

, and the purple edge has slope

-1.2

, minus unit 1’s wall at

1.2

. The hollow points lie below the roof and never win:

100=(1.0,-1.2)

, for example, sits

0.6

below the roof point above it. The dashed edges bound the polygon from below and carry no winner. Dropping the height (dotted) projects the roof onto the weight axis and splits it into three pieces, one per unit, which is the one-input counterpart of the subdivision in Figure 1 b. The arrow marked dual records that each roof vertex is a region of (b) and each roof edge a wall.

Figure 5: Theorem 5 , one token at a time. The four old tokens of Figure 1 d under the update of unit 1 from

w

\hat{w}

. Each panel draws the activation

\operatorname{act}(z)=0\oplus z

against the pre-activation; shading marks the open side of the wall (dotted). The hollow dot is the token’s pre-activation

z

w

, the filled dot its pre-activation

\hat{z}

\hat{w}

, and the bar on the right is the change in the unit’s output. The term under each panel is what that token adds to Eq. ( 2 ), and the words under the formula name the tokens each sum runs over.

A

stays open, moving from

z=1.50

\hat{z}=0.38

; its output falls by

1.12

, and it contributes the in-cell error

(\hat{z}-z)^{2}=1.26

C

closes, moving from

0.45

-0.20

; its output falls from

0.45

0

, whatever

\hat{z}

is below the wall, so it contributes its old output squared,

z^{2}=0.20

E

opens, moving from

-0.40

0.60

; its old output was

0

, so it contributes its new output squared,

\hat{z}^{\,2}=0.36

D

moves from

-2.17

-1.83

but stays closed, so its output is

0

before and after and it contributes nothing, however far it moves. The three terms add to

1.82

, which is the total squared change of the unit’s output over the four tokens, with no remainder.

run	constraint	dims/row	co mass	$\Delta$ mass	$\Delta$ share
B1	none	$0$	$9\,239$	—	—
B17c16	gate, $C=16$	$9.1$	$2\,286$	$-6\,953$	$-0.0537$
B17c64	gate, $C=64$	$23.5$	$1\,785$	$-7\,454$	$-0.0433$
B17c128	gate, $C=128$	$34.4$	$1\,695$	$-7\,544$	$-0.0366$
B16k9	shared, $k_{\mathrm{sh}}=9$	$9.0$	$2\,196$	$-7\,043$	$-0.0772$
B16k24	shared, $k_{\mathrm{sh}}=24$	$24.0$	$1\,622$	$-7\,617$	$-0.0910$

Appendix

Table 2: Closed

\to

open mass on the held-out old probe. OPT-1.3b, WikiText-103 at lr

10^{-4}

, fc1 , seeds

0

–

2

. “co mass” is the run’s mean level; the difference columns are mean paired per-seed differences against the unconstrained run, negative on

3

3

seeds in every row. “Share” is of the probe’s total squared post-activation change.

				forgetting				learning
$C$	$\bar{k}$	$k_{\mathrm{sh}}$	$n$	$\Delta$	95% CI	$p$	$\Delta<0$	$\Delta$	95% CI	$p$
OPT-1.3b (gate density $0.0392$ , ambient dim. $2\,049$ ; no constraint forgets $+0.2684$ )
$16$	$9.067$	$9$	$6$	$-0.0068$	$[-0.0246,+0.0109]$	$0.1839$	$4/6$	$+0.0011$	$[-0.0011,+0.0033]$	$0.2691$
$32$	$15.048$	$15$	$3$	$-0.0027$	$[-0.0126,+0.0073]$	$0.1845$	$2/3$	$+0.0011$	$[-0.0082,+0.0104]$	$0.6679$
$64$	$23.519$	$24$	$6$	$\mathbf{-0.0111}$	$[-0.0147,-0.0074]$	$0.0003$	$6/6$	$+0.0015$	$[-0.0002,+0.0033]$	$0.0769$
$128$	$34.421$	$34$	$6$	$\mathbf{-0.0195}$	$[-0.0247,-0.0143]$	$0.0001$	$6/6$	$+0.0025$	$[-0.0002,+0.0051]$	$0.0654$

Appendix

Table 3: Matched constrained dimensions per row. Ours at cap

C

(mean per-unit dimension

\bar{k}

) against Adam-NSCL at

k_{\mathrm{sh}}

, paired by seed;

\Delta

is ours minus Adam-NSCL, so a negative

\Delta

is ours ahead. Forgetting: one-sided paired

t

-test; learning: two-sided;

\Delta<0

counts the seed-pairs on which ours forgets less. Bold: the 95% CI excludes zero with ours ahead. WikiText-103, fc1 , lr

10^{-4}

T=2\,048

; each block has its own no-constraint run, and the pooled lines combine the rows named.

					$\mathrm{open}_{i}$ over units
model	$p$	ambient	$\overline{\mathrm{open}}$	$p\,T$	median	p25	p99	never open	mean/median
OPT-1.3b	$0.0392$	$2\,049$	$72.1$	$80$	$11$	$1$	$954$	$19.5\%$	$6.6$
OPT-125m	$0.0606$	$769$	$111.3$	$124$	$16$	$2$	$1\,320$	$14.8\%$	$7.0$

Appendix

Table 4: The reachable budget: base gate density and per-unit open sets.

p

is the fraction of (token, unit) pairs open at the base weights on the

1\,024

-token old probe.

\mathrm{open}_{i}=|O_{i}|

is counted directly on the

T=2\,048

-token support that builds the projector; its mean

\overline{\mathrm{open}}

over units is the right-hand side of Eq. ( 6 ) and bounds

\bar{k}

(

p\,T

is about

11\%

higher because

p

is measured on the probe). “Ambient” is the row dimension

d+1

Figure 6: Which (token, unit) pairs a constraint holds, at one budget. A random ReLU layer with $d=30$ inputs, $m=10$ units and $T=24$ old tokens. How to read a grid. Each row is one unit, that is one row of the layer’s first weight matrix ( fc1 in OPT), and each column is one old token, so the cell in row $i$ and column $t$ is the pair $(t,i)$ . Rows are sorted by how many tokens the unit fires on, most active on top. Columns are sorted by how much the shared subspace of (b) holds them, most held on the left. The order is the same in every grid, so a given cell sits at the same place in (a)–(d). (a) The gate pattern of the base model, read off one forward pass over the old tokens: a cell is dark when unit $i$ is open on token $t$ , that is $z_{i}(x_{t})>\theta$ , with $\theta$ the gate threshold ( $0$ for ReLU). The bar to the right of a row counts that unit’s open tokens. The top unit fires on $8$ of the $24$ tokens, the next on $6$ , and the bottom unit on none. Only $38$ of the $240$ pairs, $16\%$ , are open; at OPT-1.3b the share is $4\%$ . (b)–(d) A cell’s shade is the fraction of an isotropic weight update’s effect on that pair’s pre-activation that the constraint removes: white means the pair is left free, and full colour means it is held exactly. The dots repeat the open pairs of (a). The bar to the right of a row is the number of directions the constraint forbids in that row, and the dashed line marks the mean, which is $3$ for every family. The bar under a grid is the mean shade over the open pairs. Holding a closed pair guards only against its opening. (b) The shared subspace of Adam-NSCL, GPM, InfLoRA and AlphaEdit forbids the top $3$ eigendirections of the old inputs’ second moment, and it forbids them in every row. Every row therefore spends $3$ directions, and a cell’s shade depends only on its column: the leftmost token is held at $83\%$ in every row and the rightmost at $8\%$ . The top unit’s $8$ open tokens are held at between $9\%$ and $78\%$ . Overall $56\%$ of the open pairs’ effect is held. (c) Methods that protect whole units freeze entire rows; HAT and AGS-CL choose them by old-task use, and MIGU by new-data magnitude. A frozen row spends all $30$ of its directions (arrow), so a mean of $3$ buys one frozen row out of ten. The top row is frozen. All $24$ of its tokens are held, including the $16$ on which it is closed, and no other row holds anything, so $21\%$ of the open pairs’ effect is held. (d) Our projector gives unit $i$ the budget $k_{i}=\min(C,\#\text{open})$ with cap $C=4$ , and spends it on that unit’s own open tokens with the largest activations. The top unit therefore holds its $4$ most active open tokens exactly, and its other $4$ open tokens only in part, at $54\%$ , $22\%$ , $14\%$ and $10\%$ . A unit with $3$ open tokens holds all $3$ exactly, and the bottom unit, which never fires, receives $k_{i}=0$ and holds nothing. The budgets, top to bottom, are $4,4,4,4,4,3,3,3,1,0$ , which also average $3$ , and $85\%$ of the open pairs’ effect is held. (e) The selector controls of § 4.2 , drawn for the top unit. Its $24$ old tokens are sorted by pre-activation $z$ , shown by the bars with the open tokens on the left, and the dotted line is the wall. Eight are open, and its budget is $k_{i}=4$ . Each row marks the $4$ tokens that one selector spends the budget on. The gate takes the $4$ most active open tokens. Random-within-open takes $4$ of the $8$ open tokens at random, here sharing one with the gate. The first-order criterion takes the $4$ tokens with the largest $|\partial L/\partial z|$ ; since that derivative is zero on closed tokens, it also stays inside the open set. Random tokens draws $4$ of all $24$ , and here lands on $1$ open and $3$ closed tokens. The sign-blind criterion takes the $4$ largest $|z-\theta|$ , which are the $2$ most open and the $2$ most deeply closed tokens, and the anti-gate takes the $4$ most deeply closed. Every row spends the same $k_{i}$ ; only the tokens change.

freezing setting	matched to	dims/row	$n$	forget, learn	$\Delta$ forget ( $p$ )	$\Delta$ learn ( $p$ )
top $1.15\%$ of units	cap $64$ ( $23.5$ )	$23.51$	$6$	$+0.2101$ , $-0.5851$	$-0.1855$ ( $6/6$ , $<10^{-4}$ )	$-0.0389$ ( $6/6$ , $<10^{-4}$ )
top $1.68\%$ of units	cap $128$ ( $34.4$ )	$34.52$	$3$	$+0.1951$ , $-0.5853$	$-0.1713$ ( $3/3$ , $0.0014$ )	$-0.0357$ ( $3/3$ , $0.027$ )
$59.6\%$ of active energy	cap $64$ (energy)	$69.34$	$3$	$+0.1719$ , $-0.5871$	$-0.1380$ ( $3/3$ , $0.0006$ )	$-0.0367$ ( $3/3$ , $0.054$ )

Appendix

Table 6: Freezing whole units against the per-unit projector. OPT-1.3b, WikiText-103, fc1 , lr

10^{-4}

\Delta

is ours minus the freezing run, paired by seed; negative means ours forgets less (one-sided

p

) or learns more (two-sided

p

); the fraction counts seed pairs with

\Delta<0

family	setting	lr	forget	learn	$\Delta$ forget ( $p$ )	$\Delta$ learn ( $p$ )
none	—	$10^{-4}$	$+0.2684$	$-0.5878$	$-0.2447$ ( $0.0018$ )	$-0.0332$ ( $0.0436$ )
EWC	$\lambda=4$	$10^{-4}$	$+0.2695$	$-0.5850$	$-0.2457$ ( $0.0007$ )	$-0.0360$ ( $0.0261$ )
EWC	$\lambda=40$	$10^{-4}$	$+0.2743$	$-0.5863$	$-0.2505$ ( $0.0001$ )	$-0.0347$ ( $0.0118$ )
EWC	$\lambda=400$	$10^{-4}$	$+0.2578$	$-0.5863$	$-0.2341$ ( $0.0004$ )	$-0.0347$ ( $0.0051$ )
EWC	$\lambda=4\times 10^{3}$	$10^{-4}$	$+0.2092$	$-0.5962$	$-0.1854$ ( $<10^{-4}$ )	$-0.0248$ ( $0.0047$ )
EWC	$\lambda=4\times 10^{4}$	$10^{-4}$	$+0.0937$	$-0.6144$	$-0.0700$ ( $0.0190$ )	$-0.0066$ ( $0.3399$ )

Appendix

Table 7: Method families on fc1 . OPT-1.3b, WikiText-103, seeds

0

–

2

, forgetting and learning as in Appendix C .

\Delta

is our cap

128

minus the row, paired by seed; negative means ours forgets less (one-sided

p

) or learns more (two-sided

p

method	dims/row	forget	learn	learn	gain over none (seeds)
		end of training		WiSE-FT, zero forgetting
no constraint	—	$+0.2684$	$-0.5878$	$-0.5904$	—
ours, $C=64$	23.5	$+0.0339$	$-0.6237$	$-0.6289$	$-0.0385$ ( $3/3$ )
ours, $C=128$	34.4	$+0.0238$	$-0.6210$	$-0.6262$	$-0.0359$ ( $3/3$ )
Adam-NSCL, $k_{\mathrm{sh}}=24$	24	$+0.0468$	$-0.6252$	$-0.6215$	$-0.0311$ ( $3/3$ )
Adam-NSCL, $k_{\mathrm{sh}}=34$	34	$+0.0462$	$-0.6229$	$-0.6178$	$-0.0275$ ( $3/3$ )

Appendix

Table 8: WiSE-FT on each method. OPT-1.3b, WikiText-103, seeds

0

–

2

. End of training is

\alpha=1

; zero forgetting is the learning each method’s WiSE-FT curve reaches where the old-probe NLL returns to its base value, and its gain is paired by seed against WiSE-FT on unconstrained fine-tuning (negative = learns more; seeds on which it is negative).

method	LAMBADA	HellaSwag	ARC-e	Winogrande	SciQ	mean drop
base	58.9	53.7	51.3	59.1	78.2	—
no constraint	53.7	51.1	48.8	58.6	74.4	2.94
Adam-NSCL, $k_{\mathrm{sh}}=34$	56.1	52.6	51.3	58.5	77.3	1.08
ours, $C=128$ ( $\bar{k}=34.4$ )	57.3	52.6	50.8	58.1	77.4	1.02
Adam-NSCL, GPM energy $0.95$ ( $259$ )	57.0	53.2	51.4	58.4	76.9	0.85

Appendix

Table 9: Zero-shot accuracy after one fine-tuning transition. OPT-1.3b fine-tuned on WikiText-103 ( fc1 , lr

10^{-4}

), mean over seeds

0

–

2

, in %. Accuracy for LAMBADA (OpenAI variant) and Winogrande, length-normalised accuracy for the others. Mean drop is the base model’s mean accuracy over the five tasks minus the run’s. Cap

128

against no constraint:

+1.92

points of mean accuracy, positive on

3

3

seeds, one-sided paired

p=0.0125

(

n=3

Figure 7: What each repair objective charges for, on open and closed tokens. Four objectives are compared, as in the legend: the pre-activation error

\sum_{x}(\hat{z}-z)^{2}

, which SparseGPT and GPTQ fit; the Gauss–Newton and Fisher models of the post-activation error, which reduce to the in-cell error on the open tokens,

L_{\mathrm{gated}}

(Lemma 13 ); the exact post-activation error

L_{\mathrm{post}}

; and ours,

L_{\mathrm{gated\_h}}

, which adds to that in-cell error a penalty of weight

\lambda

on closed tokens that open, the one-sided penalty of Theorem 15 . (a) The price each objective charges one token that is open at

w

, here with

z=1

, as its new pre-activation

\hat{z}

moves; shading marks the open side of the wall. The pre-activation, curvature and our objectives charge the same parabola

(\hat{z}-z)^{2}

, drawn as three coincident curves. The exact error agrees with them while the token stays open, and stops growing once it closes. A move to

\hat{z}=-1

costs

4

under the three quadratics but only

1

under the exact error, because the unit’s output cannot fall below

0

. (b) The same for a token closed at

w

, here with

z=-1

. The pre-activation objective still charges

(\hat{z}-z)^{2}

, so a move to

\hat{z}=-2

, deeper into the closed side and with no effect on the output, costs

1

. The curvature models charge nothing anywhere, even after the token opens. The exact error charges only opening,

\max(\hat{z},0)^{2}

, and ours charges

\lambda

times that, with

\lambda=3

here. A move to

\hat{z}=1

thus costs

4

under pre-activation,

0

under curvature,

1

under the exact error and

3

under ours. (c) What each objective does to a pruned row. Take unit 1 of Figure 1 , with

w=(1,0.4)

, old tokens

A

and

C

open and

E

and

D

closed. Prune

w_{1,1}

0

and let each objective refit

w_{1,2}

alone. The horizontal axis is the refitted

w_{1,2}

, and the curve is the exact error along it (log scale). The dotted lines mark where

E

opens (

w_{1,2}=0.50

) and where

A

and

C

open again (

1.65

and

2.85

). No point of this line keeps every gate:

A

and

C

need

w_{1,2}>2.85

, while

E

needs

w_{1,2}\leq 0.50

. In this toy no refit therefore improves on leaving the pruned row alone at

w_{1,2}=0.4

, where

A

C

and

E

are all closed and the exact error is

2.45

; the objectives differ in how much they lose. The curvature fit sees only

A

and

C

and restores both exactly, by moving to

w_{1,2}=4.13

. That opens

E

z_{E}=7.26

, and the exact error there is

52.8

22\times

that of no repair. The pre-activation fit, at

0.81

, keeps

A

and

C

closed and opens

E

0.61

, for an exact error of

2.83

. Ours at

\lambda=10

stops at

0.54

, just past

E

’s wall with

z_{E}=0.09

, and its exact error of

2.46

equals that of no repair to within

1\%

. The inset, on a linear scale, enlarges the three fits next to

E

’s wall.

model	activation	block	$\theta$	$n_{\mathrm{wall}}$	$s$	mask-only density
OPT-125m ( Zhang et al., 2022 )	ReLU	plain FFN	$0$	12	$0.7$	$0.0589$
OPT-350m	ReLU	plain FFN	$0$	24	$0.7$	$0.0848$
OPT-1.3b	ReLU	plain FFN	$0$	24	$0.7$	$0.0352$
Pythia-160m ( Biderman et al., 2023 )	GELU	plain FFN	$0$	12	$0.7$	—
Pythia-410m	GELU	plain FFN	$0$	24	$0.5$	$0.1428$
MiniCPM-S-1B-sft ( Song et al., 2025 )	FATReLU	SwiGLU	$0.03$	52	$0.5$	$0.1899$

Appendix

Table 10: Models. The wall matrix is the one whose output passes through the activation and which the repair modifies;

\theta

is recovered at load time by bisection on the resolved act_fn ; the density column is the mask-only gate density at sparsity

s

(Pythia-160m has none). Activations: ReLU, GELU ( Hendrycks & Gimpel, 2016 ) and the thresholded FATReLU ( Kurtz et al., 2020 ) ; SwiGLU blocks ( Shazeer, 2020 ) .

family	models	$s$	cells	max. identity gap
OPT	OPT-125m, OPT-350m, OPT-1.3b	$0.1$ – $0.9$	$27$	$5.80\times 10^{-5}$
7B ReLU family	ReluLLaMA-7B, ProSparse-7B	$0.1$ – $0.9$	$18$	$9.02\times 10^{-5}$

Appendix

Table 11: Relative gap of the case split behind Eq. ( 2 ) on pruned layers (mask alone): the summed absolute first-order residual against the summed

|z_{\hat{w}}|

of the pairs that change cell, maximum over the cells of each family.

$s$	no repair	$L_{\mathrm{pre}}$	$L_{\mathrm{gated}}$	$L_{\mathrm{post}}$	$L_{\mathrm{gated\_h}}(30)$	$L_{\mathrm{post\_w}}(10)$
$0.5$	$19.76$	$20.76$	$20.41$	$20.54$	$\mathbf{20.34}$	$20.54$
$0.6$	$29.37$	$34.33$	$31.56$	$31.46$	$\mathbf{30.04}$	$30.91$
$0.7$	$97.25$	$153.64$	$100.66$	$100.07$	$\mathbf{84.50}$	$94.12$
$0.8$	$2643.77$	$2334.70$	$879.56$	$\mathbf{740.52}$	$902.77$	$776.50$

Appendix

Table 12: WikiText-2 perplexity at context 2048, OPT-1.3b, dense

=14.62

. One frozen mask per row; only the functional changes.

L_{\mathrm{post}}<L_{\mathrm{pre}}

at every sparsity, and at

3

of the

4

by more than the WikiText-2 noise floor

0.0150

nats.

	residual share			flips
objective	in-cell	o $\to$ c	c $\to$ o	o $\to$ c	c $\to$ o
$L_{\mathrm{gated}}$	$.602$	$.027$	$\mathbf{.372}$	$.0018$	$.0115$
$L_{\mathrm{gated\_h}}(1)$	$.767$	$.061$	$.171$	$.0027$	$.0070$
$L_{\mathrm{post}}$	$.747$	$.143$	$.111$	$.0044$	$.0046$
$L_{\mathrm{pre}}$	$.805$	$.137$	$.058$	$.0052$	$.0034$
$L_{\mathrm{post\_w}}(10)$	$.724$	$.224$	$\mathbf{.053}$	$.0071$	$.0025$

Appendix

Table 13: Shares of the post-activation residual by term, and the fraction of

(t,i)

pairs crossing in each direction, OPT-1.3b

s=0.7

, averaged over the 24 wall matrices.

model	activation	$\lambda=1$	best $\lambda>1$	attained at	seed 2 gap	other groups
MiniCPM-S-1B, $s{=}0.5$	SwiGLU $+$ FATReLU	$3.4636$	$3.4594$	$L_{\mathrm{post\_wv}}(10)$	—	$+0.0043$
OPT-1.3b, $s{=}0.5$	ReLU	$3.6847$	$3.6805$	$L_{\mathrm{post\_w}}(10)$	—	—
OPT-1.3b, $s{=}0.6$	ReLU	$4.0404$	$\mathbf{3.9915}$	$L_{\mathrm{gated\_h}}(30)$	$+0.0437$	$+0.0283$ to $+0.0489$
OPT-1.3b, $s{=}0.7$	ReLU	$4.9442$	$\mathbf{4.7631}$	$L_{\mathrm{gated\_h}}(30)$	$+0.1715$	$+0.1072$ to $+0.1811$
OPT-1.3b, $s{=}0.8$	ReLU	$6.4546$	$6.4381$	$L_{\mathrm{post\_w}}(3)$	—	$+0.0062$ to $+0.0165$
OPT-125m, $s{=}0.7$	ReLU	$5.6261$	$5.6121$	$L_{\mathrm{post\_w}}(10)$	$+0.0086$	—

Appendix

Table 14: Held-out probe loss of the best

\lambda>1

member of an objective family against its

\lambda=1

member, sharing configuration, no-repair anchor, calibration seed and budgets. Bold : the gap exceeds

3\times

the seed-noise floor (Appendix I ) and a second calibration seed reproduces its sign; “seed 2 gap” is the gap of the same

\lambda

at that seed, “other groups” the gap in the other comparability groups of the configuration.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

Oct 7, 2026

A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks

Continual pre-training and fine-tuning in Large Language Models (LLMs) inevitably induce catastrophic forgetting, typically mitigated by replay using often-inaccessible original data. In this data-free regime, we analyze where forgetting occurs and why. Systematic parameter freezing across five settings up to 1.4B reveals that forgetting concentrates selectively in the output embeddings of tokens rarely seen in the new corpus, whereas the same sqrt(v-hat) band of the body is inert and new learning resides elsewhere. This localization is governed by the vocabulary deficiency of the corpus rather than the training mode, allowing pre-retraining risk ranking from token counts alone within a fixed base model. Mechanistically, absent tokens receive persistent one-sided softmax gradients that Adam's second-moment (sqrt(v-hat)) normalization amplifies into full-sized updates. We therefore propose an intervention: raising Adam's epsilon exclusively for the output projection during training. Across eight settings spanning 160M to 12B parameters and four model families, this removes 39.4% to 67.9% of forgetting across all seven stable configurations without degrading target learning or requiring per-model tuning. The defense combines additively or better with replay (79.8% on Qwen/Korean) and rescues released-head LoRA from a 23-fold forgetting surge. Because post-hoc editing of the drifted rows recovers under 5% of forgetting, the intervention must operate during training. Our findings indicate that a single-line optimizer adjustment may serve as the primary defense against catastrophic forgetting where the corpus starves the vocabulary.

Jonghyun Han, Younghoon Song, Jongyoul Park

Seoul National University of Science and Technology · Korea Institute of Land & Infrastructure Safety Technology

Catastrophic Forgetting Continual Learning for LLMs LLM Fine-Tuning

Ask PDF

May 25, 2026

Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

Models trained on a new task typically degrade on prior tasks, a phenomenon known as forgetting. Traditionally, mitigating forgetting has required replaying stored exemplars from prior tasks, which is often impractical. By contrast, language models can sample from their own training distribution, and we show that these self-generated samples serve as effective replay data, nearly eliminating forgetting. We find that forgetting nonetheless persists when the model has little remaining capacity: models pretrained close to saturation cannot absorb new information without overwriting prior knowledge. When capacity is not the limiting factor, low learning rates reduce forgetting but require substantially more training steps. Replay breaks this tradeoff, enabling fast, high-learning-rate finetuning without forgetting.

Martin Marek, Dongkyu Cho, Shikai Qiu +3

New York University

Continual Learning for LLMs LLM Fine-Tuning Replay-Based Continual Learning Catastrophic Forgetting Fine-Tuning Language Models

Ask PDF

Oct 6, 2026

When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting

Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can even produce forgetting that undoes itself: recall of the old facts collapses, recovers as training continues on new facts alone, and only then erodes for good. We seek to understand when such forgetting is not catastrophic. A minimal associative memory reproduces these dynamics with three ingredients: keys with shared structure, concentrated new values, and normalization in the network. Finetuning moves all old representations along a common direction, hiding the old facts while preserving their relative geometry; normalization withdraws this shift once the new facts are learned, whereas fact-specific changes accumulate and cause the erosion. Moreover, subtracting the common shift eliminates the collapse in a Transformer trained on synthetic data, and removing a single direction from each weight update restores old facts in a pretrained language model. Forgetting thus combines a shared, reversible loss of access with a slow erosion of individual facts, and only the second is catastrophic. Which one dominates depends on whether the new data move old memories together or apart.

Vedant Palit, Florent Draye, Nicolas Zucchet +2

MPI for Intelligent Systems, Tübingen · Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI +3

Memorization in Language Models LLM Fine-Tuning Catastrophic Forgetting

Ask PDF

Abstract

LLM Fine-Tuning

Explore similar work

A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks

Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replay

When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting