TRAP: Understanding and Mitigating Privacy Memorization in Language Models

Muhammed Ustaomeroglu; Ziyue Xu; Hanshen Xiao; Peter Cnudde; Guannan Qu; Holger R. Roth

cs.CLSep 26, 2026

TRAP: Understanding and Mitigating Privacy Memorization in Language Models

Authors: Muhammed Ustaomeroglu, Ziyue Xu, Hanshen Xiao, Peter Cnudde, Guannan Qu, Holger R. Roth

Organizations: Carnegie Mellon University · NVIDIA · Purdue University

Abstract

Fine-tuning a language model on sensitive records can leave it able to reproduce them. We ask when this memorization arises and how to prevent it without knowing in advance which spans are sensitive. Our starting point is that most memorization scores and attacks share one statistical core: whether the model assigns a token more probability than some reference would. Taking as the reference a model trained on the complementary half of the same corpus gives the Target Reference Advantage (TRA), a per-token signal that separates what a model fit to a particular record from what it learned across records, and is cheap and differentiable. We then study what drives memorization during fine-tuning: it keeps growing well past the validation minimum, is larger on small datasets and at higher learning rates, and higher when the underlying task is harder. Early stopping removes much of it, but because it is chosen by aggregate validation loss it helps least for rare, hard-to-predict spans embedded in otherwise learnable text, which is exactly what sensitive information tends to be. We therefore introduce TRAP, a one-sided penalty on tokenwise TRA that acts only where the target model pulls ahead of its reference. On student essays with annotated personal information and clinical cases with patient identifiers, TRAP brings memorization near the level of an untrained model at little utility cost, where generic regularizers barely move and differential privacy gives up most of what fine-tuning bought.

Figures & tables

Figure 1: What moves memorization. All panels report

\operatorname{nAUC}

on the sensitive spans, averaged over scores, pooling Pythia-70M to 1B ( MedCase to 1.4B) and seeds, except (d), which reports raw TRA; the less averaged versions are in Appendix D . (a) Each configuration trained twice, best-val (horizontal) against overtrained (vertical): a point above the diagonal memorized more when trained longer. (b) Memorization along training, one median curve per testbed, with the epoch axis rescaled so that

1

is each run’s best-val epoch. (c) APEX -L: against the rule level that generates the secret; shading marks levels never learned. (d) A rare minority that can only be memorized: APEX -FavNum examples, whose answer is a random number absent from the text, at

1

10\%

of a

2000

-example APEX training set (blue), and the same minority trained alone (grey); TRA on the minority’s secrets in nats per token at best-val, pooling Pythia-70M and 160M. (e) Against learning rate at best-val. (f) MedCase (green) and essays (orange) along training at three learning rates each (lighter colour is smaller), against the training loss reached, on a log axis from about four (left) to a hundredth (right). (g) Against the number of training examples. (h) Against model size with the recipe fixed: MedCase in both regimes, essays at best-val for three learning rates (darker colour is larger). In all panels, bars and bands show the spread over what was pooled (interquartile range in (b), full range elsewhere).

Figure 2: Memorization against utility. Held-out cross-entropy (horizontal) against the worst-case

\operatorname{nAUC}

over the scores each run was evaluated with (vertical); lower is better on both. Each line is a method family’s lower-left frontier over its hyperparameter values (seed means where seeds exist): TRAP pools its frozen and iterated references, TRAP + LoRA is the same penalty through an adapter, DP the two optimizers, and the grey line pools the remaining baselines into one frontier (drawn one by one in Figures 29 , 30 and 34 ); for the baselines and DP the remaining hyperparameter values are the faint dots. Hollow markers with dashed lines are the same families trained on

\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}

; the star is the undefended model, the hollow star its union-trained twin, the green dot and dotted line the untrained model. Points beyond

2.5\times

the undefended cross-entropy are left out and counted in the axis label; the untrained model is drawn at the right edge when it lies beyond. (a) Essays at Pythia-1B. (b) MedCase at Pythia-1B. (c) The rare-minority mixture of Section 3.1 at Pythia-160M.

Appendix figures & tables52 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: One training example from each testbed. Colour marks what the loss covers: blue is context the model reads but is not scored on, green the instruction and any prefilled prefix, and amber the completion, the only part the cross-entropy and the TRAP penalty see. An essay is amber throughout, since training predicts every token of it, so the annotated sensitive spans (here the closing name, email and phone) fall inside the loss. In APEX the applicant’s fields are scattered over five sections and

519

words, among one to three other applicants, and the three scored fields (email, phone, address) appear verbatim in the dossier; in APEX -L the target is instead a portal number computed from six visible ingredients by a rule the dossier never states. MedCase is trained open book, with the blue and green blocks as its prompt, and evaluated closed book, with the green block alone as the prompt: with the case report removed, the synthetic patient identifier is the only evidence left, so a correct diagnosis has to come from a memorized identifier-to-diagnosis association. Excerpts are abridged where marked, and the essay’s sensitive values are shown as placeholders.

experiment	format	models	training-set sizes
learning-rate sweep	APEX	Pythia-70M, 160M	$200$ , $2198$
training-set size sweep	APEX	Pythia-70M, 160M	$50$ to $2198$
per-epoch trajectories, hardness	APEX -L	Pythia-70M, 160M, 1B	$1000$
mixture ladder (minority)	APEX -L	Pythia-70M, 160M	$20$ in $2000$
rarity mixture (minority)	APEX -FavNum	Pythia-70M, 160M	$20$ to $200$ in $2000$
hardness-by-size grid (appendix)	APEX -full	Pythia-160M	$20$ to $2198$

Appendix

Table 1: APEX formats by experiment.

Level	candidate_portal_id	interview_slot_code
L1 small lookup	P $\,\\|\,$ RoleCode $[\mathrm{role}]\,\\|\,m$ (10-entry table)	DayLetter $[\mathrm{day}]\,\\|$ first letters of src, role (5-entry table)
L2 per-symbol shift	P $\,\\|\,(m_{1}{+}3)\,\\|\,(m_{2}{+}7)\,\\|\,(q{+}3)\,\\|\,(a{+}7)$ , each mod 10	$L(\mathrm{role}_{1}{+}5)\,L(\mathrm{src}_{1}{+}11)\,L(\mathrm{day}_{1}{+}17)$
L3 pairwise sums	P $\,\\|\,(m{+}q)\bmod 100\,\\|\,(a{+}b{+}c{+}d{+}\mathrm{src}_{\mathrm{idx}})\bmod 100$	$L(\mathrm{role}_{\mathrm{idx}}{+}q)$ $L(\mathrm{src}_{\mathrm{idx}}{+}a{+}b)$ $L(\mathrm{day}{+}\mathrm{part}{+}\mathrm{mode}{+}c{+}d)$
L4 3-term linear-mod	P $\,\\|\,(167m+1103q+1667\,\mathrm{src}_{\mathrm{idx}})\bmod 10^{4}$	$L(3\,\mathrm{role}_{\mathrm{idx}}{+}5q)$ $L(7m{+}\mathrm{src}_{\mathrm{idx}})$ $L(3a{+}5b{+}7c{+}d)$
L5 6-term linear-mod	P $\,\\|\,(1009\,\mathrm{role}_{\mathrm{idx}}+331q+97m+53\,\mathrm{src}_{\mathrm{idx}}+17(a{+}b{+}c{+}d)+7\,\mathrm{day}_{\mathrm{idx}}+3\,\mathrm{part}_{\mathrm{idx}}+\mathrm{mode}_{\mathrm{idx}})\bmod 10^{4}$	analogous six-ingredient mod-26 letter construction

Appendix

Table 2: The five rules of APEX -L. Both secret fields are deterministic functions of the same six visible ingredients, and the rule is never stated in the dossier.

m

: submission minute (digits

m_{1}m_{2}

);

q

: recruiter queue;

a,b,c,d

: assessment sub-scores; role/src/day/part/mode: categorical ingredients (

\mathrm{idx}

= integer code);

L(i)

: the

i

-th letter of the alphabet, mod 26;

\|

: string concatenation.

Figure 4: The matched pairs of Figure 1 a, one panel per score. Each source contributes the scores it was scored with: the APEX -full grid probability and loss++ (there Min-K%++ with

K=20\%

, everywhere else

K=100\%

), the MedCase and per-epoch runs all four. For the 1B ladder, TRA uses the last epoch with a matched reference. The number of pairs above the diagonal is printed in each panel.

metric	best-val	overtrained	worse when overtrained
probability ratio on the secret (geometric mean)	$2.6\times$	$111.6\times$	$32/35$
exposure (bits)	$2.36$	$3.60$	$33/35$
TRA on the secret (nats)	$0.84$	$4.03$	$33/35$
top-1 rate among decoys	$0.080$	$0.271$	$32/35$
probability AUC on the secret span	$0.730$	$0.878$	$33/35$
probability AUC on all tokens	$0.452$	$0.649$	$24/35$

Appendix

Table 3: The APEX -full hardness-by-size grid,

35

matched configurations, means over them; the probability ratio is member over unseen. “Worse” counts the configurations in which the overtrained arm scores higher.

Figure 5: The per-epoch runs behind Figure 1 b (

60

in all: essays

7

, MedCase

18

, APEX -L

35

), one column per score, seed means, with a dot at the epoch of the validation minimum. (a) Essays : Pythia-70M at three learning rates (solid; three seeds at

10^{-4}

) and Pythia-1B (dashed; two seeds); exposure was scored only at the recipe rate. (b) MedCase : Pythia-70M (solid) and 160M (dashed) at three rates, three seeds each, scored from the first optimizer step (every step of the first epoch, every second step of the next two, then every epoch to

70

), so the epoch axis starts inside epoch one. (c) to (e) APEX -L at Pythia-70M and 160M (three seeds per level) and 1B (one seed), coloured by rule level; at 70M and 160M exposure comes from the two seeds that scored it along the run. For 1B, TRA stops at epoch

38

, the last epoch at which the reference had been trained to the same point.

Figure 6: Training-set size, one column per score:

\operatorname{nAUC}

against the number of training examples at the best-val checkpoint (solid, filled) and overtrained (dashed, hollow). APEX : Pythia-70M and 160M, seed means. MedCase : averaged over the Pythia scales run at each size (five scales at

200

and

2000

cases, Pythia-1B alone at

60

and

600

). Essays : Pythia-70M and 160M, seed means. MedCase ’s

20

-case rung, which rests on

20

spans, and the full essays corpus are left out. The largest APEX point is the full training set from the learning-rate sweep at the recipe rate.

Figure 7: Worst-case

\operatorname{nAUC}

over the four scores against the number of training examples in

\mathcal{D}_{\mathrm{tgt}}

on the cluster runs ( essays at Pythia-70M, MedCase at Pythia-1B), at the best-val checkpoint (solid, filled) and overtrained (dashed, hollow).

Figure 8: Training-set size, one line per model scale:

\operatorname{nAUC}

averaged over scores against the number of training examples, at the best-val checkpoint (solid, filled) and overtrained (dashed, hollow). Essays and APEX are local reruns at best-val with the recipe fixed at every size, each scored against a reference of the same size, seed means over three seeds at Pythia-70M and 160M; MedCase is one seed per scale from the size ladder, with Pythia-1B the only scale run at

60

and

600

cases. On the essays the non-member set is the same

48

spans at every size while the member set shrinks to

13

spans at

100

documents, so the smallest rung carries little evidence.

Figure 9: Size marginal of the APEX -full grid (Pythia-160M, two seeds), averaged over the five rule levels, in both regimes: the two scores the grid stores as

\operatorname{nAUC}

, exposure in bits, TRA in nats, and the geometric mean of the probability ratio on the secret. After the fixed budget the decline is monotone across all seven sizes on every metric (Spearman

\rho=-1.00

, exact

p=4\cdot 10^{-4}

Figure 10: Does a smaller training set stop later? The best-val epoch against training-set size for every size sweep with a recorded validation curve: the essays and APEX (Pythia-70M solid, 160M dashed; seed means), the APEX -full grid at its hardest rule (Pythia-160M, two seeds), and MedCase (thin lines: one per model scale and learning rate, from the size ladder and the local rate sweep; thick line: their median). Small training sets are not passed over more often before best-val on the essays or on APEX ; on MedCase they stop a few epochs later.

Figure 11: Task hardness on APEX -L in detail, behind Figure 1 c. (a)

\operatorname{nAUC}

against the rule level, one column per score, averaged over Pythia-70M, 160M and 1B (three seeds at the two smaller scales), at the best-val checkpoint (solid) and at the last scored epoch (dashed; for 1B the TRA endpoint is epoch

38

, the last epoch with a matched reference, while the other scores run to

150

). (b) Open-book accuracy on unseen applicants, the measure of whether the rule was learned, at the best-val checkpoint (solid) and the last scored epoch (dashed). (b, c) Bars span the three scales. (c) Best-val epoch. (d) Closed-book reproduction of the exact identifier for training applicants. (e) Every run at both checkpoints, learned against memorized. Shading marks the rules never learned.

Figure 12: APEX -L per scale, one row per model size ((a) Pythia-70M, (b) 160M, (c) 1B) and one column per score:

\operatorname{nAUC}

against the rule level at the validation minimum (solid) and at the last scored epoch (dashed); seed means with the seed range at 70M and 160M (three seeds). 1B is a single seed, and its TRA endpoint is epoch

38

Figure 13: The mechanism per scale on APEX -L: (a) open-book accuracy on unseen applicants, (b) the epoch of the validation minimum, (c) closed-book reproduction of the identifier for training applicants, (d) every run at both checkpoints. Bars span the seeds.

Figure 14: The 425-epoch campaign (Pythia-160M,

200

applicants, three seeds, level means), with the full-data 40-epoch runs at 160M and 1B for comparison. (a) Probability of the secret under the target model relative to its reference. (b) Open-book accuracy on unseen applicants. (c) Attacks on the derived fields after

425

epochs: AUC of the probability score against held-out applicants and against

\mathcal{D}_{\mathrm{ref}}

, its true-positive rate at

1\%

false positives, and the AUC on the three copied fields, which are identical at every level.

Figure 15: The mixture ladder (a

1\%

minority whose secret follows rule L1 to L5 inside a learnable majority; Pythia-70M and 160M, three seeds): (a) open-book accuracy on unseen applicants, (b) the epoch selected by validation, (c) open-book reproduction of the minority’s secrets by the target model and by its reference (dotted), (d) the probability ratio on the secret. Solid: best-val checkpoint; dashed, hollow: fixed ten epochs.

Figure 16: The rare-minority mixture per model scale; Pythia-70M (light) and 160M (dark), three seeds, bars the seed range. (a) Figure 1 d not pooled: TRA on the minority’s secrets in nats per token at the best-val checkpoint, inside the mixture (solid) and for the minority trained alone at the same count (dashed, hollow). (b) The best-val epoch of the same runs. (c) An APEX -L secret as the

1\%

minority instead of an impossible one:

\operatorname{nAUC}

averaged over scores by rule level, at the best-val checkpoint (solid) and after a fixed ten epochs that give every level the same number of passes (dashed, hollow).

Figure 17: The rarity runs in full; Pythia-70M (light) and 160M (dark), three seeds, bars the seed range. (a) The four scores as

\operatorname{nAUC}

. (b) The probability of the secret under the target relative to its reference. (c) The best-val epoch. (d) Exact extraction of the secret under full enumeration of the

10^{4}

candidates (dotted: the reference). (e) Greedy recall of the secret. Solid, filled: the minority inside the mixture; dashed, hollow: the minority trained alone at the same count.

Figure 18: Panel c of Figure 16 one score at a time: an APEX -L secret as the

1\%

minority,

\operatorname{nAUC}

on the minority by rule level at the best-val checkpoint (solid) and after a fixed ten epochs (dashed, hollow); Pythia-70M (light) and 160M (dark), three seeds, bars the seed range.

Figure 19: Minority memorization against the epoch selected by validation for every defense of the two cross-corpus mixtures at

1\%

share (Pythia-160M, one seed), with the majority from (a) APEX and (b) MEDEX . The vertical axis is the probability the target gives the minority’s secret relative to its reference; the dotted line is chance. Runs that train with small steps, through a lower learning rate or an adapter, run for many more epochs and stay near chance; the runs that memorize are the full-step ones. DP runs have no validation-selected epoch and are not shown, and cells that change the shared learning rate as a variant of another method are left out, as everywhere in the paper.

Figure 20: The learning-rate sweeps before averaging:

\operatorname{nAUC}

(mean over scores) at the best-val checkpoint, one line per model scale and training-set size (Pythia-70M to 1B,

20

3268

training examples), seed means with the seed range as bars; darker shades are larger models, markers distinguish training-set sizes, and dotted lines are the cluster sweep.

Figure 21: The learning-rate sweeps one score at a time, best-val checkpoint. Thin lines are the individual series (one model scale and training-set size, seed means), thick lines the mean over the series of a testbed at each rate.

Figure 22: The best-val epoch of every learning-rate series. Essays : the local sweep (solid 70M, dashed 160M) and the cluster sweep (pale). APEX :

200

applicants (circles) and

2198

(squares). MedCase : one line per scale and size, with the median.

Figure 23: Learning rate at matched fit: the runs scored along training that vary only the rate, each score against the training loss the run had reached at that point. (a, b) MedCase at Pythia-160M and 70M,

200

cases, three seeds per rate drawn separately, scored from the first optimizer step against the same-step reference (the step-level runs of Figure 5 b). (c) The essays at Pythia-70M on the full corpus, one seed at the smallest and largest rate and three at

10^{-4}

, drawn as separate lines; sixty epochs bring their training loss only to about a tenth. The middle rate is

5\cdot 10^{-5}

for MedCase and

10^{-4}

for the essays , and darker is a larger rate.

Figure 24: Held-out cross-entropy at the best-val checkpoint against the learning rate, same series and styles as Figure 22 ; the cluster sweep reports test cross-entropy, the local sweeps validation cross-entropy.

Figure 25: Training one example at a time.

\operatorname{nAUC}

averaged over the four scores against the learning rate, seed means with the range over three seeds, same tokenization, validation set and optimizer settings as the recipes, batch of one example, constant rate. “One example at a time” keeps stepping on an example while validation improves; “one step per example” takes exactly one; “strict” also undoes a first step that does not improve validation. Black is the batch recipe at its best-val checkpoint at the same rate, and the dotted grey line the batch recipe after seventy epochs.

Figure 26: Memorization against model size with the recipe held fixed,

\operatorname{nAUC}

averaged over scores. (a) MedCase at five Pythia sizes, one line per training-set size, at the best-val checkpoint (solid) and overtrained (dashed); one seed per configuration. (b) Essays at best-val at three sizes, one line per learning rate applied to all three (seed means from the cluster sweep); the dotted grey line is the scale-adapted series, which trains 70M and 160M at

10^{-4}

and 1B at

10^{-5}

. (c) APEX -L at three sizes, averaged over the five rule levels, at best-val and at the last scored epoch.

Figure 27: Panel h of Figure 1 one score at a time: MedCase with

20

cases at the best-val checkpoint (solid) and overtrained (dashed), and the essays at best-val at three fixed learning rates (darker is a larger rate).

Figure 28: Essays at Pythia-70M, every method on its own, best-val checkpoint. Lower-left is better.

Figure 29: Essays at Pythia-1B, every method on its own, best-val checkpoint.

Figure 30: MedCase at Pythia-1B, every method on its own, best-val checkpoint, seed means over three seeds; twenty member spans, so differences near the floor are directions rather than rankings. One cell is off scale, DP-AdamW at

\varepsilon=1

, at

5.9

nats against an undefended

2.28

Figure 31: MedCase at Llama-3.1-8B-Instruct, every method on its own, best-val checkpoint. The six cells off scale, beyond

2.5\times

the undefended cross-entropy of

1.62

nats, are all DP-AdamW:

\varepsilon=16

9.7

8

11.9

2

13.3

4

13.5

1

14.1

nats, and the

\varepsilon=8

twin trained on

\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}

9.1

Figure 32: MedCase at Pythia-1B trained for a fixed forty epochs instead of to the best-val epoch. One cell is off scale, DP-AdamW at

\varepsilon=1

, at

8.3

nats against an undefended

2.83

Figure 33: The rare-minority mixture at Pythia-70M, every method on its own, best-val checkpoint; seed means over three seeds, twenty minority secrets as the sensitive spans, so differences near the floor are directions rather than rankings. The undefended cross-entropy is

0.017

nats, so the

2.5\times

window leaves thirteen cells off scale: flooding at level

0.03

(

0.07

nats), DP-AdamW at

\varepsilon=16

on a lot of

1024

(

0.12

), KL anchor to the base model at

\lambda=0.5

(

0.15

), the

n

-gram regularizer at

\lambda=2

(

0.23

), DP-AdamW at

\varepsilon=64

(

0.24

) and at

\varepsilon=8

on a lot of

1024

(

0.32

), RelaxLoss at

\alpha=0.03

(

0.43

), DP-SGD at

\varepsilon=8

(

0.56

), DP-AdamW at

\varepsilon=8

\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}

(

0.94

) and on

\mathcal{D}_{\mathrm{tgt}}

(

1.15

), the confidence penalty at

\beta=0.5

(

1.59

), loss truncation at

0.2

(

3.39

), and DP-AdamW at

\varepsilon=1

(

10.0

Figure 34: The rare-minority mixture at Pythia-160M, every method on its own, best-val checkpoint; seed means over three seeds. The undefended cross-entropy is

0.012

nats; the thirteen cells off scale are flooding at level

0.03

(

0.03

nats), DP-AdamW at

\varepsilon=64

on a lot of

1024

(

0.05

), KL anchor to the base model at

\lambda=0.5

(

0.07

), the

n

-gram regularizer at

\lambda=2

(

0.15

), DP-AdamW at

\varepsilon=64

(

0.23

), DP-SGD at

\varepsilon=8

(

0.30

), DP-AdamW at

\varepsilon=16

and

8

on a lot of

1024

(

0.37

and

1.36

), DP-AdamW at

\varepsilon=8

\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}

(

1.07

) and on

\mathcal{D}_{\mathrm{tgt}}

(

1.46

), the confidence penalty at

\beta=0.5

(

1.53

), loss truncation at

0.2

(

4.06

), and DP-AdamW at

\varepsilon=1

(

13.7

Method	Settings on the frontier (cross-entropy, worst-case $\operatorname{nAUC}$ )
TRAP, frozen reference	$w{=}1,\ \epsilon{=}0.05$ ( 3.42, 0.44 ); $w{=}1,\ \epsilon{=}0.02$ ( 3.42, 0.43 ); $w{=}1,\ \epsilon{=}0$ ( 3.42, 0.42 ); $w{=}4,\ \epsilon{=}1$ ( 3.50, 0.20 ); $w{=}2,\ \epsilon{=}0.2$ ( 3.52, 0.16 ); $w{=}2,\ \epsilon{=}0.1$ ( 3.54, 0.16 ); $w{=}2,\ \epsilon{=}0.05$ ( 3.55, 0.12 ); $w{=}2,\ \epsilon{=}-0.05$ ( 3.59, 0.12 )
TRAP, iterated rounds	$w{=}0.9,\ \epsilon{=}0$ , round 2 (first round alternating) ( 3.39, 0.64 ); $w{=}1,\ \epsilon{=}0.02$ , round 2 ( 3.40, 0.46 ); $w{=}1,\ \epsilon{=}0.05$ , round 2 ( 3.41, 0.45 ); $w{=}1.25,\ \epsilon{=}0$ , round 3 ( 3.42, 0.26 ); $w{=}1.5,\ \epsilon{=}0$ , round 3 (first round alternating) ( 3.44, 0.19 ); $w{=}2,\ \epsilon{=}0$ , round 3 (first round alternating) ( 3.47, 0.18 ); $w{=}2,\ \epsilon{=}-0.05$ , round 3 (first round alternating) ( 3.49, 0.17 ); $w{=}3,\ \epsilon{=}0$ , round 2 ( 3.55, 0.16 )
TRAP + LoRA	$r{=}16,\ w{=}0.5,\ \epsilon{=}0$ ( 3.36, 0.23 ); $r{=}64,\ w{=}1,\ \epsilon{=}0$ ( 3.37, 0.17 ); $r{=}4,\ w{=}1,\ \epsilon{=}0.02$ ( 3.45, 0.16 ); $r{=}128,\ w{=}2,\ \epsilon{=}0$ ( 3.52, 0.13 )
TRAP + LoRA, iterated rounds	$r{=}16,\ w{=}0.9,\ \epsilon{=}0$ , round 2 ( 3.36, 0.17 ); $r{=}4,\ w{=}0.5,\ \epsilon{=}0$ , round 2 ( 3.40, 0.17 ); $r{=}8,\ w{=}0.9,\ \epsilon{=}0$ , round 1 ( 3.41, 0.16 ); $r{=}4,\ w{=}1.1,\ \epsilon{=}0$ , round 1 ( 3.46, 0.16 ); $r{=}16,\ w{=}1.5,\ \epsilon{=}0$ , round 1 ( 3.47, 0.15 ); $r{=}64,\ w{=}2,\ \epsilon{=}0$ , round 1 ( 3.52, 0.14 ); $r{=}16,\ w{=}2,\ \epsilon{=}0$ , round 1 ( 3.54, 0.13 ); $r{=}64,\ w{=}3,\ \epsilon{=}0$ , round 1 ( 3.65, 0.13 )
LoRA	$r{=}64$ ( 3.36, 0.21 ); $r{=}4$ ( 3.42, 0.19 )
LoRA on $\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}$	$r{=}64$ ( 3.30, 0.15 ); $r{=}16$ ( 3.33, 0.14 )

Appendix

Table 4: Essays at Pythia-70M: the settings on each method’s lower-left frontier in Figure 28 , with the point’s held-out cross-entropy and worst-case

\operatorname{nAUC}

in gray. Every run uses the uniform recipe of Appendix C : learning rate

10^{-4}

, constant schedule with

200

warmup steps,

8

documents per step, block size

2048

, fp32, no weight decay, patience

4

on the validation split, a

60

-epoch ceiling; DP runs clip at

0.1

with

\delta{=}10^{-5}

and, unless the setting says otherwise, the same lot and learning rate; only the listed hyperparameter varies.

w

and

\epsilon

are the TRAP weight and threshold,

r

the adapter rank,

\varepsilon

under DP the privacy budget.

Method	Settings on the frontier (cross-entropy, worst-case $\operatorname{nAUC}$ )
TRAP, frozen reference	$w{=}1,\ \epsilon{=}0$ ( 2.65, 0.51 ); $w{=}1,\ \epsilon{=}0.05$ ( 2.66, 0.50 ); $w{=}1.5,\ \epsilon{=}0.1$ ( 2.66, 0.32 ); $w{=}1.5,\ \epsilon{=}0.05$ ( 2.66, 0.30 ); $w{=}1.5,\ \epsilon{=}0$ ( 2.67, 0.28 ); $w{=}2,\ \epsilon{=}0.1$ ( 2.67, 0.25 ); $w{=}2,\ \epsilon{=}0.05$ ( 2.68, 0.23 ); $w{=}2,\ \epsilon{=}0$ ( 2.68, 0.21 ); $w{=}2,\ \epsilon{=}-0.02$ ( 2.68, 0.19 ); $w{=}2,\ \epsilon{=}-0.05$ ( 2.70, 0.19 ); $w{=}2,\ \epsilon{=}0.05$ ( 2.87, 0.12 )
TRAP, iterated rounds	$w{=}0.5,\ \epsilon{=}0$ , round 2 (first round mutual) ( 2.65, 0.63 ); $w{=}0.5,\ \epsilon{=}0$ , round 2 (first round alternating) ( 2.65, 0.62 ); $w{=}1,\ \epsilon{=}0$ , round 2 (first round mutual) ( 2.65, 0.60 ); $w{=}1,\ \epsilon{=}0$ , round 2 (reference: undefended model) ( 2.65, 0.56 ); $w{=}0.6,\ \epsilon{=}0$ , round 2 ( 2.66, 0.56 ); $w{=}0.25,\ \epsilon{=}0$ , round 3 ( 2.67, 0.44 ); $w{=}2,\ \epsilon{=}0$ , round 2 ( 2.72, 0.21 )
TRAP + LoRA	$r{=}4,\ w{=}1,\ \epsilon{=}0$ ( 2.66, 0.55 ); $r{=}16,\ w{=}1.5,\ \epsilon{=}0$ ( 2.67, 0.26 ); $r{=}8,\ w{=}2,\ \epsilon{=}0$ ( 2.68, 0.21 ); $r{=}256,\ w{=}2,\ \epsilon{=}0$ ( 2.68, 0.21 ); $r{=}16,\ w{=}2,\ \epsilon{=}-0.02$ ( 2.69, 0.17 )
TRAP + LoRA, iterated rounds	$r{=}4,\ w{=}0.9,\ \epsilon{=}0$ , round 1 ( 2.65, 0.56 ); $r{=}4,\ w{=}0.9,\ \epsilon{=}0$ , round 2 ( 2.65, 0.51 ); $r{=}4,\ w{=}1.3,\ \epsilon{=}0$ , round 2 ( 2.66, 0.33 ); $r{=}16,\ w{=}1.5,\ \epsilon{=}0$ , round 1 ( 2.67, 0.26 )
LoRA	$r{=}256$ ( 2.68, 0.27 )
LoRA on $\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}$	$r{=}256$ ( 2.64, 0.40 ); $r{=}4$ ( 2.64, 0.37 ); $r{=}16$ ( 2.64, 0.36 )

Appendix

Table 5: Essays at Pythia-1B: the settings on each method’s lower-left frontier in Figure 29 , with the point’s held-out cross-entropy and worst-case

\operatorname{nAUC}

in gray. Every run uses the recipe of Table 4 at learning rate

10^{-5}

; DP runs list their lot

L

, epoch budget

T

and learning rate; only the listed hyperparameter varies.

w

and

\epsilon

are the TRAP weight and threshold,

r

the adapter rank,

\varepsilon

under DP the privacy budget.

Method	Settings on the frontier (cross-entropy, worst-case $\operatorname{nAUC}$ )
TRAP, frozen reference	$w{=}0.5,\ \epsilon{=}0$ ( 2.14, 0.76 ); $w{=}1,\ \epsilon{=}0$ ( 2.17, 0.76 ); $w{=}2,\ \epsilon{=}0$ ( 2.64, 0.38 ); $w{=}2,\ \epsilon{=}0.05$ ( 2.74, 0.33 ); $w{=}4,\ \epsilon{=}0$ ( 2.92, 0.24 )
TRAP, iterated rounds	$w{=}0.5,\ \epsilon{=}0$ , round 4 ( 2.12, 0.74 ); $w{=}1,\ \epsilon{=}0$ , round 5 ( 2.15, 0.67 ); $w{=}1,\ \epsilon{=}0$ , round 2 ( 2.16, 0.65 ); $w{=}1,\ \epsilon{=}0$ , round 3 ( 2.21, 0.59 ); $w{=}2,\ \epsilon{=}0$ , round 2 ( 2.65, 0.39 ); $w{=}2,\ \epsilon{=}0$ , round 4 ( 2.94, 0.26 ); $w{=}2,\ \epsilon{=}0$ , round 3 ( 3.04, 0.17 )
LoRA	$r{=}64$ ( 3.11, 0.24 ); $r{=}32$ ( 3.69, 0.14 ); $r{=}16$ ( 4.33, 0.10 ); $r{=}8$ ( 4.77, 0.10 ); $r{=}4$ ( 5.27, 0.10 )
LoRA on $\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}$	$r{=}8$ ( 4.32, 0.13 )
DP-AdamW	$\varepsilon{=}16$ (lot 256) ( 3.93, 0.18 ); $\varepsilon{=}8$ (lot 256) ( 4.57, 0.09 )
DP-SGD	$\varepsilon{=}1$ (lot 256) ( 5.63, 0.10 ); $\varepsilon{=}4$ (lot 256) ( 5.65, 0.09 ); $\varepsilon{=}2$ (lot 256) ( 5.65, 0.09 )

Appendix

Table 6: MedCase at Pythia-1B, twenty cases, best-val: the settings on each method’s lower-left frontier in Figure 30 , with the point’s held-out cross-entropy and worst-case

\operatorname{nAUC}

in gray. Every run uses learning rate

5\cdot 10^{-5}

16

cases per step, bf16,

3\%

warmup, no weight decay, a

15

-epoch ceiling with patience

4

on the validation split; LoRA adapters on the attention projections with

\alpha{=}2r

and dropout

0.1

; only the listed hyperparameter varies.

w

and

\epsilon

are the TRAP weight and threshold,

r

the adapter rank,

\varepsilon

under DP the privacy budget.

Method	Settings on the frontier (cross-entropy, worst-case $\operatorname{nAUC}$ )
TRAP, frozen reference	$w{=}1,\ \epsilon{=}0.05$ ( 1.40, 0.64 ); $w{=}1,\ \epsilon{=}0$ ( 1.40, 0.62 ); $w{=}2,\ \epsilon{=}0$ ( 1.71, 0.30 ); $w{=}32,\ \epsilon{=}0.05$ ( 2.89, 0.18 )
TRAP, iterated rounds	$w{=}2,\ \epsilon{=}0$ , round 3 ( 2.00, 0.26 ); $w{=}2,\ \epsilon{=}0$ , round 4 ( 2.06, 0.17 )
LoRA	$r{=}16$ ( 2.91, 0.20 )
LoRA on $\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}$	$r{=}16$ ( 2.38, 0.05 )
DP-AdamW	$\varepsilon{=}16$ (lot 256) ( 9.72, 0.26 ); $\varepsilon{=}8$ (lot 256) ( 11.88, 0.07 )
DP-SGD	$\varepsilon{=}1$ (lot 256) ( 2.95, 0.22 ); $\varepsilon{=}2$ (lot 256) ( 3.03, 0.19 )

Appendix

Table 7: MedCase at Llama-3.1-8B-Instruct, twenty cases, best-val: the settings on each method’s lower-left frontier in Figure 31 , with the point’s held-out cross-entropy and worst-case

\operatorname{nAUC}

in gray. Every run uses learning rate

2\cdot 10^{-5}

8

cases per step, bf16, the stopping rule of Table 6 ; the LoRA adapters (rank

16

\alpha{=}32

, dropout

0.05

) train at learning rate

10^{-4}

; only the listed hyperparameter varies.

w

and

\epsilon

are the TRAP weight and threshold,

r

the adapter rank,

\varepsilon

under DP the privacy budget.

Method	Settings on the frontier (cross-entropy, worst-case $\operatorname{nAUC}$ )
TRAP, frozen reference	$w{=}1,\ \epsilon{=}0.05$ ( 2.29, 0.78 ); $w{=}1,\ \epsilon{=}0$ ( 2.59, 0.71 ); $w{=}2,\ \epsilon{=}0.05$ ( 3.05, 0.36 )
LoRA	$r{=}8$ ( 3.32, 0.14 )
LoRA on $\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}$	$r{=}8$ ( 2.59, 0.24 )
DP-AdamW	$\varepsilon{=}16$ (lot 256) ( 5.28, 0.10 ); $\varepsilon{=}4$ (lot 256) ( 6.40, 0.05 )
DP-SGD	$\varepsilon{=}1$ (lot 256) ( 5.51, 0.04 )
DP-AdamW on $\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}$	$\varepsilon{=}8$ (lot 256) ( 4.88, 0.10 )

Appendix

Table 8: MedCase at Pythia-1B, twenty cases, overtrained: the settings on each method’s lower-left frontier in Figure 32 , with the point’s held-out cross-entropy and worst-case

\operatorname{nAUC}

in gray. Every run uses the recipe of Table 6 trained for a fixed

40

epochs; only the listed hyperparameter varies.

w

and

\epsilon

are the TRAP weight and threshold,

r

the adapter rank,

\varepsilon

under DP the privacy budget.

Method	Settings on the frontier (cross-entropy, worst-case $\operatorname{nAUC}$ )
TRAP, frozen reference	$w{=}0.5,\ \epsilon{=}0$ ( 0.0148, 0.94 ); $w{=}1,\ \epsilon{=}0$ ( 0.0191, 0.45 ); $w{=}2,\ \epsilon{=}0.05$ ( 0.0260, 0.24 ); $w{=}2,\ \epsilon{=}0$ ( 0.0262, 0.24 ); $w{=}4,\ \epsilon{=}0$ ( 0.0285, 0.14 )
TRAP, iterated rounds	$w{=}2,\ \epsilon{=}0$ , round 2 ( 0.0278, 0.12 )
TRAP + LoRA	$r{=}64,\ w{=}1,\ \epsilon{=}0$ ( 0.0149, 0.17 ); $r{=}8,\ w{=}1,\ \epsilon{=}0$ ( 0.0191, 0.17 ); $r{=}4,\ w{=}1,\ \epsilon{=}0$ ( 0.0222, 0.13 ); $r{=}8,\ w{=}4,\ \epsilon{=}0$ ( 0.0298, 0.11 ); $r{=}4,\ w{=}4,\ \epsilon{=}0$ ( 0.0336, 0.10 )
TRAP + LoRA, iterated rounds	$r{=}64,\ w{=}2,\ \epsilon{=}0$ , round 2 ( 0.0179, 0.13 )
LoRA	$r{=}64$ ( 0.0132, 0.49 ); $r{=}8$ ( 0.0177, 0.23 ); $r{=}2$ ( 0.0204, 0.22 ); $r{=}4$ ( 0.0205, 0.18 ); $r{=}1$ ( 0.0253, 0.16 )
LoRA on $\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}$	$r{=}64$ ( 0.0106, 0.38 ); $r{=}8$ ( 0.0135, 0.38 ); $r{=}4$ ( 0.0147, 0.37 )

Appendix

Table 9: The rare-minority mixture at Pythia-70M: the settings on each method’s lower-left frontier in Figure 33 , with the point’s held-out cross-entropy and worst-case

\operatorname{nAUC}

in gray. Every run uses learning rate

5\cdot 10^{-5}

16

examples per step,

3\%

warmup, fp32, no weight decay, a

40

-epoch ceiling with patience

3

200

held-out examples; LoRA adapters on the attention projections with

\alpha{=}2r

and dropout

0.1

, ranks

1

and

2

with an

80

-epoch ceiling; DP with lot

256

, learning rate

10^{-3}

, clipping

0.1

and

\delta{=}10^{-5}

; only the listed hyperparameter varies.

w

and

\epsilon

are the TRAP weight and threshold,

r

the adapter rank,

\varepsilon

under DP the privacy budget.

Method	Settings on the frontier (cross-entropy, worst-case $\operatorname{nAUC}$ )
TRAP, frozen reference	$w{=}0.5,\ \epsilon{=}0$ ( 0.0094, 0.99 ); $w{=}1,\ \epsilon{=}0$ ( 0.0106, 0.58 ); $w{=}2,\ \epsilon{=}0.05$ ( 0.0143, 0.14 ); $w{=}2,\ \epsilon{=}0$ ( 0.0149, 0.08 ); $w{=}4,\ \epsilon{=}0$ ( 0.0197, 0.06 )
TRAP, iterated rounds	$w{=}2,\ \epsilon{=}0$ , round 2 ( 0.0152, 0.27 ); $w{=}2,\ \epsilon{=}0$ , round 3 ( 0.0181, 0.16 ); $w{=}4,\ \epsilon{=}0$ , round 2 ( 0.0254, 0.10 )
TRAP + LoRA	$r{=}64,\ w{=}1,\ \epsilon{=}0$ ( 0.0084, 0.14 ); $r{=}64,\ w{=}2,\ \epsilon{=}0$ ( 0.0106, 0.11 ); $r{=}64,\ w{=}4,\ \epsilon{=}0$ ( 0.0133, 0.09 ); $r{=}4,\ w{=}4,\ \epsilon{=}0$ ( 0.0151, 0.08 )
TRAP + LoRA, iterated rounds	$r{=}64,\ w{=}2,\ \epsilon{=}0$ , round 2 ( 0.0110, 0.11 ); $r{=}8,\ w{=}4,\ \epsilon{=}0$ , round 2 ( 0.0169, 0.10 )
LoRA	$r{=}64$ ( 0.0070, 0.51 ); $r{=}8$ ( 0.0076, 0.34 ); $r{=}4$ ( 0.0082, 0.25 ); $r{=}2$ ( 0.0091, 0.23 ); $r{=}1$ ( 0.0098, 0.17 )
LoRA on $\mathcal{D}_{\mathrm{tgt}}\cup\mathcal{D}_{\mathrm{ref}}$	$r{=}64$ ( 0.0048, 0.64 ); $r{=}8$ ( 0.0061, 0.41 )

Appendix

Table 10: The rare-minority mixture at Pythia-160M: the settings on each method’s lower-left frontier in Figure 34 , with the point’s held-out cross-entropy and worst-case

\operatorname{nAUC}

in gray. Every run uses the recipe of Table 9 ; only the listed hyperparameter varies.

w

and

\epsilon

are the TRAP weight and threshold,

r

the adapter rank,

\varepsilon

under DP the privacy budget.

Figure 35: The family frontiers one score at a time, best-val checkpoint: rows are the essays at Pythia-70M and 1B, MedCase at Pythia-1B and Llama-3.1-8B (the second and third rows are Figure 2 a and b); columns are the four scores; lines are the family frontiers.

Figure 36: The rare-minority mixture one score at a time, best-val checkpoint: rows are the two model sizes (Figure 2 c is the 160M row), columns the four scores, lines the family frontiers of the main figure.

Figure 37: Directional AUC version of Figure 28 : essays at Pythia-70M, with

0.5

marking chance; points below the line are over-corrected models that

\operatorname{nAUC}

folds upward.

Figure 38: Directional AUC version of Figure 29 : essays at Pythia-1B.

Figure 39: Directional AUC version of Figure 30 : MedCase at Pythia-1B.

Figure 40: Directional AUC version of Figure 31 : MedCase at Llama-3.1-8B-Instruct.

Figure 41: Directional AUC version of Figure 32 : MedCase at Pythia-1B trained for a fixed forty epochs.

Figure 42: Directional AUC version of Figure 33 : the rare-minority mixture at Pythia-70M.

Figure 43: Directional AUC version of Figure 34 : the rare-minority mixture at Pythia-160M.

Figure 44: TRAP with different references on the essays at Pythia-70M (a) and 1B (b), best-val checkpoint: worst-case

\operatorname{nAUC}

against held-out cross-entropy, one lower-left frontier per reference with its settings faint, the undefended model as the star and the untrained model as the green dot and dotted line. The asymmetric alternating and the two base references were run at 70M only.

Explore similar work

Oct 3, 2024cs.LG

Mitigating Memorization In Language Models

Language models (LMs) can "memorize" information, i.e., encode training data in their weights in such a way that inference-time queries can lead to verbatim regurgitation of that data. This ability to extract training data can be problematic, for example, when data are private or sensitive. In this work, we investigate methods to mitigate memorization: three regularizer-based, three finetuning-based, and eleven machine unlearning-based methods, with five of the latter being new methods that we introduce. We also introduce TinyMem, a suite of small, computationally-efficient LMs for the rapid development and evaluation of memorization-mitigation methods. We demonstrate that the mitigation methods that we develop using TinyMem can successfully be applied to production-grade LMs, and we determine via experiment that: regularizer-based mitigation methods are slow and ineffective at curbing memorization; fine-tuning-based methods are effective at curbing memorization, but overly expensive, especially for retaining higher accuracies; and unlearning-based methods are faster and more effective, allowing for the precise localization and removal of memorized information from LM weights prior to inference. We show, in particular, that our proposed unlearning method BalancedSubnet outperforms other mitigation methods at removing memorized information while preserving performance on target tasks.

Mansi Sakarvadia, Aswathy Ajith, Arham Khan +6

University of Chicago · Argonne National Laboratory · Lawrence Berkeley National Laboratory +3

Memorization in Language Models LLM Fine-Tuning Neural Network Memorization LLM Memory Privacy Leakage in Language Models Memorization in Generative Models+1

Ask PDF

Jun 26, 2026cs.CR

Decomposing Memorization Reduction in Privacy-Preserving Fine-Tuning of SLMs for CSIRTs

CSIRTs increasingly fine tune language models on vulnerability scan records, but these records expose internal network topology and create privacy risks under regulations such as GDPR and LGPD. We present the first empirical study of how DP SGD and HMAC pseudonymization interact when fine tuning small language models with 1B to 3B parameters on structured CSIRT data. We evaluate 96 LoRA adapters across four SLMs and four training regimes, including raw fine tuning, QLoRA with large batch training, and DP SGD with epsilon equal to 2 and 8. We also audit memorization using 20 planted canaries, four extraction attacks, and a dual attack targeting HMAC pseudonymized identifiers. Our results show three main findings. First, matched update controls reproduce the observed reduction in memorization by reducing the number of optimizer updates alone, accounting for 66 percent to 132 percent of the measured effect, with a mean of 100 percent across three seeds and four models. In this setting, DP SGD provides the formal privacy guarantee but does not produce additional measurable reductions in memorization. Second, HMAC pseudonymization removes the original identifiers from the exposure surface, reducing exposure by 40 percent to 61 percent, while pseudonymized identifiers remain close to the expected random baseline and do not become a secondary memorization target. Third, F1 scores remain between 0.19 and 0.28 across all 96 adapters using four shot prompting, indicating that, under the evaluated training budget, 1B to 3B SLMs do not achieve operationally useful performance.

Cristhian Kapelinski, Diego Kreutz

Federal University of Pampa (UNIPAMPA), Alegrete, Brazil

Privacy-Preserving Language Models Memorization in Language Models Differentially Private Stochastic Gradient Descent Small Language Models LLMs for Cybersecurity

Ask PDF

Aug 27, 2026cs.CR

Memorization Is Not Extraction: Tight Differential-Privacy Bounds and Audit Blind Spots

Memorization in large language models is measured through a zoo of definitions whose formal relations are unknown, and differential privacy (DP) is treated as a proxy against all of them at once. We pin down the exact DP constant for the two that carry the practical weight, counterfactual memorization and adaptive extraction, and show that they do not control each other. Under

f

-DP, every adaptive extraction protocol with list budget

m

succeeds with probability at most

1-f(κ)

for the oblivious baseline

κ

, and the bound is tight on a dense set of baselines: DP uniformly controls extraction exactly up to a threshold in how well the secret can be guessed a priori. Min-entropy certifies that baseline distribution-free, since

H_\infty\geε\log_2 e+\log_2(m/τ)

holds extraction below a risk level

τ\le1/2

under pure

ε

-DP for every prior, and is exact on uniform priors. On the memorization side,

f

-DP caps the counterfactual memorization of any bounded score at an advantage functional

η(f)

, equal to

\tanh(ε/2)

under pure DP; for

k\ge2

duplicated copies the naive

ε\mapsto kε

bound

\tanh(kε/2)

is unattainable, the exact constant being a closed-form staircase attained by geometric noisy counting. That cap is attained inside the local score class used in practice, and it is there that the two measures separate: one mechanism is memorized yet unextractable, another fully extractable yet exactly invisible to every loss-based score. The two-sided blind spot this opens for loss-based auditing and unlearning verification survives on billion-parameter models: a reserved-trigger release is recovered verbatim from one prompt while the audits practitioners deploy certify it clean.

Xujun Che, Depeng Xu, Shuhan Yuan

University of North Carolina at Charlotte · Utah State University

Privacy Leakage in Language Models Memorization in Language Models Differential Privacy Language Model Auditing

Ask PDF