Memorization in large language models is measured through a zoo of definitions whose formal relations are unknown, and differential privacy (DP) is treated as a proxy against all of them at once. We pin down the exact DP constant for the two that carry the practical weight, counterfactual memorization and adaptive extraction, and show that they do not control each other. Under f-DP, every adaptive extraction protocol with list budget m succeeds with probability at most 1−f(κ) for the oblivious baseline κ, and the bound is tight on a dense set of baselines: DP uniformly controls extraction exactly up to a threshold in how well the secret can be guessed a priori. Min-entropy certifies that baseline distribution-free, since H∞≥εlog2e+log2(m/τ) holds extraction below a risk level τ≤1/2 under pure ε-DP for every prior, and is exact on uniform priors. On the memorization side, f-DP caps the counterfactual memorization of any bounded score at an advantage functional η(f), equal to tanh(ε/2) under pure DP; for k≥2 duplicated copies the naive ε↦kε bound tanh(kε/2) is unattainable, the exact constant being a closed-form staircase attained by geometric noisy counting. That cap is attained inside the local score class used in practice, and it is there that the two measures separate: one mechanism is memorized yet unextractable, another fully extractable yet exactly invisible to every loss-based score. The two-sided blind spot this opens for loss-based auditing and unlearning verification survives on billion-parameter models: a reserved-trigger release is recovered verbatim from one prompt while the audits practitioners deploy certify it clean.
Figures & tables
Figure 1: Relations among memorization measures. Solid edges are proved, dashed ones are open. The violet arrows are the bridges proved here, labeled with the constant each delivers; the red edge is also a result, a two-way separation on the local class Hloc , so neither of those two measures controls the other; thin gray arrows are prior bridges, and the dashed node is the relation we leave open.
prior f -DP bounds
this paper
adversary
one reconstruction
adaptive prompting,
guess
list budget m
decoding
not modeled
any decoding rule
baseline
unconditional
prior conditional on
the corpus remainder
direction
upper bound
upper bound and converse
Table 1: Theorem 5.1 against the prior f -DP bounds on record-level attack success [ 8 , 9 , 10 , 11 , 12 , 13 ] . Row one is the prior bound; the rows below it are established here.
Measure
Adversary game
Bridge from f -DP
Tightness
MI advantage
membership test
exact region [ 31 , 6 ]
known
Reconstruction
ReRo game
prior-baseline bound [ 8 ]
known
Counterfactual mem.
score gap on Hloc
η(f) (Thm. 4.1 )
tight, within Hloc
k -copy mem.
k -hop advantage
ηk∗ (Lem. 4.4 )
exact; < group bound
Extraction
adaptive, list budget m
1−f(κπ(m)) (Thm. 5.1 )
tight on dense κ
Distributional mem.
corpus correlation
—
open (Conj. 7.2 )
Table 2: The measures as adversary games and their quantitative bridges from f -DP. The red edge of Fig. 1 (Theorem 6.1 ) separates rows three and five in both directions on Hloc .
Figure 2: Advantage curves 1−α−f(α) for the three canonical guarantees. The marked maximizers carry the closed forms of Theorem 4.1 : tanh(ε/2) at 1/(eε+1) , δ+(1−δ)tanh(ε/2) at (1−δ)/(eε+1) , and 2Φ(μ/2)−1 at Φ(−μ/2) .
Figure 3: Lemma 4.4 . Left: the cap map fˉ(b)=min(eεb,1−e−ε(1−b)) and the optimal k=4 orbit (cobweb): exponential growth up to the breakpoint 1/(eε+1) , then geometric approach to 1 . Right: the exact constant ηk∗ versus the unattainable naive bound tanh(kε/2) ; the k=2 gap is 0.762−0.632 at ε=1 .
Figure 4: The extraction boundary 1−f(κ) against the oblivious baseline (log–log), with the top axis reading the baseline as secret min-entropy. The horizontal line is a target risk τ=0.1 , and the marker on the baseline axis is α∗(τ) , carried by the dashed rule up to the ε=3 crossing. Everything to its right has 1−f(κ)>τ , so by Corollary 5.5 every “DP yet extractable” witness has a baseline there; to its left H∞≥H∗ and DP holds extraction below τ .
Figure 5: The two constructions of Theorem 6.1 ( v=4 , k=2 , L=2 shown). Left: the true path z is circled, pruned branches are dashed, and the brace marks the surviving paths, over which the posterior is uniform (Lemma D.1 ). Right: the rewritten trigger subtree, and beside it the document’s own prefixes, which the construction leaves identical to the base model.
Measure
P
C
B
E
Extraction (sup-protocol)
✓ (Prop. 7.1 f)
✓ (Prop. 7.1 c)
× (Prop. 7.1 d)
open
Verbatim emission (fixed protocol)
× (Cor. 6.4 )
✓ (Prop. 7.1 c)
× (Prop. 7.1 d)
✓ [ 18 ]
Counterfactual, sup-score
✓ (Prop. 7.1 a)
✓ (Prop. 7.1 c)
open
× (Prop. 7.1 e)
Counterfactual on Hloc
× (Prop. 7.1 b)
✓ (Prop. 7.1 c)
open
× (Prop. 7.1 e); proxy ✓ [ 22 ]
MI advantage
✓ (data processing; [ 6 ] )
✓ [ 5 , 6 ]
✓ [ 31 ]
× (Prop. 7.1 e); w/ shadows ✓ [ 30 , 32 ]
Distributional mem.
open
open
open
open
Table 3: Axiom satisfaction. The access model for E is black-box query access to the single released model; “proxy” marks estimability of the defining single-model score (what practice measures [ 22 ] ), and “w/ shadows” marks estimability under the enlarged access of shadow-model training on population data [ 30 , 32 ] . Cells carrying Prop./Thm./Cor. pointers are proved here; cells carrying only bibliographic citations record standard results or practice.
Figure 6: A canary planted k times in a Zipf corpus, released as geometric noisy counts ( ε=1 ). Left: the measured best-threshold advantage (Monte-Carlo error bars) attains the exact constant ηk∗ of Lemma 4.4 and stays strictly below the group bound tanh(kε/2) . Right: an entropy sweep over uniform priors, where κπ(m)=m2−H∞ exactly, so the plotted curve is the frontier 1−fε(κ) of Theorem 5.1 . Dots are the top- m list adversary ( m=4 ) and crosses the subset-release gadget of Lemma 5.2 ; the dotted rule is the target risk τ and the dashed one the threshold H∗ .
Figure 7: Construction II realized by SGD. A two-layer transformer is poisoned with 64 trigger pairs ( 0.6% of a Markov–Zipf corpus), three seeds per condition. Left: on x∗ ’s own prefixes, the poisoned − clean teacher-forcing gap, in units of the seed-to-seed noise floor, stays inside the ±1 band: the poisoning gap is indistinguishable from the disagreement between two clean models. Right: extraction rates on the poisoned and clean runs, and ∣memh∣ for the two local scores, teacher-forcing (TF) and log-loss (LL), on a log axis.
Figure 8: Measured memh(x) against the per-instance parameter εx for 500 instances (Remark 4.5 ). The extremal randomized-response family sits on the bound tanh(εx/2) ; a family post-processed through a binary symmetric channel of crossover γ , marked BSC( γ ), sits on its own predicted curve strictly below; no point exceeds the bound.
Figure 9: SGD on camouflage-tree data at (v,k,L)=(16,4,4) . Left: learned on-path conditionals at each depth, sorted within the planted k -set: the mass concentrates on the set ( ≥0.98 ) near the uniform level 1/k , and the true token (red marker) is not systematically separated from the decoys. Right: exact enumeration of all 65,536 completions: the posterior over the 256 surviving paths is near-uniform, the dashed line marking the uniform level k−L (normalized entropy 0.984 ); the true path (red line) sits at 1.5× uniform.
Figure 10: Two channels, one length scale, for the reserved-trigger channel realized by ordinary fine-tuning. Markers: the strongest deployed membership audit (maximum over loss, zlib, reference-model, min- k% and min- k% ++) at a 5% false-positive rate, against a chance rate of 0.05 . The z≤2 points come from runs with 400 held-out non-members, the z≥4 points from runs with 48 . The two grey rules are the extraction channels, drawn once because neither moves with the length or the family: extraction at the trigger ρ sits at the top, extraction from the record’s own prefix at the bottom.
model
method
ρ -ext
naive c -ext
retain ppl
Pythia-1.4B
NPO
0.81
0.02
22.5→28.4
GradDiff
1.00
0.04
22.5→42.8
Qwen2.5-1.5B
NPO
1.00
0.12
21.3→21.8
GradDiff
1.00
0.08
21.3→60.2
Table 4: Extraction after a deletion request for the document (c,z) , run until a verbatim-memorization verifier reports the content forgotten, which it does for both secrets in all four configurations. Columns are extraction of the trigger secret at ρ , extraction of the naive duplicate at c , and retain perplexity before and after. Gradient difference (GradDiff) pays for the same core outcome with retain perplexity, which is the pathology negative preference optimization (NPO) [ 21 ] was designed to remove; Appendix F gives both recipes.
Figure 11: The record-level counterfactual on real models. Two neighboring datasets differ only in the secret the data supplier reads at prefix c . Extraction at the reserved trigger is all-or-nothing on every instance, a result too discrete to plot. Left: what the local scores see. The same record inserted in plaintext is visible at 0.79 to 0.94 , while reaching it only through ρ attenuates memh by the marked factor. Right: per instance, on a symmetric log scale, with the median in red. Two estimates of the same seed-to-seed noise floor disagree by up to 81× at z=2 , so no count of instances “at the floor” is quoted.
Figure 12: Audit disagreement on real models. Each cell is that audit’s leak signal, rescaled so that 0 is chance and 1 full detection (rates enter directly, AUCs as 2AUC−1 ). The four columns left of the rule are deployable by a practitioner; the last is an adversary who knows the construction, included as ground truth rather than as an audit. The camouflage row is the one case where even that adversary is capped, at the mk−L of Theorem 6.1 (i). The last row plants a secret below H∗ : the cells marked guessed are rates a decoder that never saw the model matches from the prior alone, so they are neither blind nor leak (Corollary 5.3 ).
Fine-tuning a language model on sensitive records can leave it able to reproduce them. We ask when this memorization arises and how to prevent it without knowing in advance which spans are sensitive. Our starting point is that most memorization scores and attacks share one statistical core: whether the model assigns a token more probability than some reference would. Taking as the reference a model trained on the complementary half of the same corpus gives the Target Reference Advantage (TRA), a per-token signal that separates what a model fit to a particular record from what it learned across records, and is cheap and differentiable. We then study what drives memorization during fine-tuning: it keeps growing well past the validation minimum, is larger on small datasets and at higher learning rates, and higher when the underlying task is harder. Early stopping removes much of it, but because it is chosen by aggregate validation loss it helps least for rare, hard-to-predict spans embedded in otherwise learnable text, which is exactly what sensitive information tends to be. We therefore introduce TRAP, a one-sided penalty on tokenwise TRA that acts only where the target model pulls ahead of its reference. On student essays with annotated personal information and clinical cases with patient identifiers, TRAP brings memorization near the level of an untrained model at little utility cost, where generic regularizers barely move and differential privacy gives up most of what fine-tuning bought.
Muhammed Ustaomeroglu, Ziyue Xu, Hanshen Xiao +3
Carnegie Mellon University · NVIDIA · Purdue University
Language models are almost always quantized before they are deployed, and a growing line of work asks whether quantization also lowers their privacy risk. That work measures privacy almost entirely with membership inference. We think this is the wrong thing to measure for the risk that most people actually worry about, namely a model reproducing its training data word for word, and we measure that directly. Using the Pythia models and the public set of sequences each of them is known to have memorized, we track verbatim extraction across five precision levels, from full precision down to four bits, and across three model sizes, while measuring general capability (perplexity) at every point. We find two things. Quantization is a selective forgetter: verbatim memorization falls off faster than capability at every precision and every model size we tried, and this holds under two unrelated quantization algorithms and two evaluation corpora. But the selectivity is not enough to make quantization a privacy defense, which cuts against the optimistic reading of earlier membership-inference results. At the largest model we study, four-bit quantization still reproduces most of the memorized sequences while giving up only a few percent of capability, and the fraction of memorized data that survives quantization grows with model size. We conclude that compression should not be treated as a way to remove memorized training data, and that extraction, not membership inference, is the number practitioners should be watching. All code, sampled evaluation data, and per-configuration results are released.
Memorization in large language models has been studied almost exclusively through prefix-conditioned extraction, a natural choice for autoregressive models. However, diffusion language models (DLMs) can denoise masked tokens at arbitrary positions. Thus, prefix-only probing reveals only one facet of memorization in DLMs and significantly underestimates the risk of training-data extraction. In order to realistically model extractability of training data in DLMs, we introduce \emph{infilling extraction}, a data-extraction protocol parameterized by an arbitrary binary mask that subsumes prefix-only probing and accounts for the bidirectional inductive bias of DLMs. Instantiating it on LLaDA-8B and Dream-7B across five extraction modes, three training pipelines, and three corpora covering verbatim and partial leakage, we find that mask geometry governs extractability: edge-conditioned masks \emph{extract up to three times more} verbatim sequences than prefix-conditioned ones, and bidirectional access opens channels inaccessible in autoregressive models. In particular, we show that a realistic adversary with access to training data where personally identifiable information has been redacted, can even achieve higher recall on extracting redacted email addresses from DLMs than from scale-matched autoregressive models. Tunable parameters for decoding measurably affect extraction performance, while a follow-up supervised finetuning stage does not eliminate the prior memorization.
Yihan Wang, N. Asokan
University of Waterloo · KTH Royal Institute of Technology