Recent work shows that standard greedy-decoding extraction methods for quantifying memorization in LLMs miss how extraction risk varies across sequences. Probabilistic extraction -- computing the probability of generating a target suffix given a prefix under a decoding scheme -- addresses this, but is tractable only for verbatim memorization, missing near-verbatim instances that pose similar privacy and copyright risks. Quantifying near-verbatim extraction risk is expensive: the set of near-verbatim suffixes is combinatorially large, and reliable Monte Carlo (MC) estimation can require ~100,000 samples per sequence. To mitigate this cost, we introduce decoding-constrained beam search, which yields deterministic lower bounds on near-verbatim extraction risk at a cost comparable to ~20 MC samples per sequence. Across experiments, our approach surfaces information invisible to verbatim methods: many more extractable sequences, substantially larger per-sequence extraction mass, and patterns in how near-verbatim extraction risk manifests across model sizes and types of text.
Figures & tables
Figure 1: Probabilistic extraction. For θ= Llama 1 13B and a training sequence z from The Great Gatsby , we show prefix z(pre):=z1:a and 3 continuations z^(cont):=z^a+1:a+T under ϕ= top- k=40 (Equation 1 ) with conditional probabilities Prθ,k(z^(cont)∣z(pre)) (Equation 2 ). We diff each z^(cont) with the target suffix z(suf):=za+1:a+T (character space: blue additions, red deletions) and compute the Levenshtein distance (token space, Equation 5 ). We highlight verbatim extraction (i.e., z^(cont)=z(suf) , 0.1431≥τmin=0.001 ), which is not the greedy continuation (top row). All three z^(cont) are near-verbatim matches to z(suf) (Section 3 ).
Figure 2: Monte Carlo (MC) estimation. For Levenshtein distance ≤5 ( p^z,5Lev ), we plot convergence for a single sequence z from The Great Gatsby for Llama 2 7B , showing the pooled MC estimate with a 95% confidence interval over 3 replicates. Our algorithm ( k -CBS, Section 4 ) produces a deterministic, provably correct lower bound (LB) of ≈0.01 . It captures 89.4% of the mean MC estimate at M=104 samples, at a cost of ≈20 MC samples (Appendix C.3 )—a budget at which MC produces no hits.
Figure 3: Comparing extraction rates. For OLMo 2 7B , 13B , and 32B , we show rates for verbatim ( ε=0 ) and near-verbatim extraction for Levε∈{1,…,5} . For greedy near-verbatim, one generates the single greedy z^(cont) and checks if Lev(z^(cont),z(suf))≤ε . We use a sample of 10,000 sequences from Wikipedia from OLMo 2 ’s training data; to assess validity, we also run negative controls on 5,000 held-out sequences scraped from Wikipedia that post-date OLMo 2 ’s training cutoff. Greedy rates are computed exactly. Probabilistic rates are computed with k -CBS (Section 4.2 ); they may miss some valid instances of extraction, and thus should be interpreted as lower bounds on extraction rates. We set B=20 for k -CBS, so those results cost ∼20× more than greedy extraction (Section 3 ).
Figure 4: Near-verbatim mass vs. verbatim mass. Llama 2 on The Great Gatsby ; each point is one sequence. Axes show near-verbatim ( pz,5Lev , Levε=5 ) vs. verbatim ( pz ) extraction mass on a log – log scale. Red / orange points are “unlocked” by near-verbatim extraction (to the left of the τmin dotted reference line, pz<τmin , but pz,5Lev≥τmin ); blue points are verbatim-extractable ( pz≥τmin ). Points above the dashed y=x line show increased extraction risk when near-verbatim mass is accounted for.
Figure 5: CCDF of per-sequence near-verbatim mass gain. For Levε=5 mass minus verbatim mass ( p^z,5Lev−pz ), a point (x,y) means y% of sequences have extraction-mass gain ≥x .
Figure 6: Verbatim share. For Llama 2 7B and one sequence from The Great Gatsby , we show the verbatim share (Equation 7 ): 27.56% of the total near-verbatim mass pz,5Lev≈0.06 .
Figure 7: Distribution of verbatim share. We compute verbatim share (Equation 7 ) for each extracted sequence and plot the distribution. Higher values indicate extraction mass is dominated by verbatim memorization; lower values indicate mass spread across near-verbatim variants. We annotate the median verbatim-mass shares.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
pz,εdist
⌈M⌉
δ=0.005
δ=0.05
δ=0.5
10−1
51
29
7
10−2
528
299
69
10−3
5,296
2,995
693
Appendix
Table 1: Samples required to hit at least once. Minimum number of MC samples M to hit the ε -ball at least once with probability ≥1−δ .
Figure 8: Monte Carlo (MC) estimation of near-verbatim extraction probability. For Levenshtein distance ≤5 ( pz,5Lev ), we plot convergence for a single sequence z from The Great Gatsby for Llama 2 7B , showing the pooled MC estimate with a 95% confidence interval over 3 replicates. Our algorithm ( k -CBS) produces a deterministic, provably correct lower bound (LB) of ≈0.01 , which captures 89.4% of the mean MC estimate at M=104 samples, at a cost of ≈20 MC samples (see Appendix C.3 for cost accounting)—a budget at which MC produces no hits.
Method
Token evals
vs. greedy
Candidates returned
Greedy / Teacher forcing
99 / 100
1×
1
k -CBS ( B=20 )
1,030
≈10×
up to 800 (deterministic)
MC ( M=20 , same budget)
1,030
≈10×
20 (random)
MC ( M=3,000 , detection)
147,050
≈1,500×
3,000 (random)
MC ( M=105 , estimation)
4,900,050
≈49,500×
105 (random)
Appendix
Table 2: Token-evaluation cost comparison for a single prefix–suffix pair ( a=50 , T=50 , B=20 , k=40 ).
z^(cont)
Prθ,ϕ(z^(cont)∣z(pre))
Lev
Ham
1
0.1477
1
1
2
0.1431
0
0
3
0.0671
2
22
4
0.0535
1
1
5
0.0409
3
22
6
0.0385
1
1
Appendix
Table 3: Example k -CBS output. Top- 10 generations ( ∣z^(cont)∣=32 tokens) for Llama 1 13B baseline k -CBS ( B=20,k=40 ) for the 25 -token prefix z(pre) : They were careless people, Tom and Daisy - they smashed up things and creatures and then retreated . This is a quote from The Great Gatsby ( Fitzgerald, 1925 ) , a book contained in Llama 1 ’s training data. We diff each z^(cont) with z(suf) , showing deletions in red and crossed out, and additions in blue. All exact-matched text is in black. Row 2 (highlighted in yellow) shows the verbatim generation of the target suffix, i.e., z^(cont)=z(suf) ; there is no diff highlighting.
D[i,j]
j=0 ( ∅ )
j=1 ( a )
j=2 ( ab )
j=3 ( abc )
i=0 ( ∅ )
0
1
2
3
i=1 ( b )
1
1↘
1↘
2→
i=2 ( bc )
2
2↓ ↘
2↓ ↘
1↘
Appendix
Figure 9: Fully worked example. Wagner-Fischer table for z(suf)=(a,b,c) and z^1:2(cont)=(b,c) . Colored arrows show the winning operation at each cell: ↘ match/substitute, ↓ delete, → insert. The last row is [2,2,2,1] with minjD[2,j]=1 at j=3 ; the diagonal D[2,2]=2 is not the minimum, illustrating why we minimize over all j .
D[i,j]
j=0 ( ∅ )
j=1 ( a )
j=2 ( ab )
j=3 ( abc )
j=4 ( abcd )
i=0 ( ∅ )
0
1
2
3
4
i=1 ( b )
1
1↘
1↘
2→
3→
min=1
i=2 ( bc )
2
2↓ ↘
2↓ ↘
1↘
2→
min=1
i=3 ( bca )
3
2↘
3↓ → ↘
2↓
2↘
min=2
i=4 ( bcad )
4
3↓
3↘
3↓
2↘
min=2
Appendix
Figure 10: Streaming example. z(suf)=(a,b,c,d) and z^1:4(cont)=(b,c,a,d) . Row minima (shown at right) evolve as 1,1,2,2 . With ε=1 , pruning triggers at t=3 ; with ε=2 , the path remains viable and ends at D[4,4]=2 . Colors as in Figure 9 .
Table 4: Reported experimental configurations. Configurations across datasets, models, and constraints (where appropriate, beam width B , distance metric dist∈{Ham,Lev} , and tolerance ε ). We report the number of GPUs and batch size for each experiment. NC = negative control; stride = number of characters used for chunking in book runs.
OLMo 2 7B
OLMo 2 13B
OLMo 2 32B
Pipeline
Train
Held-out
Train
Held-out
Train
Held-out
Verbatim greedy
0.25%
0.00%
0.33%
0.00%
0.61%
0.02%
Verbatim probabilistic
0.69%
0.00%
1.02%
0.00%
1.61%
0.04%
Verbatim k -CBS
0.55%
0.00%
0.87%
0.00%
1.42%
0.04%
Appendix
Table 5: Verbatim extraction rates for OLMo 2 on Wikipedia. Verbatim extraction rates via greedy (deterministic generation), teacher forcing, and k -CBS on training data ( 10,000 sequences) and held-out negative controls ( 5,000 sequences). Probabilistic and k -CBS rates use threshold pz≥τmin=0.001 .
OLMo 2 7B
OLMo 2 13B
OLMo 2 32B
Verbatim probabilistic
0.69%
1.02%
1.61%
Verbatim k -CBS
0.55%
0.87%
1.42%
Recovery ratio
0.80
0.85
0.88
Missed by k -CBS
14
16
19
Appendix
Table 6: Verbatim probabilistic vs. verbatim k -CBS on OLMo 2 (training data). k -CBS with beam width B=20 may under-count verbatim extraction. We report rates ( τmin=0.001 ) for B=20 , the k -CBS recovery ratio, and the number of sequences missed by k -CBS.
Figure 11: Comparing extraction rates for OLMo 2 . For OLMo 2 7B , 13B , and 32B , we show greedy and k -CBS probabilistic rates for verbatim extraction ( ε=0 ) and near-verbatim extraction for ε∈{1,…,5} . We use a sample of 10,000 sequences from Wikipedia from OLMo 2 ’s training data; to assess validity, we also run analogous negative controls on 5,000 sequences scraped from Wikipedia that post-date OLMo 2 ’s training cutoff. The greedy rates are exact. The probabilistic rates are computed with k -CBS (Section 4.2 ); they may miss some valid instances of extraction, and thus should be interpreted as lower bounds on extraction rates.
7B
13B
32B
greedy
prob.
greedy
prob.
greedy
prob.
v.
nv.
v.
nv.
v.
nv.
v.
nv.
v.
nv.
v.
nv.
1. Provincial Board
✓
✓
✓
✓
✓
✓
2. 98th Academy Awards
✓
✓
✓
✓
✓
3. 2015 WinStar
✓
✓
4. Batman
✓
✓
✓
Appendix
Table 7: Held-out sequences in negative controls for OLMo 2 . Sequences that are extractable verbatim (v.) and near-verbatim (nv.) under both greedy and probabilistic extraction. A verbatim-extractable sequence is necessarily near-verbatim extractable. For probabilistic extraction, the near-verbatim mass often significantly exceeds the verbatim mass.
Figure 12: Near-verbatim mass vs. verbatim mass for OLMo 2 . OLMo 2 on Wikipedia (training subset); each point is one sequence. Axes show near-verbatim ( pz,5Lev , Levε=5 ) vs. verbatim ( pz ) extraction mass on a log – log scale. Red / orange points are “unlocked” by near-verbatim extraction (to the left of the τmin dotted reference line, pz<τmin , but pz,5Lev≥τmin ); blue points are verbatim-extractable ( pz≥τmin ). Points above the dashed y=x line show increased extraction risk when near-verbatim mass is accounted for.
Figure 20
Figure 14: Illustrating different views of ε -shell share for OLMo 2 on Wikipedia. ( a ) Heatmaps across model size of per-sequence mass share by ε -shell (Equation 56 ), sorted by verbatim share (Equation 57 ). ( b ) Violin plots comparing the distribution of per-sequence verbatim share (Equation 57 ) across model sizes, with the median verbatim share annotated. ( c ) Violin plots showing distributions over the per- ε -shell mass share (Equation 56 ) per model. Each shell shows the mass share contributed by the given Levenshtein distance. Note that each of the three ε=0 violin plots correspond to those in the cross-model comparison in ( b ).
Pythia 1B
Pythia 2.8B
Pythia 6.9B
Pythia 12B
Pipeline
Train
Held-out
Train
Held-out
Train
Held-out
Train
Held-out
Verbatim greedy
0.74%
0.00%
1.81%
0.00%
2.86%
0.00%
3.84%
0.00%
Verbatim probabilistic
1.92%
0.00%
3.53%
0.00%
5.16%
0.00%
6.52%
0.00%
Verbatim k -CBS
1.54%
0.00%
3.21%
0.00%
4.83%
0.00%
6.10%
0.00%
Appendix
Table 9: Verbatim extraction rates for Pythia on Enron emails. Verbatim extraction rates via greedy (deterministic generation), teacher forcing, and k -CBS on training data ( 10,000 sequences) and held-out negative controls ( 2,000 sequences). Probabilistic and k -CBS rates use threshold pz≥τmin=0.001 .
Pythia 1B
Pythia 2.8B
Pythia 6.9B
Pythia 12B
Verbatim probabilistic
1.92%
3.53%
5.16%
6.52%
Verbatim k -CBS
1.54%
3.21%
4.83%
6.10%
Recovery ratio
0.80
0.91
0.94
0.94
Missed by k -CBS
38
32
35
44
Appendix
Table 10: Verbatim probabilistic vs. verbatim k -CBS on Pythia (training data). k -CBS with beam width B=20 may under-count verbatim extraction. We report rates ( τmin=0.001 ), the k -CBS recovery ratio, and the number of sequences missed by k -CBS.
Figure 15: Comparing extraction rates for Pythia . For Pythia 1B , 2.8B , and 6.9B , and 12B we show greedy and k -CBS probabilistic rates for verbatim extraction ( ε=0 ) and near-verbatim extraction for ε∈{1,…,5} . We use a sample of 10,000 sequences from Enron emails from Pythia ’s training data; to assess validity, we also run analogous negative controls on 2,000 sequences from TREC 2007 Spam (“ham” subset). The greedy rates are exact. The probabilistic rates are computed with k -CBS (Section 4.2 ); they may miss some valid instances of extraction, and thus should be interpreted as lower bounds on extraction rates.
Figure 16: Near-verbatim mass vs. verbatim mass for Pythia . Pythia on Enron emails; each point is one sequence. Axes show near-verbatim ( pz,5Lev , Levε=5 ) vs. verbatim ( pz ) extraction mass on a log – log scale. Red / orange points are “unlocked” by near-verbatim extraction (to the left of the τmin dotted reference line, pz<τmin , but pz,5Lev≥τmin ); blue points are verbatim-extractable ( pz≥τmin ). Points above the dashed y=x line show increased extraction risk when near-verbatim mass is accounted for.
Figure 26
Figure 18: Illustrating different views of ε -shell share for Pythia . ( a ) Heatmaps across model size of per-sequence mass share by ε -shell (Equation 56 ), sorted by verbatim share (Equation 57 ). ( b ) Violin plots comparing the distribution of per-sequence verbatim share (Equation 57 ) across model sizes, with the median verbatim share annotated. ( c ) Violin plots showing distributions over the per- ε -shell mass share (Equation 56 ) per model. Each shell shows the mass share contributed by the given Levenshtein distance. Note that each of the three ε=0 violin plots correspond to those in the cross-model comparison in ( b ).
Table 12: Verbatim extraction rates for Llama 2 . Verbatim extraction rates via greedy (deterministic generation), teacher forcing, and k -CBS ( Levε=5 ). For all experiments, held-out negative controls use the first three chapters of The People’s Dictator ( 10,457 sequences). Probabilistic and k -CBS rates use threshold pz≥τmin=0.001 .
Table 13: Verbatim probabilistic vs. verbatim k -CBS on Llama 2 . We report extraction rates ( τmin=0.001 , B=20 , Levε=5 ), the k -CBS recovery ratio, and the number of sequences missed by k -CBS.
Figure 19: Comparing extraction rates for Llama 2 . For Llama 2 7B , 13B , and 70B we show greedy and Levε=5k -CBS probabilistic rates for verbatim extraction ( ε=0 ) and near-verbatim extraction for ε∈{1,…,5} (based on subsetting the pruned algorithm’s results). To assess validity, we also run analogous negative controls on 10,457 sequences from The People’s Dictator (first three chapters). The greedy rates are exact. The probabilistic rates are computed with Lev -pruned k -CBS (Section 4.3 ); they may miss some valid instances of extraction, and thus should be interpreted as lower bounds on extraction rates.
Figure 20: Near-verbatim mass vs. verbatim mass for Llama 2 . Llama 2 on three public domain books; each point is one sequence. Axes show near-verbatim ( pz,5Lev , Levε=5 ) vs. verbatim ( pz ) extraction mass on a log – log scale. Red / orange points are “unlocked” by near-verbatim extraction (to the left of the τmin dotted reference line, pz<τmin , but pz,5Lev≥τmin ); blue points are verbatim-extractable ( pz≥τmin ). Points above the dashed y=x line show increased extraction risk when near-verbatim mass is accounted for.
Figure 21: CCDF of population per-sequence near-verbatim mass gain for Llama 2 . For Levε=5 mass minus verbatim mass ( p^z,5Lev−pz ), a point (x,y) means y% of sequences have extraction-mass gain ≥x . Plotted over the whole training set sample for each book ( 13,390 sequences from The Great Gatsby ; 6,152 sequences from Winnie the Pooh ).
Figure 22: Illustrating ε -shell share for Llama 2 . Violin plots showing distributions over the per- ε -shell mass share (Equation 56 ) per model and two books. Each shell shows the mass share contributed by the given Levenshtein distance.
Figure 23: Fixed-set mass grows dramatically with model size. By restricting to the sequences extractable at all three model sizes, we hold the set of sequences fixed and ask: how much more probability does a larger model place on the same memorized content?
Figure 24: Heatmaps comparing verbatim and near-verbatim extraction risk across books for Llama 2 models. For each book, we show three pairs of heatmaps—one pair for each Llama 2 model size, with each pair showing the verbatim extraction probability ( pz ) and the near-verbatim extraction probability ( p^z,5Lev ). On a given heatmap, each vertical strip corresponds to a character position in the source text; color intensity shows the maximum extraction probability across all overlapping 50 -token suffixes that cover that position, on a log scale. White indicates no extractable suffix ( <τmin ).
Figure 25: Evolution of near-verbatim mass vs. verbatim mass for Llama 3.1 8B and Pride and Prejudice . Each point is one sequence. Axes show near-verbatim ( pz,5Lev , Levε=5 ) vs. verbatim ( pz ) extraction mass on a log – log scale. Red / orange points are “unlocked” by near-verbatim extraction (to the left of the τmin dotted reference line, pz<τmin , but pz,5Lev≥τmin ); blue points are verbatim-extractable ( pz≥τmin ). Points above the dashed y=x line show increased extraction risk when near-verbatim mass is accounted for. The top panel is a reference for what verbatim extraction conveys. There are only blue points, and they are all on the line y=x ; there are 5,643 verbatim extractable sequences. The middle panel shows ε=1 ; even at this distance, there are a large number of “unlocked” points that have zero verbatim mass ( red ) or sub-threshold verbatim mass ( orange ); the total number of extractable points goes up from 5,643 to 7,132 sequences. The bottom panel shows ε=5 . There are even more unlocked sequences ( 13,565 ), and the risk spreads upward for all three categories.
Figure 26: Extraction counts and rates by beam width. For Winnie the Pooh across Llama 2 model sizes, each cell shows the number of near-verbatim extractable sequences ( p^z,εLev≥τmin=0.001 ), with the corresponding extraction rate in parentheses. For a given model’s plot, red shading in a cell indicates relative decreases compared to B=40 within each row (so the B=40 cells are all white). In terms of absolute numbers, widening the beam from B=20 to B=40 yields at most 26 additional extractable sequences ( Llama 2 70B , Levε=5 ). in terms of relative differences, Llama 2 13B shows the biggest relative decrease from B=40 to B=20 for Levε=5 (which is why the red shading changes are more pronounced).
Figure 27: Extraction mass by beam width. For Winnie the Pooh across Llama 2 model sizes, each cell shows mean extraction risk ( ±1 standard deviation) by beam width for extractable sequences. For a given model’s plot, red shading in a cell indicates the relative drop in mean ± standard deviation, compared to B=40 within each row (so the B=40 cells are all white). The mass distribution is stable across beam widths, with relative differences under 5% . Note that this measurement is about attempting to gauge stability—large swings in mass—as opposed to seeing how much mass decreases at smaller beam widths. Mean mass can also shift downward at larger beam widths, if an expanded beam width reveals low-mass extractable sequences that smaller beam widths prune. (See, e.g., Llama 2 13B for Levε=5 .)
Figure 28: Wall-clock runtime (minutes) by beam width and model size. The effective batch size is held roughly constant across beam widths ( B× batch size ≈400 for 70B, ≈200 for 7B/13B). Red cell shading indicates overhead relative to B=20 . Doubling the beam width approximately doubles runtime, while the extraction gains shown in Figure 26 are minimal.
Figure 29: Extraction counts by configured run, based on filtering to a specific distance threshold. For Winnie the Pooh and each Llama 2 model size, we run baseline k -CBS (Section 4.2 & Appendix D ), ( a ) Lev -pruned k -CBS (Section 4.3 & Appendix E.2 ), and ( b ) Ham -pruned k -CBS (Section 4.3 & Appendix E.2 ). For each plot, read vertically, each column fixes a distance threshold and compares how extraction counts vary across configured run settings. The top row in all plots is baseline k -CBS (no pruning); subsequent rows show pruned runs at ε=0,…,5 , for ( a ) the Levenhstein distance and ( b ) the Hamming distance. The diagonal (where pruning ε row matches the evaluation threshold column) represents each pruning setting’s “native” operating point. Gray cells indicate distance thresholds above the run’s pruning ε (unavailable). Blue cell shading encodes the difference from baseline, reflecting that pruning found more extractable sequences. White shows no change. Red would indicate fewer extractable sequences (though we do not observe this).
7B
13B
70B
Extractable sequences by metric
Verbatim
23 (0.37%)
47 (0.76%)
624 (10.14%)
Levε=5
36 (0.59%)
64 (1.04%)
895 (14.55%)
BLEU≥0.75
41 (0.67%)
71 (1.15%)
1012 (16.45%)
Set overlap: Levε≤5 vs. BLEU≥0.75
Both
36
64
889
Appendix
Table 15: Extractable sequence counts under three similarity metrics on Winnie the Pooh . Each row counts sequences whose aggregated candidate mass under the given metric meets the extraction threshold τmin , applied to baseline k -CBS outputs ( B=20 ). The bottom section shows the overlap between Lev d≤5 and BLEU ≥0.75 : most extractable sequences are captured by both, but each metric finds a small number of sequences missed by the other.
Indistinguishability properties such as differential privacy bounds or low empirically measured membership inference are widely treated as proxies to show a model is sufficiently protected against broader memorization risks. However, we show that indistinguishability properties are neither sufficient nor necessary for preventing data extraction in LLM APIs. We formalize a privacy-game separation between extraction and indistinguishability-based privacy, showing that indistinguishability and inextractability are incomparable: upper-bounding distinguishability does not upper-bound extractability. To address this gap, we introduce (l,b)-inextractability as a definition that requires at least 2b expected queries for any black-box adversary to induce the LLM API to emit a protected l-gram substring. We instantiate this via a worst-case extraction game and derive a rank-based extraction risk upper bound for targeted exact extraction, as well as extensions to cover untargeted and approximate extraction. The resulting estimator captures the extraction risk over multiple attack trials and prefix adaptations. We show that it can provide a tight and efficient estimation for standard greedy extraction and an upper bound on the probabilistic extraction risk given any decoding configuration. We empirically evaluate extractability across different models, clarifying its connection to distinguishability, demonstrating its advantage over existing extraction risk estimators, and providing actionable mitigation guidelines across model training, API access, and decoding configurations in LLM API deployment. Our code is publicly available at: https://github.com/Emory-AIMS/Inextractability.
Language models are almost always quantized before they are deployed, and a growing line of work asks whether quantization also lowers their privacy risk. That work measures privacy almost entirely with membership inference. We think this is the wrong thing to measure for the risk that most people actually worry about, namely a model reproducing its training data word for word, and we measure that directly. Using the Pythia models and the public set of sequences each of them is known to have memorized, we track verbatim extraction across five precision levels, from full precision down to four bits, and across three model sizes, while measuring general capability (perplexity) at every point. We find two things. Quantization is a selective forgetter: verbatim memorization falls off faster than capability at every precision and every model size we tried, and this holds under two unrelated quantization algorithms and two evaluation corpora. But the selectivity is not enough to make quantization a privacy defense, which cuts against the optimistic reading of earlier membership-inference results. At the largest model we study, four-bit quantization still reproduces most of the memorized sequences while giving up only a few percent of capability, and the fraction of memorized data that survives quantization grows with model size. We conclude that compression should not be treated as a way to remove memorized training data, and that extraction, not membership inference, is the number practitioners should be watching. All code, sampled evaluation data, and per-configuration results are released.
Language models (LMs) tend to memorize portions of their training data and emit verbatim spans. When the underlying sources are sensitive or copyright-protected, such reproduction raises issues of consent and compensation for creators and compliance risks for developers. We propose Anchored Decoding, a plug-and-play inference-time method for suppressing verbatim copying: it enables decoding from any risky LM trained on mixed-license data by keeping generation in bounded proximity to a permissively trained safe LM. Anchored Decoding adaptively allocates a user-chosen information budget over the generation trajectory and enforces per-step constraints that yield a sequence-level guarantee, enabling a tunable risk-utility trade-off. To make Anchored Decoding practically useful, we introduce a new permissively trained safe model (TinyComma 1.8B), as well as AnchoredByte Decoding, a byte-level variant of our method that enables cross-vocabulary fusion via the ByteSampler framework (Hayase et al., 2025). Across six model pairs on long-form metrics for copying risk and utility, Anchored and AnchoredByte Decoding define a new Pareto frontier, preserving near-original fluency and factuality while closing up to 75% of the measurable copying gap between the risky baseline and a safe reference, at a modest inference overhead.
Jacqueline He, Jonathan Hayase, Wen-tau Yih +3
University of Washington · Allen Institute for Artificial Intelligence