Estimating near-verbatim extraction risk in language models with decoding-constrained beam search

A. Feder Cooper; Mark A. Lemley; Christopher De Sa; Lea Duesterwald; Allison Casasola; Jamie Hayes; Katherine Lee; Daniel E. Ho

$p_{{\bm{z}},\varepsilon}^{\mathsf{dist}}$	$\lceil M\rceil$
	$\delta=0.005$	$\delta=0.05$	$\delta=0.5$
$10^{-1}$	$51$	$29$	$7$
$10^{-2}$	$528$	$299$	$69$
$10^{-3}$	$5{,}296$	$2{,}995$	$693$

Appendix

Table 1: Samples required to hit at least once. Minimum number of MC samples

M

to hit the

\varepsilon

-ball at least once with probability

\geq 1-\delta

.

Figure 8: Monte Carlo (MC) estimation of near-verbatim extraction probability. For Levenshtein distance

\leq\!5

(

p^{\mathsf{Lev}}_{{\bm{z}},\,5}

), we plot convergence for a single sequence

{\bm{z}}

from The Great Gatsby for Llama 2 7B , showing the pooled MC estimate with a 95% confidence interval over 3 replicates. Our algorithm (

k

-CBS) produces a deterministic, provably correct lower bound (LB) of

\approx\!0.01

, which captures

89.4\%

of the mean MC estimate at

M=10^{4}

samples, at a cost of

\approx\!20

MC samples (see Appendix C.3 for cost accounting)—a budget at which MC produces no hits.

Method	Token evals	vs. greedy	Candidates returned
Greedy / Teacher forcing	$99$ / $100$	$1{\times}$	$1$
$k$ -CBS ( $B=20$ )	$1{,}030$	${\approx}\,10{\times}$	up to $800$ (deterministic)
MC ( $M=20$ , same budget)	$1{,}030$	${\approx}\,10{\times}$	$20$ (random)
MC ( $M=3{,}000$ , detection)	$147{,}050$	${\approx}\,1{,}500{\times}$	$3{,}000$ (random)
MC ( $M=10^{5}$ , estimation)	$4{,}900{,}050$	${\approx}\,49{,}500{\times}$	$10^{5}$ (random)

Appendix

Table 2: Token-evaluation cost comparison for a single prefix–suffix pair (

a=50

,

T=50

,

B=20

,

k=40

).

	$\hat{{\bm{z}}}_{\textnormal{(cont)}}$	$\Pr_{\theta,\phi}(\hat{{\bm{z}}}_{\textnormal{(cont)}}\mid{\bm{z}}_{\textnormal{(pre)}})$	$\mathsf{Lev}$	$\mathsf{Ham}$
1		0.1477	1	1
2		0.1431	0	0
3		0.0671	2	22
4		0.0535	1	1
5		0.0409	3	22
6		0.0385	1	1

Appendix

Table 3: Example

k

-CBS output. Top-

10

generations (

|\hat{{\bm{z}}}_{\textnormal{(cont)}}|=32

tokens) for Llama 1 13B baseline

k

-CBS (

B=20,\,k=40

) for the

25

-token prefix

{\bm{z}}_{\textnormal{(pre)}}

: They were careless people, Tom and Daisy - they smashed up things and creatures and then retreated . This is a quote from The Great Gatsby ( Fitzgerald, 1925 ) , a book contained in Llama 1 ’s training data. We diff each

\hat{{\bm{z}}}_{\textnormal{(cont)}}

with

{\bm{z}}_{\textnormal{(suf)}}

, showing deletions in red and crossed out, and additions in blue. All exact-matched text is in black. Row 2 (highlighted in yellow) shows the verbatim generation of the target suffix, i.e.,

\hat{{\bm{z}}}_{\textnormal{(cont)}}={\bm{z}}_{\textnormal{(suf)}}

; there is no diff highlighting.

$D[i,j]$	$j\!=\!0$ ( $\emptyset$ )	$j\!=\!1$ ( $a$ )	$j\!=\!2$ ( $ab$ )	$j\!=\!3$ ( $abc$ )
$i\!=\!0$ ( $\emptyset$ )	$0$	$1$	$2$	$3$
$i\!=\!1$ ( $b$ )	$1$	$1$ $\searrow$	$1$ $\searrow$	$2$ $\rightarrow$
$i\!=\!2$ ( $bc$ )	$2$	$2$ $\downarrow$ $\searrow$	$2$ $\downarrow$ $\searrow$	$1$ $\searrow$

Appendix

Figure 9: Fully worked example. Wagner-Fischer table for

{\bm{z}}_{\textnormal{(suf)}}=(a,b,c)

and

\hat{{\bm{z}}}^{\textnormal{(cont)}}_{1:2}=(b,c)

. Colored arrows show the winning operation at each cell:

\searrow

match/substitute,

\downarrow

delete,

\rightarrow

insert. The last row is

[2,2,2,1]

with

\min_{j}D[2,j]=1

at

j\!=\!3

; the diagonal

D[2,2]\!=\!2

is not the minimum, illustrating why we minimize over all

j

.

$D[i,j]$	$j\!=\!0$ ( $\emptyset$ )	$j\!=\!1$ ( $a$ )	$j\!=\!2$ ( $ab$ )	$j\!=\!3$ ( $abc$ )	$j\!=\!4$ ( $abcd$ )
$i\!=\!0$ ( $\emptyset$ )	$0$	$1$	$2$	$3$	$4$
$i\!=\!1$ ( $b$ )	$1$	$1$ $\searrow$	$1$ $\searrow$	$2$ $\rightarrow$	$3$ $\rightarrow$	$\min\!=\!1$
$i\!=\!2$ ( $bc$ )	$2$	$2$ $\downarrow$ $\searrow$	$2$ $\downarrow$ $\searrow$	$1$ $\searrow$	$2$ $\rightarrow$	$\min\!=\!1$
$i\!=\!3$ ( $bca$ )	$3$	$2$ $\searrow$	$3$ $\downarrow$ $\rightarrow$ $\searrow$	$2$ $\downarrow$	$2$ $\searrow$	$\min\!=\!2$
$i\!=\!4$ ( $bcad$ )	$4$	$3$ $\downarrow$	$3$ $\searrow$	$3$ $\downarrow$	$2$ $\searrow$	$\min\!=\!2$

Appendix

Figure 10: Streaming example.

{\bm{z}}_{\textnormal{(suf)}}=(a,b,c,d)

and

\hat{{\bm{z}}}^{\textnormal{(cont)}}_{1:4}=(b,c,a,d)

. Row minima (shown at right) evolve as

1,1,2,2

. With

\varepsilon\!=\!1

, pruning triggers at

t\!=\!3

; with

\varepsilon\!=\!2

, the path remains viable and ends at

D[4,4]\!=\!2

. Colors as in Figure 9 .

Domain	Dataset	Models	NC	Constraint	Run config
Verbatim probabilistic (teacher forcing); greedy generation
Books	Winnie the Pooh	Llama 2 {7B, 13B, 70B}	No	–	batch $200$ (7B,13B), $400$ (70B); stride $20$ ; GPUs $1$ / $1$ / $4$
Books	The Great Gatsby	Llama 2 {7B, 13B, 70B}	No	–	batch $200$ (7B,13B), $400$ (70B); stride $20$ ; GPUs $1$ / $1$ / $4$
Books	Orlando	Llama 2 {7B, 13B, 70B}	No	–	batch $200$ (7B,13B), $400$ (70B); stride $20$ ; GPUs $1$ / $1$ / $4$
Books	The People’s Dictator	Llama 2 {7B, 13B, 70B}; Llama 3.1 8B	Yes	–	batch $200$ (7B,13B,8B), $400$ (70B); stride $20$ ; GPUs $1$ / $1$ / $1$ / $4$
Books	Pride and Prejudice	Llama 3.1 8B	No	–	batch $200$ ; stride $20$ ; GPUs $1$

Appendix

Table 4: Reported experimental configurations. Configurations across datasets, models, and constraints (where appropriate, beam width

B

, distance metric

\mathsf{dist}\in\{\mathsf{Ham},\mathsf{Lev}\}

, and tolerance

\varepsilon

). We report the number of GPUs and batch size for each experiment. NC = negative control; stride = number of characters used for chunking in book runs.

	OLMo 2 7B		OLMo 2 13B		OLMo 2 32B
Pipeline	Train	Held-out	Train	Held-out	Train	Held-out
Verbatim greedy	0.25%	0.00%	0.33%	0.00%	0.61%	0.02%
Verbatim probabilistic	0.69%	0.00%	1.02%	0.00%	1.61%	0.04%
Verbatim $k$ -CBS	0.55%	0.00%	0.87%	0.00%	1.42%	0.04%

Appendix

Table 5: Verbatim extraction rates for OLMo 2 on Wikipedia. Verbatim extraction rates via greedy (deterministic generation), teacher forcing, and

k

-CBS on training data (

10{,}000

sequences) and held-out negative controls (

5{,}000

sequences). Probabilistic and

k

-CBS rates use threshold

p_{\bm{z}}\geq\tau_{\min}=0.001

.

	OLMo 2 7B	OLMo 2 13B	OLMo 2 32B
Verbatim probabilistic	0.69%	1.02%	1.61%
Verbatim $k$ -CBS	0.55%	0.87%	1.42%
Recovery ratio	0.80	0.85	0.88
Missed by $k$ -CBS	14	16	19

Appendix

Table 6: Verbatim probabilistic vs. verbatim

k

-CBS on OLMo 2 (training data).

k

-CBS with beam width

B\!=\!20

may under-count verbatim extraction. We report rates (

\tau_{\min}=0.001

) for

B\!=\!20

, the

k

-CBS recovery ratio, and the number of sequences missed by

k

-CBS.

Figure 11: Comparing extraction rates for OLMo 2 . For OLMo 2 7B , 13B , and 32B , we show greedy and

k

-CBS probabilistic rates for verbatim extraction (

\varepsilon\!=\!0

) and near-verbatim extraction for

\varepsilon\,\in\{1,\ldots,5\}

. We use a sample of

10{,}000

sequences from Wikipedia from OLMo 2 ’s training data; to assess validity, we also run analogous negative controls on

5{,}000

sequences scraped from Wikipedia that post-date OLMo 2 ’s training cutoff. The greedy rates are exact. The probabilistic rates are computed with

k

-CBS (Section 4.2 ); they may miss some valid instances of extraction, and thus should be interpreted as lower bounds on extraction rates.

7B

13B

32B

greedy

prob.

greedy

prob.

greedy

prob.

v.

nv.

v.

nv.

v.

nv.

v.

nv.

v.

nv.

v.

nv.

1. Provincial Board

✓

2. 98th Academy Awards

✓

3. 2015 WinStar

✓

4. Batman

✓

Appendix

Table 7: Held-out sequences in negative controls for OLMo 2 . Sequences that are extractable verbatim (v.) and near-verbatim (nv.) under both greedy and probabilistic extraction. A verbatim-extractable sequence is necessarily near-verbatim extractable. For probabilistic extraction, the near-verbatim mass often significantly exceeds the verbatim mass.

Figure 12: Near-verbatim mass vs. verbatim mass for OLMo 2 . OLMo 2 on Wikipedia (training subset); each point is one sequence. Axes show near-verbatim (

p_{{\bm{z}},5}^{\mathsf{Lev}}

,

\mathsf{Lev}\,\varepsilon\!=\!5

) vs. verbatim (

p_{\bm{z}}

) extraction mass on a

\log

–

\log

scale. Red / orange points are “unlocked” by near-verbatim extraction (to the left of the

\tau_{\min}

dotted reference line,

p_{\bm{z}}\!<\!\tau_{\min}

, but

p_{{\bm{z}},5}^{\mathsf{Lev}}\!\geq\!\tau_{\min}

); blue points are verbatim-extractable (

p_{\bm{z}}\!\geq\!\tau_{\min}

). Points above the dashed

y\!=\!x

line show increased extraction risk when near-verbatim mass is accounted for.

Figure 20

Figure 14: Illustrating different views of

\varepsilon

-shell share for OLMo 2 on Wikipedia. ( a ) Heatmaps across model size of per-sequence mass share by

\varepsilon

-shell (Equation 56 ), sorted by verbatim share (Equation 57 ). ( b ) Violin plots comparing the distribution of per-sequence verbatim share (Equation 57 ) across model sizes, with the median verbatim share annotated. ( c ) Violin plots showing distributions over the per-

\varepsilon

-shell mass share (Equation 56 ) per model. Each shell shows the mass share contributed by the given

\mathsf{Levenshtein}

distance. Note that each of the three

\varepsilon=0

violin plots correspond to those in the cross-model comparison in ( b ).

	Pythia 1B		Pythia 2.8B		Pythia 6.9B		Pythia 12B
Pipeline	Train	Held-out	Train	Held-out	Train	Held-out	Train	Held-out
Verbatim greedy	0.74%	0.00%	1.81%	0.00%	2.86%	0.00%	3.84%	0.00%
Verbatim probabilistic	1.92%	0.00%	3.53%	0.00%	5.16%	0.00%	6.52%	0.00%
Verbatim $k$ -CBS	1.54%	0.00%	3.21%	0.00%	4.83%	0.00%	6.10%	0.00%

Appendix

Table 9: Verbatim extraction rates for Pythia on Enron emails. Verbatim extraction rates via greedy (deterministic generation), teacher forcing, and

k

-CBS on training data (

10{,}000

sequences) and held-out negative controls (

2{,}000

sequences). Probabilistic and

k

-CBS rates use threshold

p_{\bm{z}}\geq\tau_{\min}=0.001

.

	Pythia 1B	Pythia 2.8B	Pythia 6.9B	Pythia 12B
Verbatim probabilistic	1.92%	3.53%	5.16%	6.52%
Verbatim $k$ -CBS	1.54%	3.21%	4.83%	6.10%
Recovery ratio	0.80	0.91	0.94	0.94
Missed by $k$ -CBS	38	32	35	44

Appendix

Table 10: Verbatim probabilistic vs. verbatim

k

-CBS on Pythia (training data).

k

-CBS with beam width

B\!=\!20

may under-count verbatim extraction. We report rates (

\tau_{\min}=0.001

), the

k

-CBS recovery ratio, and the number of sequences missed by

k

-CBS.

Figure 15: Comparing extraction rates for Pythia . For Pythia 1B , 2.8B , and 6.9B , and 12B we show greedy and

k

-CBS probabilistic rates for verbatim extraction (

\varepsilon\!=\!0

) and near-verbatim extraction for

\varepsilon\,\in\{1,\ldots,5\}

. We use a sample of

10{,}000

sequences from Enron emails from Pythia ’s training data; to assess validity, we also run analogous negative controls on

2{,}000

sequences from TREC 2007 Spam (“ham” subset). The greedy rates are exact. The probabilistic rates are computed with

k

-CBS (Section 4.2 ); they may miss some valid instances of extraction, and thus should be interpreted as lower bounds on extraction rates.

Figure 16: Near-verbatim mass vs. verbatim mass for Pythia . Pythia on Enron emails; each point is one sequence. Axes show near-verbatim (

p_{{\bm{z}},5}^{\mathsf{Lev}}

,

\mathsf{Lev}\,\varepsilon\!=\!5

) vs. verbatim (

p_{\bm{z}}

) extraction mass on a

\log

–

\log

scale. Red / orange points are “unlocked” by near-verbatim extraction (to the left of the

\tau_{\min}

dotted reference line,

p_{\bm{z}}\!<\!\tau_{\min}

, but

p_{{\bm{z}},5}^{\mathsf{Lev}}\!\geq\!\tau_{\min}

); blue points are verbatim-extractable (

p_{\bm{z}}\!\geq\!\tau_{\min}

). Points above the dashed

y\!=\!x

line show increased extraction risk when near-verbatim mass is accounted for.

Figure 26

Figure 18: Illustrating different views of

\varepsilon

-shell share for Pythia . ( a ) Heatmaps across model size of per-sequence mass share by

\varepsilon

-shell (Equation 56 ), sorted by verbatim share (Equation 57 ). ( b ) Violin plots comparing the distribution of per-sequence verbatim share (Equation 57 ) across model sizes, with the median verbatim share annotated. ( c ) Violin plots showing distributions over the per-

\varepsilon

-shell mass share (Equation 56 ) per model. Each shell shows the mass share contributed by the given

\mathsf{Levenshtein}

distance. Note that each of the three

\varepsilon=0

violin plots correspond to those in the cross-model comparison in ( b ).

Table 12: Verbatim extraction rates for Llama 2 . Verbatim extraction rates via greedy (deterministic generation), teacher forcing, and

k

-CBS (

\mathsf{Lev}\,\varepsilon=5

). For all experiments, held-out negative controls use the first three chapters of The People’s Dictator (

10{,}457

sequences). Probabilistic and

k

-CBS rates use threshold

p_{\bm{z}}\geq\tau_{\min}=0.001

.

Table 13: Verbatim probabilistic vs. verbatim

k

-CBS on Llama 2 . We report extraction rates (

\tau_{\min}=0.001

,

B=20

,

\mathsf{Lev}\,\varepsilon=5

), the

k

-CBS recovery ratio, and the number of sequences missed by

k

-CBS.

Figure 19: Comparing extraction rates for Llama 2 . For Llama 2 7B , 13B , and 70B we show greedy and

\mathsf{Lev}\,\varepsilon=5

k

-CBS probabilistic rates for verbatim extraction (

\varepsilon\!=\!0

) and near-verbatim extraction for

\varepsilon\,\in\{1,\ldots,5\}

(based on subsetting the pruned algorithm’s results). To assess validity, we also run analogous negative controls on

10{,}457

sequences from The People’s Dictator (first three chapters). The greedy rates are exact. The probabilistic rates are computed with

\mathsf{Lev}

-pruned

k

-CBS (Section 4.3 ); they may miss some valid instances of extraction, and thus should be interpreted as lower bounds on extraction rates.

Figure 20: Near-verbatim mass vs. verbatim mass for Llama 2 . Llama 2 on three public domain books; each point is one sequence. Axes show near-verbatim (

p_{{\bm{z}},5}^{\mathsf{Lev}}

,

\mathsf{Lev}\,\varepsilon\!=\!5

) vs. verbatim (

p_{\bm{z}}

) extraction mass on a

\log

–

\log

scale. Red / orange points are “unlocked” by near-verbatim extraction (to the left of the

\tau_{\min}

dotted reference line,

p_{\bm{z}}\!<\!\tau_{\min}

, but

p_{{\bm{z}},5}^{\mathsf{Lev}}\!\geq\!\tau_{\min}

); blue points are verbatim-extractable (

p_{\bm{z}}\!\geq\!\tau_{\min}

). Points above the dashed

y\!=\!x

line show increased extraction risk when near-verbatim mass is accounted for.

Figure 21: CCDF of population per-sequence near-verbatim mass gain for Llama 2 . For

\mathsf{Lev}\,\varepsilon\!=\!5

mass minus verbatim mass (

\hat{p}_{{\bm{z}},5}^{\mathsf{Lev}}-p_{\bm{z}}

), a point

(x,y)

means

y\%

of sequences have extraction-mass gain

\geq x

. Plotted over the whole training set sample for each book (

13{,}390

sequences from The Great Gatsby ;

6{,}152

sequences from Winnie the Pooh ).

Figure 22: Illustrating

\varepsilon

-shell share for Llama 2 . Violin plots showing distributions over the per-

\varepsilon

-shell mass share (Equation 56 ) per model and two books. Each shell shows the mass share contributed by the given

\mathsf{Levenshtein}

distance.

Figure 23: Fixed-set mass grows dramatically with model size. By restricting to the sequences extractable at all three model sizes, we hold the set of sequences fixed and ask: how much more probability does a larger model place on the same memorized content?

Figure 24: Heatmaps comparing verbatim and near-verbatim extraction risk across books for Llama 2 models. For each book, we show three pairs of heatmaps—one pair for each Llama 2 model size, with each pair showing the verbatim extraction probability (

p_{\bm{z}}

) and the near-verbatim extraction probability (

\hat{p}^{\mathsf{Lev}}_{{\bm{z}},5}

). On a given heatmap, each vertical strip corresponds to a character position in the source text; color intensity shows the maximum extraction probability across all overlapping

50

-token suffixes that cover that position, on a

\log

scale. White indicates no extractable suffix (

<\tau_{\min}

).

Figure 25: Evolution of near-verbatim mass vs. verbatim mass for Llama 3.1 8B and Pride and Prejudice . Each point is one sequence. Axes show near-verbatim (

p_{{\bm{z}},5}^{\mathsf{Lev}}

,

\mathsf{Lev}\,\varepsilon\!=\!5

) vs. verbatim (

p_{\bm{z}}

) extraction mass on a

\log

–

\log

scale. Red / orange points are “unlocked” by near-verbatim extraction (to the left of the

\tau_{\min}

dotted reference line,

p_{\bm{z}}\!<\!\tau_{\min}

, but

p_{{\bm{z}},5}^{\mathsf{Lev}}\!\geq\!\tau_{\min}

); blue points are verbatim-extractable (

p_{\bm{z}}\!\geq\!\tau_{\min}

). Points above the dashed

y\!=\!x

line show increased extraction risk when near-verbatim mass is accounted for. The top panel is a reference for what verbatim extraction conveys. There are only blue points, and they are all on the line

y=x

; there are

5{,}643

verbatim extractable sequences. The middle panel shows

\varepsilon=1

; even at this distance, there are a large number of “unlocked” points that have zero verbatim mass ( red ) or sub-threshold verbatim mass ( orange ); the total number of extractable points goes up from

5{,}643

to

7{,}132

sequences. The bottom panel shows

\varepsilon=5

. There are even more unlocked sequences (

13{,}565

), and the risk spreads upward for all three categories.

Figure 26: Extraction counts and rates by beam width. For Winnie the Pooh across Llama 2 model sizes, each cell shows the number of near-verbatim extractable sequences (

\hat{p}^{\mathsf{Lev}}_{{\bm{z}},\,\varepsilon}\geq\tau_{\min}=0.001

), with the corresponding extraction rate in parentheses. For a given model’s plot, red shading in a cell indicates relative decreases compared to

B=40

within each row (so the

B=40

cells are all white). In terms of absolute numbers, widening the beam from

B=20

to

B=40

yields at most

26

additional extractable sequences ( Llama 2 70B ,

\mathsf{Lev}\,\varepsilon=5

). in terms of relative differences, Llama 2 13B shows the biggest relative decrease from

B=40

to

B=20

for

\mathsf{Lev}\,\varepsilon=5

(which is why the red shading changes are more pronounced).

Figure 27: Extraction mass by beam width. For Winnie the Pooh across Llama 2 model sizes, each cell shows mean extraction risk (

\pm 1

standard deviation) by beam width for extractable sequences. For a given model’s plot, red shading in a cell indicates the relative drop in mean

\pm

standard deviation, compared to

B=40

within each row (so the

B=40

cells are all white). The mass distribution is stable across beam widths, with relative differences under

5\%

. Note that this measurement is about attempting to gauge stability—large swings in mass—as opposed to seeing how much mass decreases at smaller beam widths. Mean mass can also shift downward at larger beam widths, if an expanded beam width reveals low-mass extractable sequences that smaller beam widths prune. (See, e.g., Llama 2 13B for

\mathsf{Lev}\,\varepsilon=5

.)

Figure 28: Wall-clock runtime (minutes) by beam width and model size. The effective batch size is held roughly constant across beam widths (

B\times

batch size

\approx

400

for 70B,

\approx

200

for 7B/13B). Red cell shading indicates overhead relative to

B=20

. Doubling the beam width approximately doubles runtime, while the extraction gains shown in Figure 26 are minimal.

Figure 29: Extraction counts by configured run, based on filtering to a specific distance threshold. For Winnie the Pooh and each Llama 2 model size, we run baseline

k

-CBS (Section 4.2 & Appendix D ), ( a )

\mathsf{Lev}

-pruned

k

-CBS (Section 4.3 & Appendix E.2 ), and ( b )

\mathsf{Ham}

-pruned

k

-CBS (Section 4.3 & Appendix E.2 ). For each plot, read vertically, each column fixes a distance threshold and compares how extraction counts vary across configured run settings. The top row in all plots is baseline

k

-CBS (no pruning); subsequent rows show pruned runs at

\varepsilon=0,\ldots,5

, for ( a ) the Levenhstein distance and ( b ) the Hamming distance. The diagonal (where pruning

\varepsilon

row matches the evaluation threshold column) represents each pruning setting’s “native” operating point. Gray cells indicate distance thresholds above the run’s pruning

\varepsilon

(unavailable). Blue cell shading encodes the difference from baseline, reflecting that pruning found more extractable sequences. White shows no change. Red would indicate fewer extractable sequences (though we do not observe this).

	7B	13B	70B
Extractable sequences by metric
Verbatim	23 (0.37%)	47 (0.76%)	624 (10.14%)
$\mathsf{Lev}\,\varepsilon=5$	36 (0.59%)	64 (1.04%)	895 (14.55%)
$\mathsf{BLEU}\geq 0.75$	41 (0.67%)	71 (1.15%)	1012 (16.45%)
Set overlap: $\mathsf{Lev}\,\varepsilon\leq 5$ vs. $\mathsf{BLEU}\geq 0.75$
Both	36	64	889

Appendix

Table 15: Extractable sequence counts under three similarity metrics on Winnie the Pooh . Each row counts sequences whose aggregated candidate mass under the given metric meets the extraction threshold

\tau_{\min}

, applied to baseline

k

-CBS outputs (

B\!=\!20

). The bottom section shows the overlap between Lev

d\leq 5

and BLEU

\geq 0.75

: most extractable sequences are captured by both, but each metric finds a small number of sequences missed by the other.

Estimating near-verbatim extraction risk in language models with decoding-constrained beam search

Abstract

Figures & tables

Appendix

Memorization in Language Models

Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs

Bits and Memories: Measuring Verbatim Extraction Across LLM Quantization

Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model

Abstract

Memorization in Language Models

Explore similar work

Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs

Bits and Memories: Measuring Verbatim Extraction Across LLM Quantization

Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model