Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects

Xiaoyu Luo; Wenrui Yu; Qiongxiu Li; Johannes Bjerva

cs.CLMar 2, 2026

Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects

Authors: Xiaoyu Luo, Wenrui Yu, Qiongxiu Li, Johannes Bjerva

Organizations: Department of Computer Science, Aalborg University, Copenhagen, Denmark · Department of Electronic Systems, Aalborg University, Copenhagen, Denmark

Abstract

Autoregressive language models (ARMs) have been shown to memorize and occasionally reproduce training data verbatim, raising concerns about privacy and copyright liability. Diffusion language models (DLMs) have recently emerged as a competitive alternative, yet their memorization behavior remains largely unexplored due to fundamental differences in generation dynamics. To address this gap, we present a systematic theoretical and empirical characterization of memorization in DLMs. We propose a generalized probabilistic extraction framework that unifies prefix-conditioned decoding and diffusion-based generation under arbitrary masking patterns and stochastic sampling trajectories. Theorem 4.3 establishes a monotonic relationship between sampling resolution and memorization: increasing resolution strictly increases the probability of exact training data extraction, implying that autoregressive decoding corresponds to a limiting case of diffusion-based generation by setting the sampling resolution maximal. Extensive experiments across model scales and sampling strategies validate our theoretical predictions. Under aligned prefix-conditioned evaluations, we further demonstrate that DLMs exhibit substantially lower memorization-based leakage of personally identifiable information (PII) compared to ARMs.

Figures & tables

Figure 1: A randomly sampled verbatim PII memorization example from diffusion language model ( LLaDA-8B) . This case shows that finer-grained generation resolution (i.e., more steps) is more prone to verbatim recovery of training data. $\langle$ M $\rangle$ denotes a mask token; blue highlights matched strings, and red highlights mismatches.

Figure 2: Empirical vs. theoretical memorization probability measured by DLM-1.1B on 200 SlimPajama examples. Empirical $p_{\boldsymbol{z}}$ is estimated from 100,000 random-decoding generations; theoretical $\hat{p}_{\boldsymbol{z}}$ is estimated from 1,024 mask-patterns at a fixed mask ratio ( $[0.2,0.3]$ ).

Figure 3: Empirical log-probability ratio across decoding steps ( DLM-1.1B ). We sample 200 mask-patterns and, for each, run $10{,}000$ random decoding generations to estimate success probabilities at steps $\{1,2,5,10,\text{per-token}\}$ . We plot the median log-probability difference to one-step (baseline), with $95\%$ bootstrap Confidence Intervals (CIs) across generation-trajectories.

Model	Step	Email		Phone
Model	Step	$p$ = 50%	$p$ = 99%	$p$ = 50%	$p$ = 99%
DLM-1.1B	One	0	0	0	0
DLM-1.1B	Max	16	7	0	0
ARM-1.1B	N/A	213	139	5	3
LLaDA-8B	One	9	8	7	3
LLaDA-8B	Max	179	122	23	17

Table 1: Generalized

(n,p)

-discoverable Extraction. Number of memorized samples at target probabilities

p

under a query budget of

n=10{,}000

3{,}000

completion prompts, for different PII types.

Figure 4: Per-token-step email hit rates under different model scales (170M/629M/1.1B). Each bar reports the Extractable rate over $3{,}000$ samples, measured as whether an email can be recovered within a querying budget $n$ , at fixed $p=0.5$ .

Figure 5: Validating memorization of training data. We compare the reconstruction likelihood from the Enron (train) against the unseen from the TREC 2007 Spam (test), which is from the same domain.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Paired Q-Q plots (left) and CDF comparisons (right) for three cases for DLM.

Panel of Figure 7 — Figure 7: Paired Q-Q plots (left) and CDF comparisons (right) for three cases for ARM.

Panel of Figure 8 — Figure 8: Comparison of empirical and theoretical memorization probabilities under different sampling masked-patterns quantities. As the number of samples increases, the empirical values converge towards the theoretical identity line.

Model	Step	Email				Phone
Model	Step	$p$ = 10%	$p$ = 50%	$p$ = 90%	$p$ = 99%	$p$ = 10%	$p$ = 50%	$p$ = 90%	$p$ = 99%
DLM-170M	One	0	0	0	0	0	0	0	0
DLM-170M	Max	7	2	0	0	0	0	0	0
DLM-629M	One	0	0	0	0	0	0	0	0
DLM-629M	Max	17	10	7	4	0	0	0	0
DLM-1.1B	One	0	0	0	0	0	0	0	0

Appendix

Table 2: Generalized

(n,p)

-discoverable Extraction. Number of memorized samples at target probabilities

p

under a query budget of

n=10{,}000

3{,}000

completion prompts, for different PII types.

Panel of Figure 9 — Figure 9: Validating memorization of training data under different DLM scales. We compare the reconstruction likelihood from the Enron (train) against the unseen from the TREC 2007 Spam (test), which is from the same domain.

Training Data Extraction

See all 11 papers

Extracting Training Data from Diffusion Language Models via Infilling
May 22, 2026Yihan Wang, N. AsokanData Leakage Masked Diffusion Models
University of Waterloo · KTH Royal Institute of Technology
Membership Inference in Fine-tuned Diffusion Language Models via Token-level Memorization Asymmetry
Sep 1, 2026Shengfang Zhai, Leo Marchyok, Yuling Shi +4Memorization in Language Models Membership Inference Attacks
National University of Singapore · Oregon State University · Shanghai Jiao Tong University +1
MemDLM: Memory-Enhanced DLM Training
Mar 23, 2026Zehua Pei, Hui-Ling Zhen, Weizhe Lin +4Diffusion Language Model Training Diffusion Language Models
The Chinese University of Hong Kong · Huawei Technologies Co., Ltd

May 22, 2026J/K move · Enter open · S save

Extracting Training Data from Diffusion Language Models via Infilling

Yihan Wang, N. Asokan

University of Waterloo · KTH Royal Institute of Technology

Data Leakage Masked Diffusion Models Memorization in Language Models Training Data Extraction Privacy Leakage in Language Models Memorization in Generative Models+2

Open paper AskPDF arXiv

Memorization in large language models has been studied almost exclusively through prefix-conditioned extraction, a natural choice for autoregressive models. However, diffusion language models (DLMs) can denoise masked tokens at arbitrary positions. Thus, prefix-only probing reveals only one facet of memorization in DLMs and significantly underestimates the risk of training-data extraction. In order to realistically model extractability of training data in DLMs, we introduce infilling extraction, a data-extraction protocol parameterized by an arbitrary binary mask that subsumes prefix-only probing and accounts for the bidirectional inductive bias of DLMs. Instantiating it on LLaDA-8B and Dream-7B across five extraction modes, three training pipelines, and three corpora covering verbatim and partial leakage, we find that mask geometry governs extractability: edge-conditioned masks extract up to three times more verbatim sequences than prefix-conditioned ones, and bidirectional access opens channels inaccessible in autoregressive models. In particular, we show that a realistic adversary with access to training data where personally identifiable information has been redacted, can even achieve higher recall on extracting redacted email addresses from DLMs than from scale-matched autoregressive models. Tunable parameters for decoding measurably affect extraction performance, while a follow-up supervised finetuning stage does not eliminate the prior memorization.

Abstract

Training Data Extraction

Explore similar work

Extracting Training Data from Diffusion Language Models via Infilling

Membership Inference in Fine-tuned Diffusion Language Models via Token-level Memorization Asymmetry

MemDLM: Memory-Enhanced DLM Training