Weak Ties, Strong Signals: Efficient Training Data Detection in Diffusion LLMs via Independent Token Sampling

Hongyao Yu; Tianqu Zhuang; Ziyuan Xu; Hao Fang; Jiaxin Hong; Bin Chen; Shu-Tao Xia

cs.LGAug 26, 2026

Weak Ties, Strong Signals: Efficient Training Data Detection in Diffusion LLMs via Independent Token Sampling

Authors: Hongyao Yu, Tianqu Zhuang, Ziyuan Xu, Hao Fang, Jiaxin Hong, Bin Chen, Shu-Tao Xia

Organizations: Tsinghua University · Harbin Institute of Technology, Shenzhen

Abstract

Diffusion large language models (dLLMs) offer a compelling alternative to autoregressive models, yet they may expose sensitive training data during denoising. Detecting such usage is challenging because dLLMs lack the efficient one-pass probability decomposition of causal architectures. Existing methods rely on random masking to obtain tractable token-wise detection signals under limited query budgets, but fail to control dependencies among masked tokens. We demonstrate that this token-wise approximation introduces a non-negative structural estimation error, which is theoretically characterized by the cumulative conditional mutual information (CMI) among masked tokens and can obscure subtle memorization signals. This insight suggests that reliable detection requires masked token sets with weak internal dependency. To avoid the prohibitive cost of directly estimating CMI over token combinations, we propose Independent Token Sampling (ITS), a query-efficient framework that uses an attention-derived pairwise dependency proxy to approximate the CMI-aware selection criterion. ITS further incorporates a diversity-promoting strategy to improve token coverage across sampling rounds, yielding aggregated token-wise signals that are less affected by dependency-induced approximation error. Experiments on multiple datasets show that ITS consistently outperforms state-of-the-art baselines across different models and datasets, achieving an AUC improvement of 0.18 on the ArXiv dataset while maintaining strong performance under limited query budgets. The code is available at https://github.com/Chrisqcwx/DLLM-MIA .

Figures & tables

Figure 1: Illustration of token sampling strategies. Existing random sampling methods often fail to detect training data in dLLMs. By contrast, our method selects weakly dependent token subsets, enabling more robust training data detection.

Panel of Figure 2 — Figure 2: Distribution of sample-level detection scores under different token sampling strategies on LLaDA fine-tuned with ArXiv dataset. Each score is computed for one sample as the proportion of selected tokens where the target model assigns a higher probability than the reference model, and the distribution is obtained by aggregating scores over many samples. Compared with (a) high-dependency sampling and (b) random sampling, our (c) low-dependency strategy yields a clearer separation between training data (red) and test data (blue).

Figure 3: Overview of our proposed ITS detection method

Method	ArXiv			Pile CC			AG News
Method	AUC	T@10%	T@1%	AUC	T@10%	T@1%	AUC	T@10%	T@1%
Loss	$0.52_{\pm 0.01}$	$0.13_{\pm 0.01}$	$0.01_{\pm 0.01}$	$0.52_{\pm 0.01}$	$0.13_{\pm 0.01}$	$0.02_{\pm 0.01}$	$0.52_{\pm 0.00}$	$0.10_{\pm 0.01}$	$0.01_{\pm 0.01}$
Zlib	$0.49_{\pm 0.01}$	$0.11_{\pm 0.02}$	$0.01_{\pm 0.01}$	$0.50_{\pm 0.01}$	$0.11_{\pm 0.02}$	$0.01_{\pm 0.00}$	$0.53_{\pm 0.01}$	$0.12_{\pm 0.01}$	$\mathbf{0.02}_{\pm 0.01}$
Min-k%	$0.51_{\pm 0.01}$	$0.13_{\pm 0.01}$	$0.01_{\pm 0.01}$	$0.52_{\pm 0.01}$	$0.13_{\pm 0.01}$	$0.02_{\pm 0.01}$	$0.52_{\pm 0.00}$	$0.10_{\pm 0.01}$	$0.01_{\pm 0.01}$
Min-k%++	$0.49_{\pm 0.01}$	$0.08_{\pm 0.01}$	$0.01_{\pm 0.00}$	$0.50_{\pm 0.02}$	$0.08_{\pm 0.02}$	$0.01_{\pm 0.00}$	$0.51_{\pm 0.01}$	$0.11_{\pm 0.01}$	$0.01_{\pm 0.00}$
DF-MIA	$0.52_{\pm 0.01}$	$0.14_{\pm 0.01}$	$0.01_{\pm 0.00}$	$0.54_{\pm 0.02}$	$0.11_{\pm 0.01}$	$0.01_{\pm 0.00}$	$0.61_{\pm 0.02}$	$0.18_{\pm 0.02}$	$0.01_{\pm 0.00}$

Table 1: Detection Performance on LLaDA.

Figure 4: Training data detection performance on LoRA-tuned models across different ranks

r

and modules.

Figure 5: Detection performance across varying text retain ratios

Panel of Figure 6 — Figure 6: Impact of key parameters on detection AUC. (a) Training epochs (b) Sampling rounds (c) Diversity-promoting penalty $\lambda$ .

Dependency level	AUC	T@10%	T@1%
High	0.57	0.16	0.02
Medium	0.81	0.49	0.13
Low (Ours)	0.87	0.64	0.28

Table 2: Ablation study on dependency levels.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	ArXiv			Pile CC			AG News
Method	AUC	T@10%	T@1%	AUC	T@10%	T@1%	AUC	T@10%	T@1%
Loss	$0.52_{\pm 0.01}$	$0.13_{\pm 0.01}$	$0.01_{\pm 0.00}$	$0.51_{\pm 0.01}$	$0.11_{\pm 0.01}$	$0.02_{\pm 0.01}$	$0.53_{\pm 0.01}$	$0.11_{\pm 0.01}$	$0.01_{\pm 0.00}$
Zlib	$0.49_{\pm 0.01}$	$0.12_{\pm 0.01}$	$0.01_{\pm 0.00}$	$0.48_{\pm 0.01}$	$0.09_{\pm 0.02}$	$0.01_{\pm 0.00}$	$0.54_{\pm 0.01}$	$0.14_{\pm 0.01}$	$0.02_{\pm 0.01}$
Min-k%	$0.52_{\pm 0.01}$	$0.13_{\pm 0.01}$	$0.01_{\pm 0.00}$	$0.51_{\pm 0.01}$	$0.11_{\pm 0.01}$	$0.02_{\pm 0.01}$	$0.53_{\pm 0.01}$	$0.11_{\pm 0.01}$	$0.01_{\pm 0.00}$
Min-k%++	$0.50_{\pm 0.01}$	$0.08_{\pm 0.01}$	$0.01_{\pm 0.00}$	$0.49_{\pm 0.01}$	$0.07_{\pm 0.02}$	$0.01_{\pm 0.00}$	$0.51_{\pm 0.01}$	$0.11_{\pm 0.01}$	$0.01_{\pm 0.00}$
DF-MIA	$0.52_{\pm 0.01}$	$0.10_{\pm 0.01}$	$0.00_{\pm 0.00}$	$0.51_{\pm 0.02}$	$0.08_{\pm 0.01}$	$0.01_{\pm 0.00}$	$0.60_{\pm 0.01}$	$0.19_{\pm 0.02}$	$0.02_{\pm 0.01}$

Appendix

Table 3: Detection Performance on Dream.

Rank	AUC		T@10%		T@1%
Rank	SAMA	ITS	SAMA	ITS	SAMA	ITS
1	0.48	0.50	0.10	0.09	0.01	0.01
2	0.50	0.52	0.09	0.12	0.01	0.01
4	0.51	0.53	0.11	0.13	0.01	0.01
8	0.52	0.56	0.13	0.13	0.01	0.02
16	0.52	0.58	0.10	0.17	0.02	0.02

Appendix

Table 4: Detection performance with different LoRA ranks. The target modules are attention layers.

Rank	AUC		T@10%		T@1%
Rank	SAMA	ITS	SAMA	ITS	SAMA	ITS
1	0.63	0.76	0.20	0.40	0.03	0.08
2	0.69	0.85	0.28	0.54	0.06	0.23
4	0.68	0.86	0.27	0.59	0.06	0.25
8	0.73	0.90	0.34	0.68	0.10	0.24
16	0.76	0.93	0.40	0.76	0.08	0.43

Appendix

Table 5: Detection performance with different LoRA ranks. The target modules are MLP layers.

Retain Ratio	AUC		T@10%		T@1%
Retain Ratio	SAMA	ITS	SAMA	ITS	SAMA	ITS
30%	0.59	0.63	0.17	0.20	0.01	0.02
50%	0.64	0.76	0.21	0.40	0.05	0.07
75%	0.67	0.81	0.28	0.47	0.05	0.15
90%	0.67	0.83	0.27	0.53	0.06	0.22
100%	0.70	0.88	0.34	0.64	0.06	0.26

Appendix

Table 6: Detection performance with different text retain ratios.

Epoch	AUC		T@10%		T@1%
Epoch	SAMA	ITS	SAMA	ITS	SAMA	ITS
1	0.47	0.50	0.07	0.09	0.01	0.02
2	0.52	0.56	0.11	0.16	0.01	0.02
3	0.63	0.76	0.21	0.38	0.02	0.12
4	0.68	0.86	0.28	0.61	0.05	0.26
5	0.75	0.93	0.38	0.78	0.11	0.38

Appendix

Table 7: Detection performance with different fine-tuning epochs.

$T$	AUC		T@10%		T@1%
$T$	SAMA	ITS	SAMA	ITS	SAMA	ITS
1	0.71	0.77	0.34	0.41	0.07	0.09
2	0.72	0.84	0.34	0.58	0.07	0.18
4	0.70	0.88	0.34	0.64	0.06	0.26
8	0.71	0.89	0.33	0.66	0.08	0.33
12	0.72	0.91	0.36	0.75	0.06	0.33

Appendix

Table 8: Detection performance with different sampling rounds

T

$\lambda$	AUC	T@10%	T@1%
0.00	0.84	0.59	0.17
0.05	0.87	0.63	0.24
0.10	0.88	0.64	0.26
0.15	0.87	0.61	0.25
0.20	0.87	0.63	0.25
0.30	0.87	0.64	0.25

Appendix

Table 9: Detection performance with different diversity-promoting penalty

\lambda

Figure 7: Detection performance under different temperature.

$t$	AUC		T@10%		T@1%
$t$	SAMA	ITS	SAMA	ITS	SAMA	ITS
0.75	0.69	0.86	0.29	0.62	0.06	0.26
0.80	0.69	0.86	0.31	0.60	0.06	0.23
0.90	0.69	0.87	0.30	0.64	0.08	0.26
0.95	0.70	0.88	0.30	0.64	0.07	0.26
1.00	0.70	0.88	0.34	0.64	0.06	0.26

Appendix

Table 10: Detection performance with different temperature.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 280 papers

Mar 9, 2026

DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention

Masked diffusion language models enable parallel token decoding, providing a promising alternative to the sequential nature of autoregressive generation. However, their iterative denoising process remains computationally expensive because it repeatedly processes the entire sequence at every step. We observe that across these diffusion steps, most token representations remain stable; only a small subset, which we term salient tokens, contributes meaningfully to the next update. Leveraging this temporal sparsity, we present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only these salient tokens. DyLLM identifies saliency by measuring the cosine similarity of attention contexts between adjacent denoising steps. It recomputes feed-forward and attention operations only for salient tokens while reusing cached activations for the remainder. Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput while largely preserving the baseline accuracy of representative open-source diffusion LLMs, LLaDA, and Dream.

Younjoo Lee, Seungkyun Dan, Junghoo Lee +2

Seoul National University, Seoul, Republic of Korea

Diffusion Language Model Inference Diffusion Language Model Decoding Diffusion Model Inference Acceleration Token Pruning

Ask PDF

Jun 10, 2026

Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models

Diffusion large language models (dLLMs) offer an efficient alternative to autoregressive models through parallel decoding, yet existing post-training methods largely rely on random masking strategies that overlook intrinsic token dependencies. In this work, we present an empirical analysis of attention in dLLMs and show that tokens attending more strongly to unmasked context exhibit greater generation stability and play a critical role in reasoning. Motivated by these findings, we propose AGDO, an attention-guided denoising and optimization framework that aligns both training and optimization with attention-derived dependencies. AGDO determines the denoising order based on attention structure and emphasizes attention-critical tokens during supervised fine-tuning and reinforcement learning. Experiments on mathematical and coding benchmarks demonstrate that AGDO consistently improves reasoning performance, outperforming state-of-the-art post-training methods for dLLMs.

Jia Deng, Junyi Li, Wayne Xin Zhao +3

Gaoling School of Artificial Intelligence, Renmin University of China · Department of Data Science, City University of Hong Kong · Beijing Key Laboratory of Research on Large Models and Intelligent Governance +2

Masked Diffusion Models Diffusion Model Fine-Tuning Diffusion Language Model Decoding Diffusion Language Models

Ask PDF

Mar 16, 2026

Mask Is What DLLM Needs: A Masked Data Training Paradigm for Diffusion LLMs

Discrete diffusion models offer global context awareness and flexible parallel generation. However, uniform random noise schedulers in standard DLLM training overlook the highly non-uniform information density inherent in real-world sequences. This wastes optimization resources on low-density structural glues while leaving high-density logical pivot points severely under-optimized. To address this, we propose an Information Density Driven Smart Noise Scheduler. By extracting information-dense hubs and applying Complementary Priority Masking, our method decouples a single training instance into mutually reinforcing reasoning and syntax samples, forcing the model to master both logical deduction and foundational sequence structure. Experiments demonstrate that our approach improves average accuracy by ~4% across four Code and Math reasoning benchmarks, significantly outperforming uniform baselines. Mechanistic analyses further reveal that probabilistic priority masking effectively mitigates contextual collapse during block diffusion training. Overall, this density-aware strategy efficiently unlocks the reasoning potential of diffusion language models at minimal annotation cost, emerging as a promising new masked data training paradigm for Diffusion LLMs. Our processed dataset can be found at https://huggingface.co/datasets/malr07/opc-sft-stage2-dense-extracted.

Linrui Ma, Yufei Cui, Kai Han +1

Noah’s Ark Lab, Huawei Montreal, Canada & Beijing, China

Diffusion Language Models Blockwise Diffusion Language Models LLM Reasoning

Ask PDF

Abstract

Privacy Leakage in Language Models

Explore similar work

DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention

Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models

Mask Is What DLLM Needs: A Masked Data Training Paradigm for Diffusion LLMs