cs.LGAug 26, 2026

Weak Ties, Strong Signals: Efficient Training Data Detection in Diffusion LLMs via Independent Token Sampling

Authors: Hongyao Yu, Tianqu Zhuang, Ziyuan Xu, Hao Fang, Jiaxin Hong, Bin Chen, Shu-Tao Xia

Organizations: Tsinghua University · Harbin Institute of Technology, Shenzhen

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Diffusion large language models (dLLMs) offer a compelling alternative to autoregressive models, yet they may expose sensitive training data during denoising. Detecting such usage is challenging because dLLMs lack the efficient one-pass probability decomposition of causal architectures. Existing methods rely on random masking to obtain tractable token-wise detection signals under limited query budgets, but fail to control dependencies among masked tokens. We demonstrate that this token-wise approximation introduces a non-negative structural estimation error, which is theoretically characterized by the cumulative conditional mutual information (CMI) among masked tokens and can obscure subtle memorization signals. This insight suggests that reliable detection requires masked token sets with weak internal dependency. To avoid the prohibitive cost of directly estimating CMI over token combinations, we propose Independent Token Sampling (ITS), a query-efficient framework that uses an attention-derived pairwise dependency proxy to approximate the CMI-aware selection criterion. ITS further incorporates a diversity-promoting strategy to improve token coverage across sampling rounds, yielding aggregated token-wise signals that are less affected by dependency-induced approximation error. Experiments on multiple datasets show that ITS consistently outperforms state-of-the-art baselines across different models and datasets, achieving an AUC improvement of 0.18 on the ArXiv dataset while maintaining strong performance under limited query budgets. The code is available at https://github.com/Chrisqcwx/DLLM-MIA .

Figures & tables

Figure 1: Illustration of token sampling strategies. Existing random sampling methods often fail to detect training data in dLLMs. By contrast, our method selects weakly dependent token subsets, enabling more robust training data detection.
Figure 2: Distribution of sample-level detection scores under different token sampling strategies on LLaDA fine-tuned with ArXiv dataset. Each score is computed for one sample as the proportion of selected tokens where the target model assigns a higher probability than the reference model, and the distribution is obtained by aggregating scores over many samples. Compared with (a) high-dependency sampling and (b) random sampling, our (c) low-dependency strategy yields a clearer separation between training data (red) and test data (blue).
Figure 3: Overview of our proposed ITS detection method
MethodArXivPile CCAG News
AUCT@10%T@1%AUCT@10%T@1%AUCT@10%T@1%
Loss0.52±0.010.52_{\pm 0.01}0.13±0.010.13_{\pm 0.01}0.01±0.010.01_{\pm 0.01}0.52±0.010.52_{\pm 0.01}0.13±0.010.13_{\pm 0.01}0.02±0.010.02_{\pm 0.01}0.52±0.000.52_{\pm 0.00}0.10±0.010.10_{\pm 0.01}0.01±0.010.01_{\pm 0.01}
Zlib0.49±0.010.49_{\pm 0.01}0.11±0.020.11_{\pm 0.02}0.01±0.010.01_{\pm 0.01}0.50±0.010.50_{\pm 0.01}0.11±0.020.11_{\pm 0.02}0.01±0.000.01_{\pm 0.00}0.53±0.010.53_{\pm 0.01}0.12±0.010.12_{\pm 0.01}0.02±0.01\mathbf{0.02}_{\pm 0.01}
Min-k%0.51±0.010.51_{\pm 0.01}0.13±0.010.13_{\pm 0.01}0.01±0.010.01_{\pm 0.01}0.52±0.010.52_{\pm 0.01}0.13±0.010.13_{\pm 0.01}0.02±0.010.02_{\pm 0.01}0.52±0.000.52_{\pm 0.00}0.10±0.010.10_{\pm 0.01}0.01±0.010.01_{\pm 0.01}
Min-k%++0.49±0.010.49_{\pm 0.01}0.08±0.010.08_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.50±0.020.50_{\pm 0.02}0.08±0.020.08_{\pm 0.02}0.01±0.000.01_{\pm 0.00}0.51±0.010.51_{\pm 0.01}0.11±0.010.11_{\pm 0.01}0.01±0.000.01_{\pm 0.00}
DF-MIA0.52±0.010.52_{\pm 0.01}0.14±0.010.14_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.54±0.020.54_{\pm 0.02}0.11±0.010.11_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.61±0.020.61_{\pm 0.02}0.18±0.020.18_{\pm 0.02}0.01±0.000.01_{\pm 0.00}
Table 1: Detection Performance on LLaDA.
Figure 4: Training data detection performance on LoRA-tuned models across different ranks rr and modules.
Figure 5: Detection performance across varying text retain ratios
Figure 6: Impact of key parameters on detection AUC. (a) Training epochs (b) Sampling rounds (c) Diversity-promoting penalty λ\lambda .
Dependency levelAUCT@10%T@1%
High0.570.160.02
Medium0.810.490.13
Low (Ours)0.870.640.28
Table 2: Ablation study on dependency levels.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

MethodArXivPile CCAG News
AUCT@10%T@1%AUCT@10%T@1%AUCT@10%T@1%
Loss0.52±0.010.52_{\pm 0.01}0.13±0.010.13_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.51±0.010.51_{\pm 0.01}0.11±0.010.11_{\pm 0.01}0.02±0.010.02_{\pm 0.01}0.53±0.010.53_{\pm 0.01}0.11±0.010.11_{\pm 0.01}0.01±0.000.01_{\pm 0.00}
Zlib0.49±0.010.49_{\pm 0.01}0.12±0.010.12_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.48±0.010.48_{\pm 0.01}0.09±0.020.09_{\pm 0.02}0.01±0.000.01_{\pm 0.00}0.54±0.010.54_{\pm 0.01}0.14±0.010.14_{\pm 0.01}0.02±0.010.02_{\pm 0.01}
Min-k%0.52±0.010.52_{\pm 0.01}0.13±0.010.13_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.51±0.010.51_{\pm 0.01}0.11±0.010.11_{\pm 0.01}0.02±0.010.02_{\pm 0.01}0.53±0.010.53_{\pm 0.01}0.11±0.010.11_{\pm 0.01}0.01±0.000.01_{\pm 0.00}
Min-k%++0.50±0.010.50_{\pm 0.01}0.08±0.010.08_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.49±0.010.49_{\pm 0.01}0.07±0.020.07_{\pm 0.02}0.01±0.000.01_{\pm 0.00}0.51±0.010.51_{\pm 0.01}0.11±0.010.11_{\pm 0.01}0.01±0.000.01_{\pm 0.00}
DF-MIA0.52±0.010.52_{\pm 0.01}0.10±0.010.10_{\pm 0.01}0.00±0.000.00_{\pm 0.00}0.51±0.020.51_{\pm 0.02}0.08±0.010.08_{\pm 0.01}0.01±0.000.01_{\pm 0.00}0.60±0.010.60_{\pm 0.01}0.19±0.020.19_{\pm 0.02}0.02±0.010.02_{\pm 0.01}
Appendix
Table 3: Detection Performance on Dream.
RankAUCT@10%T@1%
SAMAITSSAMAITSSAMAITS
10.480.500.100.090.010.01
20.500.520.090.120.010.01
40.510.530.110.130.010.01
80.520.560.130.130.010.02
160.520.580.100.170.020.02
Appendix
Table 4: Detection performance with different LoRA ranks. The target modules are attention layers.
RankAUCT@10%T@1%
SAMAITSSAMAITSSAMAITS
10.630.760.200.400.030.08
20.690.850.280.540.060.23
40.680.860.270.590.060.25
80.730.900.340.680.100.24
160.760.930.400.760.080.43
Appendix
Table 5: Detection performance with different LoRA ranks. The target modules are MLP layers.
Retain RatioAUCT@10%T@1%
SAMAITSSAMAITSSAMAITS
30%0.590.630.170.200.010.02
50%0.640.760.210.400.050.07
75%0.670.810.280.470.050.15
90%0.670.830.270.530.060.22
100%0.700.880.340.640.060.26
Appendix
Table 6: Detection performance with different text retain ratios.
EpochAUCT@10%T@1%
SAMAITSSAMAITSSAMAITS
10.470.500.070.090.010.02
20.520.560.110.160.010.02
30.630.760.210.380.020.12
40.680.860.280.610.050.26
50.750.930.380.780.110.38
Appendix
Table 7: Detection performance with different fine-tuning epochs.
TTAUCT@10%T@1%
SAMAITSSAMAITSSAMAITS
10.710.770.340.410.070.09
20.720.840.340.580.070.18
40.700.880.340.640.060.26
80.710.890.330.660.080.33
120.720.910.360.750.060.33
Appendix
Table 8: Detection performance with different sampling rounds TT .
λ\lambdaAUCT@10%T@1%
0.000.840.590.17
0.050.870.630.24
0.100.880.640.26
0.150.870.610.25
0.200.870.630.25
0.300.870.640.25
Appendix
Table 9: Detection performance with different diversity-promoting penalty λ\lambda .
Figure 7: Detection performance under different temperature.
ttAUCT@10%T@1%
SAMAITSSAMAITSSAMAITS
0.750.690.860.290.620.060.26
0.800.690.860.310.600.060.23
0.900.690.870.300.640.080.26
0.950.700.880.300.640.070.26
1.000.700.880.340.640.060.26
Appendix
Table 10: Detection performance with different temperature.
More in this topic

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 280 papers

Explore similar work

CardsList

  1. DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention

    Mar 9, 2026Younjoo Lee, Seungkyun Dan, Junghoo Lee +2Diffusion Language Model InferenceDiffusion Language Model Decoding

    Seoul National University, Seoul, Republic of Korea

  2. Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models

    Jun 10, 2026Jia Deng, Junyi Li, Wayne Xin Zhao +3Masked Diffusion ModelsDiffusion Model Fine-Tuning

    Gaoling School of Artificial Intelligence, Renmin University of China · Department of Data Science, City University of Hong Kong · Beijing Key Laboratory of Research on Large Models and Intelligent Governance +2

  3. Mask Is What DLLM Needs: A Masked Data Training Paradigm for Diffusion LLMs

    Mar 16, 2026Linrui Ma, Yufei Cui, Kai Han +1Diffusion Language ModelsBlockwise Diffusion Language Models

    Noah’s Ark Lab, Huawei Montreal, Canada & Beijing, China