The rapid evolution of Large Language Models is fueled by massive datasets, yet this reliance poses severe privacy risks, as models can inadvertently memorize and even directly output sensitive training data. Synthetic text generation offers a promising solution to this challenge; however, existing methods struggle to reconcile rigorous privacy guarantees with high data utility. In this paper, we propose a new paradigm for private synthetic text generation: Guided Diffusion. Leveraging the unique steerability of Diffusion Language Models, we introduce novel sampling and re-masking procedures that actively guide the iterative denoising steps. These mechanisms steer the generative process to reconstruct fluent text that adheres to a target semantic distribution. We demonstrate that our framework is naturally compatible with Differential Privacy (DP): once the semantic embeddings of the private corpus are protected, the DP guarantee carries over to the generated text by post-processing. Empirically, we show that our approach generates high-quality synthetic text with precise semantic alignment. Furthermore, we validate its effectiveness in DP synthetic text generation: across two datasets and two diffusion backbones, it outperforms Private Evolution and DP-SGD-based fine-tuning in distributional fidelity and downstream utility, while generating significantly faster than Private Evolution. We further extend the framework to Data-Specific Indistinguishability (DSI), which adds a formal certificate against data poisoning and memorization at a small utility cost.
Figures & tables
Figure 1: High-level denoising pipeline for DLMs. At each step t , the model predicts token probabilities for the masked input xt . A candidate sequence xt−1′ is sampled, and low-confidence tokens are re-masked to produce the refined state xt−1 .
Figure 2: Evaluation of semantic alignment and generation quality. Left: Cosine similarity between generated and target embeddings. Guided Diffusion significantly improves alignment compared to unguided generation. Right: Perplexity (PPL) under Qwen3-8B of the original references (Orig.) and of guided and unguided generations. Guided text remains fluent: on IMDB its PPL matches unguided generation, and on Yahoo it is higher than unguided generation but still well below the references.
Method
Yahoo
IMDB
Guided Diffusion (Full)
0.90 ±0.04
0.59 ±0.11
w/o semantic-aware re-masking
0.79 ±0.02
0.35 ±0.10
w/o semantic-aware sampling
0.80 ±0.02
0.34 ±0.10
Table 1: Ablation of semantic-aware sampling and re-masking. We report the cosine similarity between generated and target embeddings on Yahoo and IMDB as mean ±std ; bold marks the best result. Disabling either substantially degrades alignment, confirming both stages are essential.
Figure 3: Trustworthy synthetic data generation with Guided Diffusion. For DP, the input is a private dataset, and the trustworthy aggregation is (ε,δ) -DP aggregation. For DSI (§ 5 ), the input is a dataset collected from multiple, potentially malicious sources, and the trustworthy aggregation is DSI aggregation.
Distributional fidelity
Backbone
Method
Utility (%) ↑
FD ↓
JSD ↓
Dist-2 ↑
MAUVE ↑
ICLR26 (utility: macro-F1)
LLaDA-8B- Instruct
PE
13.34 ±1.38
0.7753 ±0.0032
0.3401 ±0.0006
0.0730
0.6366 ±0.0054
DP-SGD+LoRA
20.12 ±4.16
0.6026 ±0.0015
0.3523 ±0.0007
0.0431
0.6475 ±0.0031
Guided Diffusion
24.85 ±2.80
0.3995 ±0.0020
0.2787 ±0.0008
0.0891
0.6923 ±0.0042
Dream-7B
PE
12.49 ±0.57
0.7531 ±0.0041
0.3060 ±0.0005
0.0785
0.6409 ±0.0079
Table 2: DP synthetic text generation at (ε,δ)=(1,10−6) with 2,000 synthetic samples, reported as mean ±std over four seeds. Utility is macro-F1 on ICLR26 and causal next-token accuracy on PubMed . Bold marks the best result per dataset and backbone.
Method
Accuracy (%)
Guided Diffusion (Full)
32.68
w/o semantic-aware sampling
18.01
w/o semantic-aware re-masking
26.73
Table 3: Ablation of semantic-aware sampling and re-masking on PubMed with LLaDA-8B-Instruct under ε=1 .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Method
ε
Accuracy (%)
Without fine-tuning
0.07
Fine-tuning on real data (non-DP)
34.60
PE †
1
27.90
∞
28.60
DP-SGD-FT (GPT-2 Large) †
1
31.10
∞
33.10
Appendix
Table 4: Effect of the privacy budget on PubMed accuracy, with δ=10−6 . Guided Diffusion uses LLaDA-8B-Instruct . † Results from PE’s official implementation, where each baseline uses its default generator.
Synthetic samples
Macro-F1 (%)
100
13.13 ±5.59
500
20.46 ±6.30
1,000
20.83 ±5.71
2,000
24.85 ±2.80
Appendix
Table 5: Effect of the number of synthetic samples on ICLR26 ( LLaDA-8B-Instruct , ε=1 ), reported as mean ±std .
Setting
Guide sim. ↑
MPNet sim. ↑
PPL ↓
Default ( T=448 , α=20 , b/a=0.1 )
0.6920
0.5396
4.218
T=224
0.6756
0.5335
7.948
α=5
0.6929
0.5345
4.482
α=10
0.6930
0.5405
4.352
b/a=0.2
0.6919
0.5385
4.247
b/a=0.5
0.6910
0.5415
4.203
Appendix
Table 6: Effect of generation hyperparameters on ICLR26 . Each row changes one hyperparameter from the default.
Measurement
Value
Wall-clock time per 16 -sample batch
631.86 s
Amortized time per sample
39.49 s
2,000 samples on 1 / 8 A100s
21.94 h / 2.74 h
Peak allocated memory
16.35 GiB
DLM forward passes per batch
448
Logical encoder calls / internal micro-batches per batch
897 / 10,527
Appendix
Table 7: Runtime profile of Guided Diffusion ( LLaDA-8B-Instruct , batch size 16 , BF16, one A100-SXM4-40GB).
Method
ACC (%)
ASR (%)
Orig-benign
47.80
0
Orig-backdoor
48.82
100
Synthetic-w/o-DSI
39.85
0
Synthetic-w/-DSI
37.74
0
Appendix
Table 8: Evaluation against a semantic backdoor attack. ASR is measured on 60 held-out triggered examples. DSI Guided Diffusion preserves the empirical 0% ASR while adding a formal indistinguishability certificate.
Method
Max ROUGE-L ↓
Semantic Alignment ↑
Synthetic-w/o-DSI
0.1446 ±0.0223
0.80 ±0.07
Synthetic-w/-DSI
0.1460 ±0.0187
0.76 ±0.07
Appendix
Table 9: Memorization evaluation on Yelp 1-star Grocery reviews. Max ROUGE-L below 0.2 indicates low memorization. DSI Guided Diffusion provides formal indistinguishability with a small utility cost. Results are reported as mean ±std .
Differentially private (DP) synthetic data generation plays a pivotal role in developing large language models (LLMs) on private data, where data owners cannot provide eyes-on access to individual examples. Generating DP synthetic data typically involves a difficult trade-off. On one hand, DP finetuning methods train an LLM as a synthetic data generator with formal privacy guarantees, yet it still requires the raw content of private examples for model training. However, methods that avoid direct exposure to private data are bounded by an off-the-shelf, un-finetuned model, whose outputs often lack domain fidelity. Can we train an LLM to generate high-quality synthetic text without eyes-on access to individual private examples? In this work, we introduce Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm for synthetic data generation with LLMs. DP-RFT leverages DP-protected nearest-neighbor votes from an eyes-off private corpus as a reward signal for on-policy synthetic samples generated by an LLM. The LLM iteratively learns to generate synthetic data to maximize the expected DP votes through Proximal Policy Optimization (PPO). We evaluate DP-RFT for long-form and domain-specific synthetic data generation, such as news articles, meeting transcripts, and medical article abstracts. Our experiments show that DP-RFT closes the gap between private evolution and DP finetuning methods in terms of the fidelity and downstream utility of the generated synthetic data, while respecting the private data boundary.
Fangyuan Xu, Sihao Chen, Zinan Lin +13
New York University · Microsoft · University of Southern California
Large language models (LLMs) have emerged as a powerful tool for synthetic data generation. A particularly important use case is producing synthetic replicas of private text, which requires carefully balancing privacy and utility. We propose Realistic and Privacy-Preserving Synthetic Data Generation (RPSG), which uses private seeds and integrates privacy-preserving strategies, including a formal differential privacy (DP) mechanism in the candidate selection, to generate realistic synthetic data. Comprehensive experiments against state-of-the-art private synthetic data generation methods demonstrate that RPSG achieves high fidelity to private data while providing strong privacy protection.
Qian Ma, Sarah Rajtmajer
Information Sciences and Technology, The Pennsylvania State University
Differentially private (DP) text synthesis promises to unlock sensitive corpora for model training, but it remains unclear whether DP synthetic data transmits genuinely new knowledge and capabilities present only in those corpora. This is because existing evaluations rely on tasks that are nearly solvable without training, so strong benchmark performance does not establish that DP synthesis can substitute original data access. Thus, we introduce ContinuousBench, a continuously and automatically-regenerated benchmark that measures capability gain from DP synthetic text. Each quarter, a new release pairs a never-before-seen training corpus with a derived QA set, constructed to be: (1) unsolvable sans-corpus; and (2) learnable under DP, as the tested knowledge is supported by hundreds of independent records. Researchers produce DP synthetic data from the training corpus and run our standardized training and evaluation harness on their synthetic data to measure gains. We instantiate two tracks: Geminon, a procedurally-generated dataset about fictional creatures; and News, a stream of newly crawled public news articles. Although standard benchmarks are nearly saturated, on ContinuousBench we find that non-private synthesis transfers substantial knowledge from the original corpus, while state-of-the-art DP synthesis methods generally fail to do so, even at ε=100.
Peihan Liu, Lucas Rosenblatt, Weiwei Kong +7
Columbia University · Work done as a Student Researcher at Google · NYU +4