Guided Diffusion for Trustworthy Synthetic Text Generation

Qi Pang; Hanshen Xiao; Andre Manoel; Lipika Ramaswamy; Wenting Zheng; Virginia Smith; G. Edward Suh

cs.CROct 19, 2026

Guided Diffusion for Trustworthy Synthetic Text Generation

Authors: Qi Pang, Hanshen Xiao, Andre Manoel, Lipika Ramaswamy, Wenting Zheng, Virginia Smith, G. Edward Suh

Organizations: NVIDIA · Carnegie Mellon University · Purdue University

Abstract

The rapid evolution of Large Language Models is fueled by massive datasets, yet this reliance poses severe privacy risks, as models can inadvertently memorize and even directly output sensitive training data. Synthetic text generation offers a promising solution to this challenge; however, existing methods struggle to reconcile rigorous privacy guarantees with high data utility. In this paper, we propose a new paradigm for private synthetic text generation: Guided Diffusion. Leveraging the unique steerability of Diffusion Language Models, we introduce novel sampling and re-masking procedures that actively guide the iterative denoising steps. These mechanisms steer the generative process to reconstruct fluent text that adheres to a target semantic distribution. We demonstrate that our framework is naturally compatible with Differential Privacy (DP): once the semantic embeddings of the private corpus are protected, the DP guarantee carries over to the generated text by post-processing. Empirically, we show that our approach generates high-quality synthetic text with precise semantic alignment. Furthermore, we validate its effectiveness in DP synthetic text generation: across two datasets and two diffusion backbones, it outperforms Private Evolution and DP-SGD-based fine-tuning in distributional fidelity and downstream utility, while generating significantly faster than Private Evolution. We further extend the framework to Data-Specific Indistinguishability (DSI), which adds a formal certificate against data poisoning and memorization at a small utility cost.

Figures & tables

Figure 1: High-level denoising pipeline for DLMs. At each step $t$ , the model predicts token probabilities for the masked input ${\bm{x}}_{t}$ . A candidate sequence ${\bm{x}}^{\prime}_{t-1}$ is sampled, and low-confidence tokens are re-masked to produce the refined state ${\bm{x}}_{t-1}$ .

Figure 2: Evaluation of semantic alignment and generation quality. Left: Cosine similarity between generated and target embeddings. Guided Diffusion significantly improves alignment compared to unguided generation. Right: Perplexity (PPL) under Qwen3-8B of the original references (Orig.) and of guided and unguided generations. Guided text remains fluent: on IMDB its PPL matches unguided generation, and on Yahoo it is higher than unguided generation but still well below the references.

Method	Yahoo	IMDB
Guided Diffusion (Full)	0.90 ±0.04	0.59 ±0.11
w/o semantic-aware re-masking	0.79 ±0.02	0.35 ±0.10
w/o semantic-aware sampling	0.80 ±0.02	0.34 ±0.10

Table 1: Ablation of semantic-aware sampling and re-masking. We report the cosine similarity between generated and target embeddings on Yahoo and IMDB as mean

{}_{\pm\text{std}}

; bold marks the best result. Disabling either substantially degrades alignment, confirming both stages are essential.

Figure 3: Trustworthy synthetic data generation with Guided Diffusion. For DP, the input is a private dataset, and the trustworthy aggregation is $(\varepsilon,\delta)$ -DP aggregation. For DSI (§ 5 ), the input is a dataset collected from multiple, potentially malicious sources, and the trustworthy aggregation is DSI aggregation.

ICLR26 (utility: macro-F1)
			Distributional fidelity
Backbone	Method	Utility (%) $\uparrow$	FD $\downarrow$	JSD $\downarrow$	Dist-2 $\uparrow$	MAUVE $\uparrow$
LLaDA-8B- Instruct	PE	13.34 ±1.38	0.7753 ±0.0032	0.3401 ±0.0006	0.0730	0.6366 ±0.0054
	DP-SGD+LoRA	20.12 ±4.16	0.6026 ±0.0015	0.3523 ±0.0007	0.0431	0.6475 ±0.0031
	Guided Diffusion	24.85 ±2.80	0.3995 ±0.0020	0.2787 ±0.0008	0.0891	0.6923 ±0.0042
Dream-7B	PE	12.49 ±0.57	0.7531 ±0.0041	0.3060 ±0.0005	0.0785	0.6409 ±0.0079

Table 2: DP synthetic text generation at

(\varepsilon,\delta)=(1,10^{-6})

with

2{,}000

synthetic samples, reported as mean

{}_{\pm\text{std}}

over four seeds. Utility is macro-F1 on ICLR26 and causal next-token accuracy on PubMed . Bold marks the best result per dataset and backbone.

Method	Accuracy (%)
Guided Diffusion (Full)	32.68
w/o semantic-aware sampling	18.01
w/o semantic-aware re-masking	26.73

Table 3: Ablation of semantic-aware sampling and re-masking on PubMed with LLaDA-8B-Instruct under

\varepsilon=1

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$\bm{\varepsilon}$	Accuracy (%)
Without fine-tuning		0.07
Fine-tuning on real data (non-DP)		34.60
PE †	$1$	27.90
PE †	$\infty$	28.60
DP-SGD-FT (GPT-2 Large) †	$1$	31.10
DP-SGD-FT (GPT-2 Large) †	$\infty$	33.10

Appendix

Table 4: Effect of the privacy budget on PubMed accuracy, with

\delta=10^{-6}

. Guided Diffusion uses LLaDA-8B-Instruct . † Results from PE’s official implementation, where each baseline uses its default generator.

Synthetic samples	Macro-F1 (%)
$100$	13.13 ±5.59
$500$	20.46 ±6.30
$1{,}000$	20.83 ±5.71
$2{,}000$	24.85 ±2.80

Appendix

Table 5: Effect of the number of synthetic samples on ICLR26 ( LLaDA-8B-Instruct ,

\varepsilon=1

), reported as mean

{}_{\pm\text{std}}

Setting	Guide sim. $\uparrow$	MPNet sim. $\uparrow$	PPL $\downarrow$
Default ( $T=448$ , $\alpha=20$ , $b/a=0.1$ )	0.6920	0.5396	4.218
$T=224$	0.6756	0.5335	7.948
$\alpha=5$	0.6929	0.5345	4.482
$\alpha=10$	0.6930	0.5405	4.352
$b/a=0.2$	0.6919	0.5385	4.247
$b/a=0.5$	0.6910	0.5415	4.203

Appendix

Table 6: Effect of generation hyperparameters on ICLR26 . Each row changes one hyperparameter from the default.

Measurement	Value
Wall-clock time per $16$ -sample batch	631.86 s
Amortized time per sample	39.49 s
$2{,}000$ samples on 1 / 8 A100s	21.94 h / 2.74 h
Peak allocated memory	16.35 GiB
DLM forward passes per batch	448
Logical encoder calls / internal micro-batches per batch	897 / 10,527

Appendix

Table 7: Runtime profile of Guided Diffusion ( LLaDA-8B-Instruct , batch size

16

, BF16, one A100-SXM4-40GB).

Method	ACC (%)	ASR (%)
Orig-benign	47.80	0
Orig-backdoor	48.82	100
Synthetic-w/o-DSI	39.85	0
Synthetic-w/-DSI	37.74	0

Appendix

Table 8: Evaluation against a semantic backdoor attack. ASR is measured on

60

held-out triggered examples. DSI Guided Diffusion preserves the empirical 0% ASR while adding a formal indistinguishability certificate.

Method	Max ROUGE-L $\downarrow$	Semantic Alignment $\uparrow$
Synthetic-w/o-DSI	0.1446 ±0.0223	0.80 ±0.07
Synthetic-w/-DSI	0.1460 ±0.0187	0.76 ±0.07

Appendix

Table 9: Memorization evaluation on Yelp 1-star Grocery reviews. Max ROUGE-L below

0.2

indicates low memorization. DSI Guided Diffusion provides formal indistinguishability with a small utility cost. Results are reported as mean

{}_{\pm\text{std}}

Synthetic Data Generation for Language Models

Synthetic data generation for language models creates artificial text, tasks, and interaction traces to train or evaluate models.

See all 292 papers

DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning
Feb 20, 2026Fangyuan Xu, Sihao Chen, Zinan Lin +13Differentially Private Synthetic Data Generation Synthetic Data Generation for Language Models
New York University · Microsoft · University of Southern California
Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation
Apr 8, 2026Qian Ma, Sarah RajtmajerPrivacy-Preserving Language Models Synthetic Data Generation
Information Sciences and Technology, The Pennsylvania State University
ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?
Jun 1, 2026Peihan Liu, Lucas Rosenblatt, Weiwei Kong +7Privacy-Preserving Language Models Synthetic Benchmark Generation
Columbia University · Work done as a Student Researcher at Google · NYU +4

Feb 20, 2026J/K move · Enter open · S save

DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning

Fangyuan Xu, Sihao Chen, Zinan Lin, Taiwei Shi+12

New York University · Microsoft · University of Southern California

Differentially Private Synthetic Data Generation Synthetic Data Generation for Language Models Privacy-Preserving Language Models Privacy-Preserving ML Differential Privacy RL for Generative Models

Open paper AskPDF arXiv

Differentially private (DP) synthetic data generation plays a pivotal role in developing large language models (LLMs) on private data, where data owners cannot provide eyes-on access to individual examples. Generating DP synthetic data typically involves a difficult trade-off. On one hand, DP finetuning methods train an LLM as a synthetic data generator with formal privacy guarantees, yet it still requires the raw content of private examples for model training. However, methods that avoid direct exposure to private data are bounded by an off-the-shelf, un-finetuned model, whose outputs often lack domain fidelity. Can we train an LLM to generate high-quality synthetic text without eyes-on access to individual private examples? In this work, we introduce Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm for synthetic data generation with LLMs. DP-RFT leverages DP-protected nearest-neighbor votes from an eyes-off private corpus as a reward signal for on-policy synthetic samples generated by an LLM. The LLM iteratively learns to generate synthetic data to maximize the expected DP votes through Proximal Policy Optimization (PPO). We evaluate DP-RFT for long-form and domain-specific synthetic data generation, such as news articles, meeting transcripts, and medical article abstracts. Our experiments show that DP-RFT closes the gap between private evolution and DP finetuning methods in terms of the fidelity and downstream utility of the generated synthetic data, while respecting the private data boundary.

Abstract

Synthetic Data Generation for Language Models

Explore similar work

DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning

Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation

ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?