SynBench: A Benchmark for Differentially Private Text Generation
Authors: Yidan Sun, Viktor Schlegel, Srinivasan Nandakumar, Iqra Zahid, Yuping Wu, Yulong Wu, Hao Li, Jie Zhang, Warren Del-Pinto, Goran Nenadic, Siew Kei Lam, Anil Anthony Bharath
Organizations: Imperial College London, Imperial Global Singapore · University of Manchester, United Kingdom · CFAR and IHPC, Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore · Imperial College London, United Kingdom
Synthetic text generation with Differential Privacy (DP) guarantees emerges as a principled approach that can enable the sharing of sensitive datasets across institutional and regulatory boundaries, while bounding the risks of re-identification and membership inference. LLM-based methods deliver promising results; however, comparisons are exacerbated by differing evaluation setups and "private" datasets, potential pre-training contamination is not considered and guarantees are not verified with DP audits. To advance this field, we introduce a unified evaluation framework with standardised utility and fidelity metrics and privacy audits, encompassing nine curated datasets that capture domain-specific complexities such as technical jargon, long-context dependencies, and specialised document structures. In a large-scale empirical study, we benchmark LLM-based state-of-the-art DP text generators of varying sizes (between 1--8B). Our results indicate that DP synthetic text generation remains an unsolved challenge, with quality deteriorating more as the private datasets deviate further from the generators' pre-training corpora. Our analyses suggest that synthetic data quality may be overestimated when LLMs have been pre-trained---without DP---on portions of the "private" data to be generated, and our membership inference (MIA) auditing setup for synthetic text reveals measurable membership signals even under tight privacy budgets. Finally, we note that under the "public pre-training, private generation" paradigm, the nominal (ε,δ) guarantee does not account for exposure during pre-training, motivating contamination-aware evaluation of DP synthetic text generation. Code and data splits: https://github.com/ImperialGlobalSingapore/synbench
Figures & tables
Figure 1
Dataset
∣C∣
∣D∣
∣L∣
Access mechanism
HoC
10
10301
37/58
free on HF
N2C2’08
16
620
1985/3070
DUA & manual approval
PsyTAR
7
5102
19/34
DUA
Mimic
21
4971
2726/3951
DUA & credentialed access
DMSAFN
3
3876
30/50
free on HF
Lc37Fnd
3
5620
836/1510
free on HF
Table 1: Benchmark characteristics: Number of labels ( ∣C∣ ), dataset size ( ∣D∣ ), avg/p90 token length ∣L∣ and data access. HF: huggingface. DUA: data usage agreement.
Dataset HoC ( Random : 3.7, Majority : 9.1)
BERT-large (original=71.7 ± 2.2 )
BioClinicalBERT (original=67.6 ± 5.8 )
BiomedBERT (original=74.0 ± 0.7 )
Method
ϵ=∞
ϵ=0.5
ϵ=1
ϵ=2
ϵ=4
ϵ=∞
ϵ=0.5
ϵ=1
ϵ=2
ϵ=4
ϵ=∞
ϵ=0.5
ϵ=1
ϵ=2
ϵ=4
DP-Gen
48.2
11.2
9.7
13.0
11.6
45.3
10.0
9.5
11.9
12.2
53.0
12.7
12.6
17.0
16.8
± 3.9
± 4.7
± 3.2
± 4.0
± 0.5
± 3.6
± 2.6
± 3.7
± 2.7
± 0.5
± 1.7
± 3.6
± 4.3
± 8.9
± 2.0
AUG-PE
11.0
3.8
5.4
4.4
7.1
10.4
2.9
6.7
5.2
8.3
16.9
7.0
6.9
5.5
8.2
± 6.5
± 0.4
± 4.9
± 1.6
± 2.5
± 6.2
± 3.3
± 3.4
± 2.3
± 1.4
± 7.0
± 4.5
± 2.0
± 1.9
± 6.7
Table 2: Random/majority guess baselines, and downstream classification micro-F1 scores for three models per dataset, trained on synthetic data from Llama-3.2-1B across different privacy budgets ( ϵ ): biomedical datasets. Each value is the mean over 3 seeds (small type: ± the 95% confidence interval (t-distribution)). “–”: no result (DP-Dfs settings whose synthetic data were unusable). Bold: the best method in each column; grey: not significantly above Majority (paired bootstrap over test examples, one-sided, Benjamini-Hochberg-corrected p≥0.05 ).
Dataset DMSAFN ( Random : 30.5, Majority : 41.1)
ProsusAI-finbert (original=79.6 ± 6.4 )
finbert-tone (original=73.4 ± 2.8 )
FinancialBERT (original=92.0 ± 3.9 )
Method
ϵ=∞
ϵ=0.5
ϵ=1
ϵ=2
ϵ=4
ϵ=∞
ϵ=0.5
ϵ=1
ϵ=2
ϵ=4
ϵ=∞
ϵ=0.5
ϵ=1
ϵ=2
ϵ=4
DP-Gen
69.4
43.2
49.8
44.3
48.1
67.2
38.9
47.4
37.6
38.4
81.4
51.8
60.1
53.7
56.5
± 22.9
± 2.6
± 5.8
± 17.8
± 12.0
± 3.8
± 2.7
± 1.1
± 5.1
± 13.9
± 3.9
± 3.4
± 8.7
± 10.4
± 3.7
AUG-PE
46.4
50.2
45.4
48.5
47.7
43.9
44.8
42.1
43.1
44.2
52.6
51.1
51.6
54.4
53.4
± 1.8
± 15.5
± 3.7
± 1.3
± 1.9
± 0.4
± 4.6
± 0.3
± 0.7
± 1.6
± 1.7
± 13.0
± 8.9
± 9.3
± 8.7
Table 3: As Table 2 , for the finance and legal datasets.
Method
ϵ=∞
ϵ=4
ϵ=2
ϵ=1
ϵ=0.5
M↑ /
N↓ /
L↓
M↑ /
N↓ /
L↓
M↑ /
N↓ /
L↓
M↑ /
N↓ /
L↓
M↑ /
N↓ /
L↓
HoC
( Original : 0.99 / 2.19 / 0.004)
DP-Gen
0.65 /
3.71 /
0.06
0.18 /
5.26 /
0.40
0.16 /
5.23 /
0.417
0.14 /
5.18 /
0.44
0.12 /
5.21 /
0.47
AUG-PE
0.01 /
3.80 /
1.49
0.01 /
3.92 /
1.40
0.01 /
4.06 /
1.35
0.01 /
4.45 /
1.25
0.01 /
4.72 /
1.25
DP-Dfs
0.01 /
4.70 /
0.06
0.01 /
6.49 /
0.36
0.01 /
6.62 /
0.37
0.01 /
6.855 /
0.42
0.01 /
6.972 /
1.04
EPSVec
0.01 /
3.38 /
2.74
0.01 /
3.76 /
2.85
0.01 /
4.48 /
3.56
0.01 /
3.39 /
2.78
0.01 /
4.13 /
3.13
Table 4: Fidelity of Llama-3.2-1B synthetic data to the real test set: MAUVE ( M ), KL divergence of NER-entity 1-gram distributions ( N ) and of text-length distributions ( L ). Bold: the best method per dataset, ϵ and metric (none when all methods tie). “–”: not available (DP-Dfs settings whose synthetic data were unusable).
Figure 3: The ROC curves illustrate the performance of membership inference attacks (MIA) on synthetic data generated by AUG-PE and DP-gen. The colourful dashed lines represent the theoretical upper bounds for each corresponding privacy parameter ϵ from [ 25 ] . Zoomed sections highlight FPR ranges where the empirical ROC curve lies above the theoretical bound.
Figure 7
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Size
ε=∞
ε=4
ε=2
ε=1
ε=0.5
PsyTAR
1B
76.2 ± 0.2
35.4 ± 3.6
36.9 ± 3.2
35.2 ± 4.4
32.5 ± 2.2
3B
76.4 ± 1.0
41.1 ± 1.4
37.1 ± 2.5
31.9 ± 3.9
25.2 ± 3.9
8B
77.4 ± 0.6
43.1 ± 2.9
43.9 ± 2.2
42.0 ± 5.4
35.6 ± 1.9
N2C2’08
1B
59.1 ± 14.0
68.6 ± 1.0
69.9 ± 3.4
71.7 ± 3.6
72.7 ± 1.8
3B
65.9 ± 15.0
70.8 ± 7.8
73.0 ± 2.2
70.3 ± 3.3
70.6 ± 3.3
8B
76.4 ± 8.4
69.6 ± 1.5
70.7 ± 2.2
71.5 ± 6.2
69.5 ± 5.6
Appendix
Table 6: Micro-F1 (best classifier) of synthetic data from DP-fine-tuned Llama models of increasing size. Subscripts: 95% confidence interval over 3 seeds.
Dataset
Classifier
Original
ϵ=∞
ϵ=0.5
ϵ=1
ϵ=2
ϵ=4
HoC R 3.7 / M 9.1
BERT-large
71.7 ± 2.2
65.3 ± 1.8
27.6 ± 6.4
25.0 ± 2.9
25.2 ± 4.1
25.5 ± 6.1
BioClinicalBERT
67.6 ± 5.8
61.4 ± 3.3
25.9 ± 2.5
23.8 ± 2.8
25.9 ± 2.2
22.0 ± 2.9
BiomedBERT
74.0 ± 0.7
65.4 ± 1.3
29.4 ± 4.0
26.4 ± 2.0
29.0 ± 1.0
28.6 ± 1.9
N2C2’08 R 46.1 / M 53.2
LongFormer-large
82.6 ± 14.5
68.9 ± 22.0
55.9 ± 8.4
54.0 ± 6.5
55.1 ± 5.5
56.0 ± 12.6
Clinical-BigBird
87.4 ± 2.4
72.7 ± 6.2
52.0 ± 12.4
59.6 ± 1.9
52.0 ± 10.9
50.3 ± 10.5
Clinical-LongFormer
77.9 ± 43.5
73.6 ± 9.0
52.4 ± 1.0
54.9 ± 2.1
53.1 ± 11.9
54.4 ± 4.5
Appendix
Table 7: Downstream classification micro-F1 scores with synthetic data from OLMo-2-0425-1B (fine-tuned for 10 epochs) generated with DP-Gen, across privacy budgets ( ϵ ); “Original” trains on the real data. R / M: random / majority guess baselines. Each value is the mean over 3 seeds ± the 95% confidence interval (t-distribution).
Figure 5: Utility of OLMo-2-1B DP synthetic data ( ε=4 ) for HoC against the share x of the private training set ( N=4,000 ) that OLMo-2 saw in mid-training (dolmino-mix-1124). Each point is one DP generator (303 from three independent replicates), scored by a classifier trained on its synthetic data and tested on the real HoC test set, as the mean or maximum over 3 classifier seeds. Micro-F1 is rescaled per classifier to the percentage of the gain over random guessing (0.037) that the same classifier achieves when trained on the real 4,000 records (0.719 BiomedBERT, 0.712 BioLinkBERT). Red: least-squares fit; p : one-sided permutation test of a positive slope.
Classifier (seeds)
x=0→100
Δ
p
p (collapsed =0 )
BiomedBERT (mean)
41.9 → 43.2
+1.3
0.002
0.003
BiomedBERT (max)
43.7 → 45.2
+1.5
0.0009
0.0009
BioLinkBERT (mean)
39.4 → 42.8
+3.4
< 0.0001
0.009
BioLinkBERT (max)
42.9 → 45.4
+2.5
0.0002
0.0003
all four (overall)
< 0.0001
0.0001
Appendix
Table 8: Rescaled downstream micro-F1 (%) of OLMo-2-1B DP synthetic data ( ε=4 , 303 generators) for HoC as the share x of private records seen in OLMo-2 mid-training grows from 0% to 100% ( N=4,000 ): least-squares fit at the two ends, its change, and one-sided permutation p of a positive slope (20,000 shuffles of x within each of the three replicates). Collapsed classifier runs (micro-F1 0) are left out, or counted as 0 in the last column. Overall: one test of all four views at once (weakest standardised slope).
Dataset
Regime
RMIA
LiRA
Logits
DMSAFN
SYN (No DP)
0.60
0.49
0.60
DMSAFN
SYN ( ϵ=2 )
0.54
0.45
0.48
DMSAFN
PRI (No DP)
0.45
0.50
0.54
DMSAFN
PRI ( ϵ=2 )
0.49
0.49
0.54
PsyTAR
SYN (No DP)
0.65
0.61
0.62
PsyTAR
SYN ( ϵ=2 )
0.57
0.61
0.61
Appendix
Table 9: MIA AUC across attacks and training regimes. SYN denotes a downstream classifier trained on synthetic data; PRI denotes training on private data. “No DP” vs. ϵ=2 refers to whether DP is applied in the relevant training/generation step described in Section 4 . Higher AUC indicates stronger membership distinguishability ( 0.5 is chance).
Figure 6: Empirical ε implied by the bigram reference attack along its ROC (mean over 50 repeats; bands: 2.5–97.5% quantiles; dashed: guaranteed ε ). Hatched: α<0.1 or α>0.9 , where the estimate rests on ≤ 5 records per repeat. Circles: the 2.5% quantile exceeds the guaranteed ε .
Figure 7: Metric trends across epsilon levels for AUG-PE and DP-Gen Methods. Each subplot shows how a specific metric changes with increasing epsilon levels. Solid blue lines represent AUG-PE and dashed orange lines represent DP-Gen, with shaded areas indicating variance across datasets. Metrics are reported across 9 datasets.
Differentially private (DP) text synthesis promises to unlock sensitive corpora for model training, but it remains unclear whether DP synthetic data transmits genuinely new knowledge and capabilities present only in those corpora. This is because existing evaluations rely on tasks that are nearly solvable without training, so strong benchmark performance does not establish that DP synthesis can substitute original data access. Thus, we introduce ContinuousBench, a continuously and automatically-regenerated benchmark that measures capability gain from DP synthetic text. Each quarter, a new release pairs a never-before-seen training corpus with a derived QA set, constructed to be: (1) unsolvable sans-corpus; and (2) learnable under DP, as the tested knowledge is supported by hundreds of independent records. Researchers produce DP synthetic data from the training corpus and run our standardized training and evaluation harness on their synthetic data to measure gains. We instantiate two tracks: Geminon, a procedurally-generated dataset about fictional creatures; and News, a stream of newly crawled public news articles. Although standard benchmarks are nearly saturated, on ContinuousBench we find that non-private synthesis transfers substantial knowledge from the original corpus, while state-of-the-art DP synthesis methods generally fail to do so, even at ε=100.
Peihan Liu, Lucas Rosenblatt, Weiwei Kong +7
Columbia University · Work done as a Student Researcher at Google · NYU +4
The rapid evolution of Large Language Models is fueled by massive datasets, yet this reliance poses severe privacy risks, as models can inadvertently memorize and even directly output sensitive training data. Synthetic text generation offers a promising solution to this challenge; however, existing methods struggle to reconcile rigorous privacy guarantees with high data utility. In this paper, we propose a new paradigm for private synthetic text generation: Guided Diffusion. Leveraging the unique steerability of Diffusion Language Models, we introduce novel sampling and re-masking procedures that actively guide the iterative denoising steps. These mechanisms steer the generative process to reconstruct fluent text that adheres to a target semantic distribution. We demonstrate that our framework is naturally compatible with Differential Privacy (DP): once the semantic embeddings of the private corpus are protected, the DP guarantee carries over to the generated text by post-processing. Empirically, we show that our approach generates high-quality synthetic text with precise semantic alignment. Furthermore, we validate its effectiveness in DP synthetic text generation: across two datasets and two diffusion backbones, it outperforms Private Evolution and DP-SGD-based fine-tuning in distributional fidelity and downstream utility, while generating significantly faster than Private Evolution. We further extend the framework to Data-Specific Indistinguishability (DSI), which adds a formal certificate against data poisoning and memorization at a small utility cost.
Qi Pang, Hanshen Xiao, Andre Manoel +4
NVIDIA · Carnegie Mellon University · Purdue University
Differentially private (DP) synthetic data generation plays a pivotal role in developing large language models (LLMs) on private data, where data owners cannot provide eyes-on access to individual examples. Generating DP synthetic data typically involves a difficult trade-off. On one hand, DP finetuning methods train an LLM as a synthetic data generator with formal privacy guarantees, yet it still requires the raw content of private examples for model training. However, methods that avoid direct exposure to private data are bounded by an off-the-shelf, un-finetuned model, whose outputs often lack domain fidelity. Can we train an LLM to generate high-quality synthetic text without eyes-on access to individual private examples? In this work, we introduce Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm for synthetic data generation with LLMs. DP-RFT leverages DP-protected nearest-neighbor votes from an eyes-off private corpus as a reward signal for on-policy synthetic samples generated by an LLM. The LLM iteratively learns to generate synthetic data to maximize the expected DP votes through Proximal Policy Optimization (PPO). We evaluate DP-RFT for long-form and domain-specific synthetic data generation, such as news articles, meeting transcripts, and medical article abstracts. Our experiments show that DP-RFT closes the gap between private evolution and DP finetuning methods in terms of the fidelity and downstream utility of the generated synthetic data, while respecting the private data boundary.
Fangyuan Xu, Sihao Chen, Zinan Lin +13
New York University · Microsoft · University of Southern California