Methods for training language models on sensitive data differ in how they balance capability gains against the risk of leaking private information. In this paper, we present a controlled comparison of data-level and training-level interventions for navigating the capability-privacy tradeoff on synthetic medical records with canaries to probe for leakage. Direct finetuning performs best on capability metrics but leads to the most privacy leakage, while DP-SGD reduces leakage at substantial cost to capabilities. Generating synthetic data from these records offers a more favorable balance, particularly when guided by a uniqueness-aware instruction to replace private details. Relative to naive paraphrasing, privacy-aware rewriting improves privacy protection across a variety of different rewriting models. We also find that more capable rewriting models generally reduce the risks of downstream privacy leakage, although held-out fit does not improve monotonically and increased test-time compute yields inconsistent gains. Overall, these results highlight privacy-aware rewriting as a promising direction for achieving a more favorable trade-off between capabilities and empirical privacy compared to other training-level methods, and our evaluation framework supports comparisons of future models and methods.
Figures & tables
Figure 1: Experimental framework and privacy–capability trade-off . Left: Experimental framework. Right: Results for finetuning gpt-oss-20b, with each axis averaging ranks across two privacy and two capability metrics. Higher ranks indicate less privacy leakage and greater capabilities. Arrows connect rewriting to privacy-aware rewriting, which improves both privacy and capability.
Method
Prompt
Rewriting
Paraphrase the following clinical record, preserving every medical fact:
Privacy-aware rewriting
Rewrite the following clinical record so it stays realistic and clinically coherent. Replace any detail that is rare, distinctive, or unusual enough that it could help identify this specific patient with a different but plausible fictional value; leave common, generic clinical information unchanged. Use realistic surrogate values, not placeholders.
Royce851 Keeling91 , born on March 14, 1986 , resides at 742 Juniper Hollow Road, Bedford, MA 01730 . The patient has essential hypertension and dental caries. Current medications include lisinopril 10 MG oral tablet, sodium fluoride 0.0272 MG/MG oral gel, and Mabsartinib .
Privacy-aware rewriting output
Avery Morgan , born on June 22, 1985 , resides at 58 Maple Street, Lowell, MA 01852 . The patient has essential hypertension and dental caries. Current medications include lisinopril 10 MG oral tablet, sodium fluoride 0.0272 MG/MG oral gel, and imatinib 400 MG oral tablet .
Table 1: Prompts and examples for rewriting and privacy-aware rewriting. Each prompt is followed by the source record and the completion cue Rewrite: . Bold text marks distinctive details retained by rewriting or replaced by privacy-aware rewriting.
Metric
Input to the model
PPL measured on
Verbatim perplexity
the canary record truncated immediately before the planted medication: “…Amoxicillin 250 MG / Clavulanate 125 MG Oral Tablet; sodium fluoride 0.0272 MG/MG Oral Gel;”
Mabsartinib
Semantic perplexity
“Which medication starting with ‘M’ was prescribed to Royce851 Keeling91?”
Mabsartinib
Held-out record perplexity
an entire record from the held-out set
all example tokens
Medication perplexity
a test record truncated immediately before its final medication: “…Lisinopril 10 MG Oral Tablet; Acetaminophen 500 MG Oral Tablet;”
Ibuprofen
Table 2: Perplexity-based metrics for model privacy and capability. Lower values indicate higher privacy leakage for the first two metrics and better capability for the last two.
Figure 2: Canary privacy leakage across levels of duplication. Higher values indicate less privacy leakage, and error bars show 95% confidence intervals over 20 canaries per group.
Figure 3: Privacy–capability trade-offs across rewriters. Open and filled markers denote rewriting and privacy-aware rewriting, with arrows connecting each pair.
Figure 4: Privacy–capability trade-offs for privacy-aware rewriting with gpt-oss-20b as the target model. The left shows different rewriting models, while the right shows different amounts of reasoning. Error bars denote 95% confidence intervals.
Figure 5: Privacy–capability trade-offs for privacy-aware rewriting with and without a summarization step. Error bars denote 95% confidence intervals.
Figure 6: Comparison of five independent rewrites trained for one epoch each and a single rewrite trained for five epochs. In both panels, the GPT-5.5 single-rewrite, five-epoch condition is normalized to 1× ; all other values are reported as ratios to this reference. Higher values indicate better performance on every axis.
Figure 7: Comparison of LLM-judged scores with measured outcomes using Qwen3.5-4B-Base as the target model. Left: judged quality versus capability. Right: judged privacy versus verbatim perplexity. Point labels indicate the GPT-series rewriter and reasoning-effort setting.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Privacy–capability trade-offs across methods. Arrows connect rewriting to privacy-aware rewriting for the same rewrite model.
Figure 9: Privacy–capability trade-offs across methods. Lines connect rewriting to privacy-aware rewriting for the same rewrite model. Error bars denote 95% confidence intervals.
Figure 10: Privacy–capability trade-offs across methods. Lines connect rewriting to privacy-aware rewriting for the same rewrite model. Error bars denote 95% confidence intervals.
Figure 11: Canary privacy leakage across duplication factors. Rows show Qwen3.5-4B-Base and gpt-oss-20b; columns show semantic perplexity. Higher values indicate less privacy leakage, and error bars show 95% confidence intervals over 20 canaries per group.
Figure 12: Privacy–capability trade-offs for privacy-aware rewriting with Qwen3.5-4B-Base as the target model. The left shows different rewriting models, while the right shows different amounts of reasoning. Error bars denote 95% confidence intervals.
Figure 13: Privacy–capability trade-offs for privacy-aware rewriting with gpt-oss-20b as the target model. The left shows different rewriting models, while the right shows different amounts of reasoning. Error bars denote 95% confidence intervals.
Figure 14: Privacy–capability trade-offs for privacy-aware rewriting with Qwen3.5-4B-Base as the target model. The left shows different rewriting models, while the right shows different amounts of reasoning. Error bars denote 95% confidence intervals.
Figure 15: Privacy–capability trade-offs for privacy-aware rewriting with and without a summarization step. Error bars denote 95% confidence intervals.
Figure 16: Comparison of five independent rewrites trained for one epoch each and a single rewrite trained for five epochs. In both panels, the GPT-5.5 single-rewrite, five-epoch condition is normalized to 1× ; all other values are reported as ratios to this reference. Higher values indicate better performance on every axis.
Figure 17: Comparison of LLM-judged scores with measured outcomes using Qwen3.5-4B-Base as the target model. Left: judged quality versus capability. Right: judged privacy versus verbatim perplexity. Point labels indicate the GPT-series rewriter and reasoning-effort setting.
Figure 18: Comparison of LLM-judged scores with measured outcomes using gpt-oss-20b as the target model. Left: judged quality versus capability. Right: judged privacy versus verbatim perplexity. Point labels indicate the GPT-series rewriter and reasoning-effort setting.
Figure 19: Comparison of LLM-judged scores with measured outcomes using gpt-oss-20b as the target model. Left: judged quality versus capability. Right: judged privacy versus verbatim perplexity. Point labels indicate the GPT-series rewriter and reasoning-effort setting.
Figure 20: The correlation between two privacy metrics. Verbatim versus semantic perplexity of the planted secret for each method configuration. Labeled points are the nine methods; gray points are rewriting and summary variants. The second row zooms into the shaded region below verbatim perplexity 100, where agreement drops to ρ=0.39 and 0.05 .
Figure 21: The correlation between two capability metrics. Held-out record perplexity versus medication perplexity for each method configuration. Labeled points are the nine methods; gray points are rewriting and summary variants. The second row zooms into the shaded region below held-out perplexity 2.5, where agreement is similar ( ρ=0.52 and 0.90 , over 35 and 5 points).
Figure 22: Probability of generating the coined medication. Probability that each target model generates the coined medication given the record up to it, against how many times the canary appeared in training. Each point is the mean over the 20 canaries at that duplication level, with 95% confidence intervals.
Qwen3.5-4B-Base
gpt-oss-20b
Method
Exposure
Rank 1
Exposure
Rank 1
No finetuning
1.49
0%
1.39
0%
Direct fine-tuning
12.22
86%
10.98
73%
Goldfish ( k=4 )
10.38
53%
8.90
37%
DP-SGD ( ε=8 )
1.74
0%
1.56
0%
PII scrubbing
12.16
87%
10.76
68%
Appendix
Table 3: Secret-Sharer exposure of coined medication names. Each planted coined medication is ranked against 6,300 unplanted alternatives by its whole-name conditional likelihood given the original record prefix ending immediately before the medication. We report mean exposure, log2(6,301/rank) , across 100 canaries and the percentage ranked first within their candidate sets (“Rank 1”). Exposure ranges from 0 to 12.62 bits; uniformly random ranks have an expected mean exposure of approximately 1.44 bits. Lower values indicate less exposure.
Method
Prompt
Summary-based rewriting
Summarize the following clinical record in a few sentences, keeping the clinically essential information (active diagnoses, key treatments, current medications):
Privacy-aware summary-based rewriting
Summarize the following clinical record in a few sentences, keeping the clinically essential information. Replace any detail that is rare, distinctive, or unusual enough that it could help identify this specific patient with a different but plausible fictional value; leave common, generic clinical information unchanged. Use realistic surrogate values, not placeholders.
Expansion prompt
Below is a clinical summary of a patient. Reconstruct a complete, realistic patient record consistent with it, in exactly the format of the example records shown. Include every diagnosis and medication the summary mentions; invent realistic demographics and identifiers (name, address, SSN, DOB consistent with the age) — the summary does not contain them; add plausible routine detail where the summary is silent.
Appendix
Table 4: Generation prompts. The two summary-based methods summarize first and then expand the summary back into a full record with the expansion prompt, which additionally receives two format exemplars drawn from a fresh Synthea seed, never from the private cohort. The privacy-aware summary prompt is the generic summary instruction followed by the privacy-aware rewriting instruction.
Method
What reaches training (full text)
Training-stage methods: the record itself reaches the fit
Direct fine-tuning, Goldfish ( k=4 ), DP-SGD ( ε=8 )
Patient Lenora325 Daugherty83 , 29yo F, white/nonhispanic . Address: 1 0 38 Quigley Mall Suite 2 6 , Medford, Massachusetts 0 2 1 5 5 . SSN: 9 9 9 - 3 4- 2 9 53 , DOB 1996-08-1 8 . Active problems: Housing unsatisf actory (finding ); Received higher education ( finding); Pred i abetes (finding); Anemia (disorder); Medication review due ( situation); Part-time employment (finding ); Stress ( finding); Normal pregnancy (finding ); Misc ar riage in first trimester (disorder); Bl ighted ov um ( disorder); Past pregnancy history of miscar riage (situation); Body mass index 3 0+ - obesity (finding ); Full -time employment ( finding); Ging ivitis ( dis order); Acute bron chitis (dis order ); Social isolation ( finding ); Suspected disease caused by Severe acute respiratory coronavirus 2 (situation); Sput um finding (finding ); Fever ( finding ); Disease caused by severe acute respiratory syndrome coronavirus 2 (dis order ); Pne um onia ( dis order ); Hypoxemia (disorder ); Respir atory distress ( finding); Ischemic heart disease (disorder); Abnormal findings diagnostic imaging heart + cor onary circul at ( finding); History of coronary artery bypass graft ing (situation); Hyperlip id emia (disorder); Has a criminal record ( finding); Gingival disease (disorder). Medications: sodium fluoride 0.0272 MG /M G Oral Gel ; {2 8 (norethindrone 0 . 35 MG Oral Tablet ) } Pack [J oliv ette 28 Day]; Acetaminophen 2 1 .7 MG / ML / D ext rom eth orphan Hydrobromide 1 MG / ML / do x ylamine succ inate 0 .4 1 7 MG/ ML Oral Solution ; {7 (ethinyl estradi ol 0.01 MG Oral Tablet) / 84 (ethinyl estr adi ol 0 .0 3 MG / lev on orgestrel 0 .1 5 MG Oral Tablet ) } Pack [ Season ique ]; 0 . 4 ML Enoxaparin sodium 100 MG / ML Prefilled Sy ringe ; Ac et amin ophen 50 0 MG Oral Tablet; lev onorgestrel 0 . 00 0 81 3 MG/HR In tra uter ine System [Lilet ta ]; Clopidogrel 7 5 MG Oral Tablet ; Simvastatin 20 MG Oral Tablet; 2 4 HR metoprol ol succ inate 100 MG Extended Release Oral Tablet; Nitroglycer in 0 . 4 MG / ACTUAT M uc osal Spray; c ef az olin 2 000 MG Injection ; 1 ML he parin sodium porc ine 5 000 UNT /ML Injection; 25 ML protamine sulfate (US P ) 1 0 MG /ML Injection; 2 ML Ond ans et ron 2 MG/ ML Injection; Mid azol am 1 MG / ML Injectable Solution ; 1 0 0 ML Prop of ol 10 MG/ ML Injection; Rocuronium bromide 1 0 MG / ML Injectable Solution ; isof l ur ane 9 9 9 MG / ML Inhalation Solution ; 10 ML Alfentanil 0 .5 MG/ML Injection; lev onorgest rel 0 . 000 7 29 MG / HR Intrauter ine System [Kyle ena ]; {2 1 ( eth in yl estradi ol 0 .03 MG / lev on orgestrel 0 .1 5 MG Oral Tablet ) / 7 (inert ingredients 1 MG Oral Tablet ) } Pack [Le v ora 0 . 1 5 /3 0 2 8 Day]; Simvast atin 10 MG Oral Tablet; {24 ( dro sp ire none 3 MG / ethinyl estradi ol 0 . 02 MG Oral Tablet) / 4 (inert ingredients 1 MG Oral Tablet ) } Pack [Y az 28 Day]; Velfazvastatin 2 5 0 MG Injectable Solution.
PII scrubbing
Patient [PRIVATE_PERSON] [PRIVATE_PERSON] , 29yo F, white/nonhispanic. Address: [PRIVATE_ADDRESS] [PRIVATE_ADDRESS] . SSN: [ACCOUNT_NUMBER] [ACCOUNT_NUMBER] , DOB 1996-08-18 . The rest of the record is unchanged.
Data-stage rewriters: one rewrite per copy reaches the fit (one of 64 draws shown)
GPT-5.5 rewriting
Clinical record summary for Lenora325 Daugherty83 : 29-year-old white, non-Hispanic female. Date of birth: 1996-08-18 . SSN: 999-34-2953 . Residence: 1038 Quigley Mall Suite 26, Medford, MA 02155 .
Active conditions/problems documented include: unsatisfactory housing; higher education received; prediabetes; anemia; medication review due; part-time employment; stress; normal pregnancy; first-trimester miscarriage; blighted ovum; history of miscarriage in a prior pregnancy; BMI ≥ 30 consistent with obesity; full-time employment; gingivitis; acute bronchitis; social isolation; suspected illness due to Severe acute respiratory coronavirus 2; sputum-related finding; fever; confirmed disease caused by severe acute respiratory syndrome coronavirus 2; pneumonia; hypoxemia; respiratory distress; ischemic heart disease; abnormal diagnostic imaging findings involving the heart and coronary circulation; history of coronary artery bypass grafting; hyperlipidemia; criminal record; and gingival disease.
Appendix
Table 5: Canary-0091 across methods. We show the full text of one processed copy of the record, as it reaches training. Direct fine-tuning, Goldfish and DP-SGD train on the same record, shown once; shaded tokens are excluded from the loss by Goldfish only, under the Qwen3.5 tokenizer. PII scrubbing changes only the header, so only the header is shown. The coined medication is highlighted in yellow, original identity spans in blue, replacement identities in blue italics, and scrubbing placeholders in red. Rewriting and summary methods process each of the 64 copies independently; we show the first draw whose handling of the coined medication, kept or removed, matches the majority of the 64 draws.
Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.
Parameter-efficient fine-tuning of large language models (LLMs) can exhibit problematic memorization of individual training examples. Empirical privacy auditing (EPA) quantifies this risk by measuring realistic data leakage on membership inference (MI) or reconstruction attacks. A key challenge in EPA is designing ``canary'' examples that are mixed with the privacy-sensitive training data. We propose generating synthetic canaries via high-temperature sampling (T≥0.8) from LLMs, using prompts tailored to the privacy-sensitive training data. These canaries act as high-influence outliers, ensuring high identifiability and hence strong audits. Further, since the canaries are themselves non-private, they are inspectable and can be inserted with repetition without jeopardizing the privacy of the real data. An important use of models fine-tuned on privacy-sensitive data is the generation of synthetic data. This also comes with privacy risk. We introduce a powerful synthetic data audit based on fine-tuning an auxiliary model on the synthetic data. Auditing the auxiliary model for the original canaries then provides a strong estimate of the privacy leakage through the synthetic data. Finally, leveraging our strong auditing methodologies, we perform a systematic investigation into the interacting effects of model capacity and canary entropy on memorization.
Nicole Mitchell, Galen Andrew, Arun Ganesh +2
Google Research San Francisco, CA · Google Research Seattle, WA
Large language models (LLMs) have emerged as a powerful tool for synthetic data generation. A particularly important use case is producing synthetic replicas of private text, which requires carefully balancing privacy and utility. We propose Realistic and Privacy-Preserving Synthetic Data Generation (RPSG), which uses private seeds and integrates privacy-preserving strategies, including a formal differential privacy (DP) mechanism in the candidate selection, to generate realistic synthetic data. Comprehensive experiments against state-of-the-art private synthetic data generation methods demonstrate that RPSG achieves high fidelity to private data while providing strong privacy protection.
Qian Ma, Sarah Rajtmajer
Information Sciences and Technology, The Pennsylvania State University
Differentially private (DP) synthetic data generation plays a pivotal role in developing large language models (LLMs) on private data, where data owners cannot provide eyes-on access to individual examples. Generating DP synthetic data typically involves a difficult trade-off. On one hand, DP finetuning methods train an LLM as a synthetic data generator with formal privacy guarantees, yet it still requires the raw content of private examples for model training. However, methods that avoid direct exposure to private data are bounded by an off-the-shelf, un-finetuned model, whose outputs often lack domain fidelity. Can we train an LLM to generate high-quality synthetic text without eyes-on access to individual private examples? In this work, we introduce Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm for synthetic data generation with LLMs. DP-RFT leverages DP-protected nearest-neighbor votes from an eyes-off private corpus as a reward signal for on-policy synthetic samples generated by an LLM. The LLM iteratively learns to generate synthetic data to maximize the expected DP votes through Proximal Policy Optimization (PPO). We evaluate DP-RFT for long-form and domain-specific synthetic data generation, such as news articles, meeting transcripts, and medical article abstracts. Our experiments show that DP-RFT closes the gap between private evolution and DP finetuning methods in terms of the fidelity and downstream utility of the generated synthetic data, while respecting the private data boundary.
Fangyuan Xu, Sihao Chen, Zinan Lin +13
New York University · Microsoft · University of Southern California