cs.CROct 15, 2026
SaveExploring The Empirical Privacy Landscape of Synthetic Data
Organizations: Johns Hopkins University · OpenAI
Abstract
Methods for training language models on sensitive data differ in how they balance capability gains against the risk of leaking private information. In this paper, we present a controlled comparison of data-level and training-level interventions for navigating the capability-privacy tradeoff on synthetic medical records with canaries to probe for leakage. Direct finetuning performs best on capability metrics but leads to the most privacy leakage, while DP-SGD reduces leakage at substantial cost to capabilities. Generating synthetic data from these records offers a more favorable balance, particularly when guided by a uniqueness-aware instruction to replace private details. Relative to naive paraphrasing, privacy-aware rewriting improves privacy protection across a variety of different rewriting models. We also find that more capable rewriting models generally reduce the risks of downstream privacy leakage, although held-out fit does not improve monotonically and increased test-time compute yields inconsistent gains. Overall, these results highlight privacy-aware rewriting as a promising direction for achieving a more favorable trade-off between capabilities and empirical privacy compared to other training-level methods, and our evaluation framework supports comparisons of future models and methods.
Figures & tables
Figure 1: Experimental framework and privacy–capability trade-off . Left: Experimental framework. Right: Results for finetuning gpt-oss-20b, with each axis averaging ranks across two privacy and two capability metrics. Higher ranks indicate less privacy leakage and greater capabilities. Arrows connect rewriting to privacy-aware rewriting, which improves both privacy and capability.
| Method | Prompt |
|---|---|
| Rewriting | Paraphrase the following clinical record, preserving every medical fact: |
| Privacy-aware rewriting | Rewrite the following clinical record so it stays realistic and clinically coherent. Replace any detail that is rare, distinctive, or unusual enough that it could help identify this specific patient with a different but plausible fictional value; leave common, generic clinical information unchanged. Use realistic surrogate values, not placeholders. |
| Example | |
| Input record | Name: Royce851 Keeling91 . Date of birth: 1986-03-14 . Address: 742 Juniper Hollow Road, Bedford, MA 01730 . Problems: essential hypertension; dental caries. Medications: lisinopril 10 MG oral tablet; sodium fluoride 0.0272 MG/MG oral gel; Mabsartinib . |
| Rewriting output | Royce851 Keeling91 , born on March 14, 1986 , resides at 742 Juniper Hollow Road, Bedford, MA 01730 . The patient has essential hypertension and dental caries. Current medications include lisinopril 10 MG oral tablet, sodium fluoride 0.0272 MG/MG oral gel, and Mabsartinib . |
| Privacy-aware rewriting output | Avery Morgan , born on June 22, 1985 , resides at 58 Maple Street, Lowell, MA 01852 . The patient has essential hypertension and dental caries. Current medications include lisinopril 10 MG oral tablet, sodium fluoride 0.0272 MG/MG oral gel, and imatinib 400 MG oral tablet . |
Table 1: Prompts and examples for rewriting and privacy-aware rewriting. Each prompt is followed by the source record and the completion cue Rewrite: . Bold text marks distinctive details retained by rewriting or replaced by privacy-aware rewriting.
| Metric | Input to the model | PPL measured on |
|---|---|---|
| Verbatim perplexity | the canary record truncated immediately before the planted medication: “…Amoxicillin 250 MG / Clavulanate 125 MG Oral Tablet; sodium fluoride 0.0272 MG/MG Oral Gel;” | Mabsartinib |
| Semantic perplexity | “Which medication starting with ‘M’ was prescribed to Royce851 Keeling91?” | Mabsartinib |
| Held-out record perplexity | an entire record from the held-out set | all example tokens |
| Medication perplexity | a test record truncated immediately before its final medication: “…Lisinopril 10 MG Oral Tablet; Acetaminophen 500 MG Oral Tablet;” | Ibuprofen |
Table 2: Perplexity-based metrics for model privacy and capability. Lower values indicate higher privacy leakage for the first two metrics and better capability for the last two.
Figure 2: Canary privacy leakage across levels of duplication. Higher values indicate less privacy leakage, and error bars show 95% confidence intervals over 20 canaries per group.
Figure 3: Privacy–capability trade-offs across rewriters. Open and filled markers denote rewriting and privacy-aware rewriting, with arrows connecting each pair.
Figure 4: Privacy–capability trade-offs for privacy-aware rewriting with gpt-oss-20b as the target model. The left shows different rewriting models, while the right shows different amounts of reasoning. Error bars denote 95% confidence intervals.
Figure 5: Privacy–capability trade-offs for privacy-aware rewriting with and without a summarization step. Error bars denote 95% confidence intervals.
Figure 6: Comparison of five independent rewrites trained for one epoch each and a single rewrite trained for five epochs. In both panels, the GPT-5.5 single-rewrite, five-epoch condition is normalized to ; all other values are reported as ratios to this reference. Higher values indicate better performance on every axis.
Figure 7: Comparison of LLM-judged scores with measured outcomes using Qwen3.5-4B-Base as the target model. Left: judged quality versus capability. Right: judged privacy versus verbatim perplexity. Point labels indicate the GPT-series rewriter and reasoning-effort setting.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Privacy–capability trade-offs across methods. Arrows connect rewriting to privacy-aware rewriting for the same rewrite model.
Figure 9: Privacy–capability trade-offs across methods. Lines connect rewriting to privacy-aware rewriting for the same rewrite model. Error bars denote 95% confidence intervals.
Figure 10: Privacy–capability trade-offs across methods. Lines connect rewriting to privacy-aware rewriting for the same rewrite model. Error bars denote 95% confidence intervals.
Figure 11: Canary privacy leakage across duplication factors. Rows show Qwen3.5-4B-Base and gpt-oss-20b; columns show semantic perplexity. Higher values indicate less privacy leakage, and error bars show 95% confidence intervals over 20 canaries per group.
Figure 12: Privacy–capability trade-offs for privacy-aware rewriting with Qwen3.5-4B-Base as the target model. The left shows different rewriting models, while the right shows different amounts of reasoning. Error bars denote 95% confidence intervals.
Figure 13: Privacy–capability trade-offs for privacy-aware rewriting with gpt-oss-20b as the target model. The left shows different rewriting models, while the right shows different amounts of reasoning. Error bars denote 95% confidence intervals.
Figure 14: Privacy–capability trade-offs for privacy-aware rewriting with Qwen3.5-4B-Base as the target model. The left shows different rewriting models, while the right shows different amounts of reasoning. Error bars denote 95% confidence intervals.
Figure 15: Privacy–capability trade-offs for privacy-aware rewriting with and without a summarization step. Error bars denote 95% confidence intervals.
Figure 16: Comparison of five independent rewrites trained for one epoch each and a single rewrite trained for five epochs. In both panels, the GPT-5.5 single-rewrite, five-epoch condition is normalized to ; all other values are reported as ratios to this reference. Higher values indicate better performance on every axis.
Figure 17: Comparison of LLM-judged scores with measured outcomes using Qwen3.5-4B-Base as the target model. Left: judged quality versus capability. Right: judged privacy versus verbatim perplexity. Point labels indicate the GPT-series rewriter and reasoning-effort setting.
Figure 18: Comparison of LLM-judged scores with measured outcomes using gpt-oss-20b as the target model. Left: judged quality versus capability. Right: judged privacy versus verbatim perplexity. Point labels indicate the GPT-series rewriter and reasoning-effort setting.
Figure 19: Comparison of LLM-judged scores with measured outcomes using gpt-oss-20b as the target model. Left: judged quality versus capability. Right: judged privacy versus verbatim perplexity. Point labels indicate the GPT-series rewriter and reasoning-effort setting.
Figure 20: The correlation between two privacy metrics. Verbatim versus semantic perplexity of the planted secret for each method configuration. Labeled points are the nine methods; gray points are rewriting and summary variants. The second row zooms into the shaded region below verbatim perplexity 100, where agreement drops to and .
Figure 21: The correlation between two capability metrics. Held-out record perplexity versus medication perplexity for each method configuration. Labeled points are the nine methods; gray points are rewriting and summary variants. The second row zooms into the shaded region below held-out perplexity 2.5, where agreement is similar ( and , over 35 and 5 points).
Figure 22: Probability of generating the coined medication. Probability that each target model generates the coined medication given the record up to it, against how many times the canary appeared in training. Each point is the mean over the 20 canaries at that duplication level, with 95% confidence intervals.
| Qwen3.5-4B-Base | gpt-oss-20b | |||
|---|---|---|---|---|
| Method | Exposure | Rank 1 | Exposure | Rank 1 |
| No finetuning | 1.49 | 0% | 1.39 | 0% |
| Direct fine-tuning | 12.22 | 86% | 10.98 | 73% |
| Goldfish ( ) | 10.38 | 53% | 8.90 | 37% |
| DP-SGD ( ) | 1.74 | 0% | 1.56 | 0% |
| PII scrubbing | 12.16 | 87% | 10.76 | 68% |
Table 3: Secret-Sharer exposure of coined medication names. Each planted coined medication is ranked against 6,300 unplanted alternatives by its whole-name conditional likelihood given the original record prefix ending immediately before the medication. We report mean exposure, , across 100 canaries and the percentage ranked first within their candidate sets (“Rank 1”). Exposure ranges from 0 to 12.62 bits; uniformly random ranks have an expected mean exposure of approximately 1.44 bits. Lower values indicate less exposure.
| Method | Prompt |
|---|---|
| Summary-based rewriting | Summarize the following clinical record in a few sentences, keeping the clinically essential information (active diagnoses, key treatments, current medications): |
| Privacy-aware summary-based rewriting | Summarize the following clinical record in a few sentences, keeping the clinically essential information. Replace any detail that is rare, distinctive, or unusual enough that it could help identify this specific patient with a different but plausible fictional value; leave common, generic clinical information unchanged. Use realistic surrogate values, not placeholders. |
| Expansion prompt | Below is a clinical summary of a patient. Reconstruct a complete, realistic patient record consistent with it, in exactly the format of the example records shown. Include every diagnosis and medication the summary mentions; invent realistic demographics and identifiers (name, address, SSN, DOB consistent with the age) — the summary does not contain them; add plausible routine detail where the summary is silent. |
Table 4: Generation prompts. The two summary-based methods summarize first and then expand the summary back into a full record with the expansion prompt, which additionally receives two format exemplars drawn from a fresh Synthea seed, never from the private cohort. The privacy-aware summary prompt is the generic summary instruction followed by the privacy-aware rewriting instruction.
| Method | What reaches training (full text) |
|---|---|
| Training-stage methods: the record itself reaches the fit | |
| Direct fine-tuning, Goldfish ( ), DP-SGD ( ) | Patient Lenora325 Daugherty83 , 29yo F, white/nonhispanic . Address: 1 0 38 Quigley Mall Suite 2 6 , Medford, Massachusetts 0 2 1 5 5 . SSN: 9 9 9 - 3 4- 2 9 53 , DOB 1996-08-1 8 . Active problems: Housing unsatisf actory (finding ); Received higher education ( finding); Pred i abetes (finding); Anemia (disorder); Medication review due ( situation); Part-time employment (finding ); Stress ( finding); Normal pregnancy (finding ); Misc ar riage in first trimester (disorder); Bl ighted ov um ( disorder); Past pregnancy history of miscar riage (situation); Body mass index 3 0+ - obesity (finding ); Full -time employment ( finding); Ging ivitis ( dis order); Acute bron chitis (dis order ); Social isolation ( finding ); Suspected disease caused by Severe acute respiratory coronavirus 2 (situation); Sput um finding (finding ); Fever ( finding ); Disease caused by severe acute respiratory syndrome coronavirus 2 (dis order ); Pne um onia ( dis order ); Hypoxemia (disorder ); Respir atory distress ( finding); Ischemic heart disease (disorder); Abnormal findings diagnostic imaging heart + cor onary circul at ( finding); History of coronary artery bypass graft ing (situation); Hyperlip id emia (disorder); Has a criminal record ( finding); Gingival disease (disorder). Medications: sodium fluoride 0.0272 MG /M G Oral Gel ; {2 8 (norethindrone 0 . 35 MG Oral Tablet ) } Pack [J oliv ette 28 Day]; Acetaminophen 2 1 .7 MG / ML / D ext rom eth orphan Hydrobromide 1 MG / ML / do x ylamine succ inate 0 .4 1 7 MG/ ML Oral Solution ; {7 (ethinyl estradi ol 0.01 MG Oral Tablet) / 84 (ethinyl estr adi ol 0 .0 3 MG / lev on orgestrel 0 .1 5 MG Oral Tablet ) } Pack [ Season ique ]; 0 . 4 ML Enoxaparin sodium 100 MG / ML Prefilled Sy ringe ; Ac et amin ophen 50 0 MG Oral Tablet; lev onorgestrel 0 . 00 0 81 3 MG/HR In tra uter ine System [Lilet ta ]; Clopidogrel 7 5 MG Oral Tablet ; Simvastatin 20 MG Oral Tablet; 2 4 HR metoprol ol succ inate 100 MG Extended Release Oral Tablet; Nitroglycer in 0 . 4 MG / ACTUAT M uc osal Spray; c ef az olin 2 000 MG Injection ; 1 ML he parin sodium porc ine 5 000 UNT /ML Injection; 25 ML protamine sulfate (US P ) 1 0 MG /ML Injection; 2 ML Ond ans et ron 2 MG/ ML Injection; Mid azol am 1 MG / ML Injectable Solution ; 1 0 0 ML Prop of ol 10 MG/ ML Injection; Rocuronium bromide 1 0 MG / ML Injectable Solution ; isof l ur ane 9 9 9 MG / ML Inhalation Solution ; 10 ML Alfentanil 0 .5 MG/ML Injection; lev onorgest rel 0 . 000 7 29 MG / HR Intrauter ine System [Kyle ena ]; {2 1 ( eth in yl estradi ol 0 .03 MG / lev on orgestrel 0 .1 5 MG Oral Tablet ) / 7 (inert ingredients 1 MG Oral Tablet ) } Pack [Le v ora 0 . 1 5 /3 0 2 8 Day]; Simvast atin 10 MG Oral Tablet; {24 ( dro sp ire none 3 MG / ethinyl estradi ol 0 . 02 MG Oral Tablet) / 4 (inert ingredients 1 MG Oral Tablet ) } Pack [Y az 28 Day]; Velfazvastatin 2 5 0 MG Injectable Solution. |
| PII scrubbing | Patient [PRIVATE_PERSON] [PRIVATE_PERSON] , 29yo F, white/nonhispanic. Address: [PRIVATE_ADDRESS] [PRIVATE_ADDRESS] . SSN: [ACCOUNT_NUMBER] [ACCOUNT_NUMBER] , DOB 1996-08-18 . The rest of the record is unchanged. |
| Data-stage rewriters: one rewrite per copy reaches the fit (one of 64 draws shown) | |
| GPT-5.5 rewriting | Clinical record summary for Lenora325 Daugherty83 : 29-year-old white, non-Hispanic female. Date of birth: 1996-08-18 . SSN: 999-34-2953 . Residence: 1038 Quigley Mall Suite 26, Medford, MA 02155 . |
| Active conditions/problems documented include: unsatisfactory housing; higher education received; prediabetes; anemia; medication review due; part-time employment; stress; normal pregnancy; first-trimester miscarriage; blighted ovum; history of miscarriage in a prior pregnancy; BMI 30 consistent with obesity; full-time employment; gingivitis; acute bronchitis; social isolation; suspected illness due to Severe acute respiratory coronavirus 2; sputum-related finding; fever; confirmed disease caused by severe acute respiratory syndrome coronavirus 2; pneumonia; hypoxemia; respiratory distress; ischemic heart disease; abnormal diagnostic imaging findings involving the heart and coronary circulation; history of coronary artery bypass grafting; hyperlipidemia; criminal record; and gingival disease. | |
Table 5: Canary-0091 across methods. We show the full text of one processed copy of the record, as it reaches training. Direct fine-tuning, Goldfish and DP-SGD train on the same record, shown once; shaded tokens are excluded from the loss by Goldfish only, under the Qwen3.5 tokenizer. PII scrubbing changes only the header, so only the header is shown. The coined medication is highlighted in yellow, original identity spans in blue, replacement identities in blue italics, and scrubbing placeholders in red. Rewriting and summary methods process each of the 64 copies independently; we show the first draw whose handling of the coined medication, kept or removed, matches the majority of the 64 draws.
More in this topic
Privacy Leakage in Language Models
Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.
See all 280 papers