Generating Diverse and Verified Synthetic Clinical Notes with Labels by Construction

Matthias Lehmann; Madeleine Werthebach

cs.LGNov 7, 2026

Generating Diverse and Verified Synthetic Clinical Notes with Labels by Construction

Authors: Matthias Lehmann, Madeleine Werthebach

Organizations: Strategic Innovations and Futuretechnologies, University Hospital Cologne

Abstract

Under strict privacy regulations in the clinical domain, de-identification is necessary to enable the use of clinical data such as for natural language processing. However, we currently lack sophisticated ways to tell which de-identification models actually are suitable. Specifically, we need benchmarks that are challenging, diverse, and clinically plausible, all while ensuring label correctness. To this end, we present SPARCLING, an LLM-based pipeline to generate synthetic discharge letters with character-exact PHI span annotations. Our pipeline combines structured attribute sampling, template generation, and red teaming to obtain diverse PHI-labeled discharge letters. In our experiments, we demonstrate this superior content diversity over existing public datasets. Further, we show SPARCLING's downstream utility as a benchmark to uncover failure modes of de-identifiers in the German clinical domain. Finally, our experiments show that training on our synthetic data can yield more capable de-identifiers than training on the scarce public data alone. Our code is available at https://github.com/Matt00n/phi_discharge_letter_generator.

Figures & tables

Figure 1: Overview of SPARCLING. (1) A structured profile

z_{i}

of patient, clinical, and document attributes is sampled hierarchically from a knowledge base before any LLM call. (2) The generator LLM only sees a masked profile

\tilde{z}_{i}

exposing PHI classes and grammatical features but no values, and writes a template

t_{i}

in which every PHI mention is a typed placeholder. (3) The sampled PHI values bypass the LLM and are substituted deterministically, with character offsets recorded, yielding the letter

x_{i}

and its span labels

Y_{i}

, exact by construction. (4) A deterministic known-PHI check and a red-team LLM auditing the masked letter search for leakage; any finding discards the document.

	German				English
	Sparcling	Sparcling 63	GraSCCo	GERNERMED	i2b2	MIMIC-IV
# documents	1,019	63	63	1,000	1,000	1,000
Doc-length std	4,519	3,419	2,619	74	605	4,265
Self-BLEU ( $\downarrow$ )	0.014	0.012	0.013	0.021	0.013	0.060
MTLD	310.0	317.1	248.5	38.6	182.8	100.8
Trigram novelty	0.63	0.87	0.80	0.65	0.74	0.45

Table 1: Corpus-level diversity of Sparcling-DE against its size-matched subsample ( Sparcling-DE 63 ) and four public clinical corpora. Higher is more diverse except Self-BLEU (

\downarrow

). i2b2 and MIMIC-IV are English, so their lexical values are indicative rather than directly comparable.

	SPARCLING-DE	GraSCCo	MIMIC-IV
SMOG	16.65	13.69	12.19
FKGL	20.75	14.84	9.50
Mean characters per doc	7,163	3,947	10,602
Spans per 1k word tokens	41.4	43.2	—
LLM judged structure	3.37	4.10	4.91
LLM judged clinical realism	2.31	3.90	4.89

Table 2: Surface-level and LLM-judged realism of SPARCLING-DE against two clinical reference corpora. LLM rows are mean judge scores (1–5). SPARCLING-DE matches the corpora on surface density and structure, but the LLM judge still separates it from real text on deeper clinical realism.

	Gemini	Gemma	Presidio+Flair	Presidio+spaCy	OpenMed	Rule-based
Precision	0.940	0.759	0.864	0.315	0.233	0.885
Recall	0.949	0.305	0.766	0.691	0.537	0.695
F1	0.944	0.435	0.812	0.433	0.325	0.778

Table 3: Detection performance per de-identification method (micro-averaged).

Training regime	P	R	F1
Synthetic only	0.942 $\pm$ 0.015	0.834 $\pm$ 0.032	0.884 $\pm$ 0.018
Public only	0.957 $\pm$ 0.011	0.955 $\pm$ 0.022	0.956 $\pm$ 0.014
Synthetic $\rightarrow$ Public	0.952 $\pm$ 0.013	0.975 $\pm$ 0.013	0.963 $\pm$ 0.012

Table 4: Main training results on GraSCCo at character-level. Synthetic only is a medBERT.de trained purely on SPARCLING-DE and evaluated on the held-out public folds; Public only is trained on the public training folds; Synthetic

\rightarrow

Public pretrains on SPARCLING-DE and finetunes on the public training folds. The hybrid regime improves span F1 over public-only on all folds.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Attribute	Distribution / mechanism	Constraints	Example
Style	Weighted choice over 3 styles	one per document	Standard-Klinik
Difficulty	Categorical, weights $[.29,.29,.29,.13]$	steers most ranges below	hard
Length	Categorical per difficulty	harder $\rightarrow$ longer	ausführlich
PII density $\rho$	$\mathrm{Beta}(\alpha,\beta)$ , shape by (style, length)	$\rho\in[0,1]$ ; academic+long skews high	$0.71$
Structure mode	Categorical per difficulty	harder $\rightarrow$ less structured	structured
Optional-field gate	$\mathrm{Bernoulli}(p\cdot(0.4+0.6\,\rho))$	gates every optional atom	include email?

Appendix

Table 5: Sampled document-level and structure/presentation attributes. Every draw is a deterministic function of the document seed;

\rho

denotes the document’s sampled PII density.

Attribute	Distribution / mechanism	Constraints	Example
Specialty	Gender-weighted, age-gated choice	e.g. Geriatrie $\geq$ 65	Kardiologie
Primary / secondary Dx	From specialty ICD pool	sex/age-compatible	I21.4 NSTEMI
Procedures (OPS)	Sampled per case complexity	consistent with Dx	1-275.0 Koronarangiographie
Discharge meds	Regimen pool	suppressed if $\neg$ has_meds ( $p{=}.20$ )	Bisoprolol 5 mg 1-0-0
Patient name	sample_name over origin-weighted pools	adversarial modes override	Lara Dina Schukow
Cross-origin / doubling	Bernoulli per difficulty	0 on easy; hard $.40/.22/.15$	Dimitra Magdalini Zogas

Appendix

Table 6: Sampled clinical, person, age, and date attributes. Every draw is a deterministic function of the document seed;

\rho

denotes the sampled PII density (Table 5 ).

Attribute	Distribution / mechanism	Constraints	Example
Street / house no.	pool $+\,\mathrm{Uniform}[1,180]$ , suffix $\mathrm{Bernoulli}(.2)$	—	Judengasse 131
ZIP / city	consistent city $\rightarrow$ area-code pair	hard: $\mathrm{Bernoulli}(.35)$ tricky town	04317 Leipzig
Address included	$\mathrm{Bernoulli}(0.85\cdot\text{gate})$	—	—
Phone / fax	area-code-consistent	always present	0341 9542205
Mobile	$\mathrm{Bernoulli}(0.35\cdot\text{gate})$	—	0157 7622301
Email	$\mathrm{Bernoulli}(0.30\cdot\text{gate})$ , derived from name	—	[email protected]

Appendix

Table 7: Sampled contact, identifier, institution, and quasi-identifier attributes with their negative decoys. Inclusion probabilities marked “

\cdot

gate” are further multiplied by the optional-field gate of Table 5 .

Family	Label	Meaning (DE)	Example
Names	PATIENT_NAME	Patient	Cem Çam
	RELATIVE_NAME	Angehörige/r	Ortrud Meister
	STAFF_NAME	Klinikpersonal	Dr. med. Weber
	OTHER_PERSON_NAME	sonstige Person	Christian Unger-Medwedew
Dates	DATE_BIRTH	Geburtsdatum	03.11.1960
	DATE_ADMISSION	Aufnahmedatum	06.06.2023

Appendix

Table 8: PII label ontology: direct and quasi identifiers. Direct identifiers must always be redacted (their recall is the headline privacy metric); quasi identifiers are reported separately as they re-identify only in combination.

Family	Label	Meaning (DE)	Example
Tier 3 — NON-PII negatives (3 labels)
Decoys	EPONYM	personenbezogener Fachbegriff	Hashimoto-Thyreoiditis
	LOOKALIKE	namensähnlicher Fachbegriff	Swan-Ganz-Katheter
	NONPII_ID	ID-ähnliche Nicht-PII-Nummer	PZN 70669074

Appendix

Table 9: Non-PII negatives: terms that resemble identifiers but must not be redacted. They are included by construction to measure over-redaction (false positives) rather than leakage.

Panel of Figure 3 — Figure 3: Original German system instruction supplied to the generation LLM.

Panel of Figure 4 — Figure 4: Faithful English translation of the system instruction in Figure 3 .

Panel of Figure 5 — Figure 5: Original German user prompt template. Braced fields are deterministically instantiated from the sampled profile ( $z_{i}$ ).

Panel of Figure 6 — Figure 6: Faithful English translation of the user prompt template in Figure 5 . This translation was not used during generation.

Skeleton placeholder / text	Rendered surface	Label
[[patient form=hon_first_last case=nom]]	Herr Cem Çam	PATIENT_NAME
[[patient form=hon_last case=akk]]	Herrn Çam	PATIENT_NAME
[[arzt_einweiser form=salutation case=nom]]	Sehr geehrter Herr Weber	STAFF_NAME
[[konsiliararzt form=initial_last case=gen]]	M. Fehlings	STAFF_NAME
[[geburtsdatum variant=textual]]	11. März 1962	DATE_BIRTH
[[geburtsdatum variant=iso]]	1962-03-11	DATE_BIRTH

Appendix

Table 10: Selected examples of placeholder rendering. Boldface marks the resulting gold span. Negative examples are emitted as ordinary text and therefore do not produce an annotated span.

Panel of Figure 7 — Figure 7: Red-team prompt used for residual PHI detection. The original German prompt used by the auditing LLM is shown on the left; a faithful English translation, which was not used during auditing, is shown on the right.

Panel of Figure 8 — Figure 8: Red-team prompt used for residual PHI detection. The original German prompt used by the auditing LLM is shown on the left; a faithful English translation, which was not used during auditing, is shown on the right.

Stage	Parameter	Value
Run	LLM model	gemini-3.7-flash
	base seed	1000 ( $\text{seed}_{i}=1000{+}i$ )
	parallel workers	8
	auto (drop flagged)	on
	sampling temperature	client default
Generation	skeleton min. sent. len.	15

Appendix

Table 11: Engine, generation post-processing, QC, and split hyperparameters.

Global mix (difficulty $=[.29,.29,.29,.13]$ )
Parameter	easy	med.	hard	adv.
length $w$ (kurz)	.30	.20	.10	.05
length $w$ (mittel)	.45	.35	.25	.15
length $w$ (ausf.)	.20	.30	.35	.35
length $w$ (sehr_lang)	.05	.15	.30	.45
struct. (structured)	.90	.70	.45	.30

Appendix

Table 12: Difficulty-conditioned sampling parameters (

w

: category weights;

p

: probabilities; dashes: not applied at that difficulty).

Parameter	Value / distribution
Dates (anchor 2026-09-03)
birth month / day	U[1,12] / U[1,28]
years-back values	{0..6,8,10,12,15}
years-back weights	30,12,8,8,6,6,5,5,5,4,4
admission offset (days)	U[5,365]
stay type (amb/ac/long)	.08/.78/.14

Appendix

Table 13: Entity- and field-level sampling parameters (U: uniform integer range;

\rho

: sampled PII density).

Parameter	Value
Base field-inclusion prob. (pre-density)
adresse / adresse_klinik	0.85 / 0.85
kvnr / fallnr / patid	0.75 / 0.80 / 0.65
lanr / bsnr / ik_nr	0.40 / 0.30 / 0.20
mobil / email	0.35 / 0.30
versicherer / klinik_url	0.70 / 0.50

Appendix

Table 14: Optional-field inclusion, PII-density shapes, and presentation/section parameters.

Panel of Figure 9 — Figure 9: Representative exclusions produced by our dual quality-control system. Boldface marks the substring responsible for the exclusion. Examples include a conservative deterministic false positive caused by a homograph collision, a deterministic catch of sampled PHI reproduced outside its gold span, and a residual hallucinated name detected by the red-team auditor.

	Sparcling	Sparcling 63	GraSCCo	GERNERMED	i2b2	MIMIC
Documents	1,019	63	63	1,000	1,000	1,000
Total tokens §	837,713	42,673	33,303	14,992	153,394	1,601,988
Mean doc length (chars)	7,163	5,888	3,947	102	887	10,602
Doc-length std (chars)	4,519	3,419	2,619	74	605	4,265
MTLD $\uparrow$	310.0	317.1	248.5	38.6	182.8	100.8
TTR per doc $\uparrow$	0.663	0.688	0.680	0.949	0.797	0.438

Appendix

Table 15: Full diversity metric suite across all corpora. Arrows give the "more diverse" direction (

\uparrow

higher,

\downarrow

lower). Rows marked § are corpus-size confounded (grow with document count / token count) and should be read alongside the size-matched Sparcling-DE 63 column, not the full corpus. Metrics are grouped into four classes: size metrics, per-doc lexical diversity, corpus-level lexical diversity, pairwise similarities, and content diversity. Pairwise-similarity metrics are sampled over document pairs. Section-structure and controllability metrics require generation metadata / section parsing and are only defined for Sparcling-DE .

LLM judge dimension	SPARCLING-DE	GraSCCo	MIMIC-IV
Structure	3.37	4.10	4.91
Clinical realism	2.31	3.90	4.89
Internal consistency	2.73	3.56	4.61
Temporal consistency	3.42	3.86	4.98
Care trajectory coherence	2.51	3.81	4.94
Discharge plan completeness	2.50	3.41	4.25

Appendix

Table 16: Full LLM-judge realism profile (mean scores, 1–5) across all nine dimensions. MIMIC-IV is judged near-ceiling on every axis and GraSCCo passes comfortably, whereas SPARCLING-DE scores lowest on the content-heavy axes (clinical realism, documentation quality, overall realism) while staying closest to real text on structure and temporal consistency.

LLM judge dimension	Easy	Medium	Hard	Adversarial
Structure	3.45	3.33	3.42	3.08
Clinical realism	2.47	2.34	2.18	2.10
Internal consistency	2.96	2.77	2.56	2.40
Temporal consistency	3.67	3.42	3.28	3.07
Care trajectory coherence	2.60	2.52	2.47	2.32
Discharge plan completeness	2.41	2.47	2.67	2.38

Appendix

Table 17: LLM-judge realism of SPARCLING-DE by difficulty tier (mean scores, 1–5). Overall realism decreases monotonically with intended difficulty (2.26

\rightarrow

1.70), confirming the difficulty ramp is effective; the sharpest degradation is in linguistic realism at the adversarial tier (3.41

\rightarrow

2.22), where injected contradictions read most synthetically.

Corpus pair	Fréchet $\downarrow$	MMD $\downarrow$	Unigram-JS $\downarrow$	Vocab. Jaccard $\uparrow$
SPARCLING-DE $\leftrightarrow$ GraSCCo	0.394	0.240	0.351	0.074
SPARCLING-DE $*{63}$ $\leftrightarrow$ GraSCCo	0.438	0.235	0.382	0.166
GraSCCo $\leftrightarrow$ GERNERMED	0.871	0.431	0.551	0.098
i2b2 $\leftrightarrow$ MIMIC-IV	0.579	0.441	0.357	0.166
SPARCLING-DE $\leftrightarrow$ GERNERMED	0.822	0.497	0.577	0.025
SPARCLING-DE $\leftrightarrow$ SPARCLING-DE $*{63}$	0.264	0.095	0.151	0.161

Appendix

Table 18: Selected cross-corpus distributional distances. Lower values indicate greater similarity for Fréchet distance, MMD, and Jensen–Shannon divergence, while higher vocabulary Jaccard indicates greater type overlap. The English i2b2–MIMIC-IV comparison is included only as a real–real reference scale.

	Gemini	Gemma	Flair	Presidio	OpenMed	Rule-based
PATIENT_NAME	0.981	0.406	0.969	0.904	0.608	0.634
STAFF_NAME	0.973	0.329	0.919	0.691	0.424	0.645
RELATIVE_NAME	0.985	0.365	0.976	0.866	0.670	0.644
DATE_BIRTH	1.000	0.498	0.998	0.993	0.622	1.000
AGE	0.937	0.097	0.000	0.005	0.042	0.966
ZIP	0.824	0.018	0.000	0.004	0.812	0.000

Appendix

Table 19: Per-label recall for selected PII categories. Best value per label in bold.

	Over-anonymization rate	Eponym destruction rate
Gemini 3.7 Flash	0.029	0.000
Gemma	0.033	0.010
Flair (in Presidio)	0.017	0.016
Presidio (spaCy)	0.414	0.396
OpenMed	0.583	0.619
Rule-based	0.055	0.033

Appendix

Table 20: Clinical over-redaction. Over-anonymization rate is the fraction of curated non-PII trap spans that were wrongly redacted; eponym destruction rate is the fraction of medical eponyms (disease/procedure names resembling person names) wrongly redacted. Lower is better; best value per column in bold.

Method	Eponym corr.	Look-alike red.	Faithfulness	Boundary IoU
Gemini	0	1.5%	0.9994	0.988
Flair	33	0.4%	0.9980	0.965
Rule-based	66	10.8%	0.9950	0.973
Presidio	803	41.0%	0.9080	0.964
OpenMed	1,255	46.0%	0.9740	0.849

Appendix

Table 21: Extended preservation diagnostics for selected de-identification methods. Eponym corruptions count wrongly modified medical eponyms; look-alike redaction reports the corresponding rate for other name-like clinical terms. Faithfulness measures preservation of non-redacted text, while boundary IoU measures agreement between predicted and gold redaction boundaries.

	Gemini	Gemma	Flair	Presidio	OpenMed	Rule-based
Easy	0.946	0.494	0.842	0.483	0.357	0.846
Medium	0.949	0.448	0.804	0.451	0.332	0.773
Hard	0.940	0.433	0.803	0.418	0.309	0.751
Adversarial	0.941	0.277	0.793	0.352	0.289	0.714

Appendix

Table 22: Detection F1 by document difficulty tier. Best value per tier in bold. Gemini is essentially flat across difficulty; every other method degrades markedly from easy to adversarial.

Evaluation set	Span F1	Token F1	Char. F1	Char. recall	Over-redaction
GraSCCo	$0.760\pm 0.002$	$0.878\pm 0.007$	$0.890\pm 0.003$	$0.844\pm 0.006$	0.0031
Synth. hold-out	$0.962\pm 0.000$	$0.997\pm 0.000$	$0.996\pm 0.000$	$0.998\pm 0.000$	0.0004

Appendix

Table 23: Synthetic-to-real transfer of a detector trained exclusively on SPARCLING-DE. Results are mean

\pm

standard deviation across seeds 42, 43, and 44. GraSCCo is entirely unseen during training, while the synthetic hold-out provides an in-distribution reference.

Training regime	Span F1	Token F1	Char. F1	Char. recall	Over-redaction
Real only	$0.850\pm 0.024$	$0.960\pm 0.008$	$0.956\pm 0.014$	$0.955\pm 0.022$	0.0026
Synthetic only	$0.762\pm 0.034$	$0.868\pm 0.022$	$0.884\pm 0.018$	$0.834\pm 0.032$	0.0032
Synthetic $\rightarrow$ Real	$\mathbf{0.873\pm 0.024}$	$\mathbf{0.968\pm 0.013}$	$\mathbf{0.963\pm 0.012}$	$\mathbf{0.975\pm 0.013}$	0.0031

Appendix

Table 24: Five-fold cross-validation on GraSCCo comparing real-only training, zero-shot synthetic-only transfer, and synthetic pretraining followed by fine-tuning on the available real training folds. Values are mean

\pm

standard deviation across folds.

Figure 10: Synthetic training-data scaling evaluated on GraSCCo. Increasing the amount of synthetic data primarily improves stricter span-level performance, while character recall reaches a high level already with comparatively little synthetic supervision.

	Metric	With adv.	Without adv.	$\Delta$
Span	Precision	0.864	0.824	$-0.040$
	Recall	0.682	0.681	$-0.001$
	F1	0.762	0.746	$-0.016$
Character	Precision	0.941	0.928	$-0.013$
	Recall	0.836	0.838	$+0.001$
	F1	0.886	0.881	$-0.005$

Appendix

Table 25: Drop-adversarial ablation, evaluated on GraSCCo. A PII detector trained on SPARCLING-DE with the adversarial examples included (‘With adv.”) versus removed (‘Without adv.”), tested on the external human-annotated GraSCCo corpus (63 documents). Best value per row in bold;

\Delta

is (Without

-

With). The precision benefit of adversarial training transfers to real clinical text: removing it drops span precision by 4 points (0.864

\rightarrow

0.824) at unchanged recall, confirming the effect is not an artifact of the synthetic test set.

Synthetic Clinical Data Generation

See all 27 papers

SHIELD: A Diverse Clinical Note Dataset and Distilled Small Language Models for Enterprise-Scale De-identification
May 5, 2026Jose D. Posada, David Love, Somalee Datta +1Healthcare Clinical Information Extraction
Technology & Digital Solutions Stanford Medicine · Stanford Medicine
MedDeID enables locally governed clinical-text de-identification from real or synthetic training data
Sep 9, 2026Stig Hellemans, Tom Stroobants, Elyne Scheurwegs +3Clinical NLP PII Detection
Adrem Data Lab, Department of Computer Science, University of Antwerp, Antwerp, Belgium · Antwerp University Hospital (UZA), Edegem, Belgium · Laboratory of Experimental Medicine and Pediatrics (LEMP), University of Antwerp, Antwerp, Belgium
EdgeDeID: A 0.6B Clinical De-identifier Trained Without Real Patient Data
Oct 24, 2026Sina Daneshgar, Raya Namazi, Amirali Khosravi +2Privacy-Preserving Language Models
Sharif University of Technology · Department of Radiology, Mayo Clinic · Department of Radiology and Biomedical Imaging, Yale University

May 5, 2026J/K move · Enter open · S save

SHIELD: A Diverse Clinical Note Dataset and Distilled Small Language Models for Enterprise-Scale De-identification

Jose D. Posada, David Love, Somalee Datta, Priya Desai

Technology & Digital Solutions Stanford Medicine · Stanford Medicine

Healthcare Clinical Information Extraction Language Model Distillation Clinical NLP PII Detection

Open paper AskPDF arXiv

De-identification of clinical text is a prerequisite for the secondary use of electronic health records. Existing public benchmarks such as the i2b2 2006 and 2014 corpora are over a decade old and lack the semantic and demographic diversity of modern clinical narratives. Large Language Models (LLMs) reach state-of-the-art zero-shot extraction, but their use at enterprise scale is limited by computational cost and by hospital data governance that restricts sending Protected Health Information (PHI) to cloud APIs. We introduce SHIELD (Synthetic Human-annotated Identifier-replaced Entries for Learning and De-identification), a diverse clinical note dataset of 1,381 notes with 10,229 gold-standard PHI spans across 9 categories, built with set-cover diversity sampling across demographic and document-type strata and human-in-the-loop adjudication. We evaluate four LLMs (two proprietary, two open-weight) to establish a performance ceiling on SHIELD, then show that a teacher-student distillation framework transfers these capabilities into locally deployable Small Language Models. Our best distilled model reaches micro-averaged span-level precision of 0.89 and recall of 0.88 while running on standard workstation hardware. It trails its cloud teacher on per-category recall (0.90 vs. 0.81 macro-averaged) but remains competitive given its lower cost and on-premise deployability. Cross-dataset evaluation shows that diversity-trained models generalize well on universal structured PHI categories, while institution-specific entities remain hard to transfer in both directions, which suggests pairing broad-coverage models with specialized models for high-volume, semi-structured note types. We publicly release the SHIELD dataset and the distilled DeBERTa v3 model to provide an accurate, cost-effective de-identification pipeline deployable entirely behind institutional firewalls.

Abstract

Synthetic Clinical Data Generation

Explore similar work

SHIELD: A Diverse Clinical Note Dataset and Distilled Small Language Models for Enterprise-Scale De-identification

MedDeID enables locally governed clinical-text de-identification from real or synthetic training data

EdgeDeID: A 0.6B Clinical De-identifier Trained Without Real Patient Data