Organizations: The University of Queensland, Brisbane, QLD, Australia · The University of New South Wales, Sydney, NSW, Australia · Shahid Sadoughi University of Medical Sciences and Health Services, Yazd, Iran · The University of Auckland, Auckland, New Zealand
Access to electronic health records (EHRs) for digital health research is often limited by privacy regulations and institutional barriers. Synthetic EHRs have been proposed as a way to enable safe and sovereign data sharing; however, existing methods may produce records that capture overall statistical properties of real data but present inconsistencies across clinical processes and observations. We developed an integrated pipeline to make synthetic patient trajectories clinically consistent through two synergistic steps: high-fidelity generation and scalable auditing. Using the MIMIC-IV database, we trained a knowledge-grounded generative model that represents nearly 32,000 distinct clinical events, including demographics, laboratory measurements, medications, procedures, and diagnoses, while enforcing structural integrity. To support clinical consistency at scale, we incorporated an automated auditing module leveraging large language models to filter out clinical inconsistencies (e.g., contraindicated medications) that escape probabilistic generation. We generated 18,071 synthetic patient records derived from a source cohort of 180,712 real patients. While synthetic clinical event probabilities demonstrated robust agreement (mean bias effectively 0.00) and high correlation (R2=0.99) with the real counterparts, review of a random sample of synthetic records (N=20) by three clinicians identified inconsistencies in 45-60% of them. Automated auditing reduced the difference between real and synthetic data (Cohen's effect size d between 0.59 and 1.60 before auditing, and between 0.18 and 0.67 after auditing). Downstream models trained on audited data matched or even exceeded real-data performance. We found no evidence of privacy risks, with membership inference performance indistinguishable from random guessing (F1-score=0.51).
Figures & tables
Figure 1
Figure 1: Fidelity and structural evaluation of synthetic EHR data. a Vocabulary coverage across 31,901 concepts. While standard benchmarks (blue) capture high-frequency diagnoses, Coogee models the “long tail” of clinical care (red), including specific medications (e.g., Semaglutide) and laboratory tests (e.g., HbA1c) required for high-fidelity disease modeling. b-c , Structural characteristics comparing real (blue) and synthetic (red) cohorts. The synthetic data mirrors real-world distributions for visits per patient and codes per visit, with minor truncation in the extreme upper tail due to model context window limits. d-f , Probabilistic agreement assessed via density-scaled Bland-Altman plots. Minimal mean bias (0·00) and narrow 95% limits of agreement across unigrams (marginal code probabilities), same-visit co-occurrences, and sequential-visit dependencies confirm that the model reproduces phenotype prevalence and care pathways without systematic bias. g-i , Temporal fidelity evaluation using empirical cumulative distribution functions (ECDFs). Distributional alignment (KS-stats ≤ 0 ⋅ 06) across event gaps, length of stay, and inter-visit intervals confirms the model’s ability to capture the irregular cadence of healthcare.
Figure 2: Preservation of complex clinical phenotypes and syndromic co-occurrences. Co-occurrence heatmaps of the top 150 diagnosis codes in Real (left) and Synthetic (right) cohorts, with axes sorted via hierarchical clustering. High Pearson ( r=0⋅93 ) and Spearman ( ρ=0⋅88 ) correlations demonstrate that Coogee generates coherent clinical syndromes rather than isolated codes. Annotated regions highlight the faithful reproduction of higher-order disease mechanisms, including the Cardio-Renal Syndrome and metabolic disease progression.
Round 1 (Unaudited)
Round 2 (Audited)
Evaluator
Review Strategy
Real
Syn.
P -val
Effect Size ( d )
Real
Syn.
P -val
Effect Size ( d )
Mean (SD)
Mean (SD)
Mean (SD)
Mean (SD)
Human review
Reviewer 1
Narrative
7 ⋅ 45 (1 ⋅ 83)
6 ⋅ 15 (2 ⋅ 41)
0 ⋅ 07
0 ⋅ 59
6 ⋅ 45 (2 ⋅ 18)
6 ⋅ 00 (2 ⋅ 59)
0 ⋅ 57
0 ⋅ 18
Reviewer 2
Narrative
6 ⋅ 30 (2 ⋅ 65)
4 ⋅ 80 (2 ⋅ 69)
0 ⋅ 09
0 ⋅ 55
5 ⋅ 50 (2 ⋅ 46)
4 ⋅ 05 (2 ⋅ 33)
0 ⋅ 07
0 ⋅ 59
Reviewer 3
Rule Check
8 ⋅ 10 (0 ⋅ 99)
5 ⋅ 60 (1 ⋅ 91)
< 0 ⋅ 001
1 ⋅ 60
6 ⋅ 90 (1 ⋅ 26)
5 ⋅ 80 (1 ⋅ 89)
0 ⋅ 04
0 ⋅ 67
Table 1: Impact of automated auditing on clinical consistency and realism. Comparison of realism scores assigned by clinicians and LLMs before (Round 1) and after (Round 2) automated auditing. The Effect Size (Cohen’s d ) quantifies the statistical separation between real and synthetic records (lower indicates higher indistinguishability). In Round 1, Rule-checking evaluators easily distinguished synthetic records ( d>1⋅0 , p≤0⋅001 ). In Round 2, discrimination ability decreased significantly.
PANEL A: downstream predictive utility
AUROC (95% CI)
F1-Score
Recall (Sensitivity)
Clinical Task
Real
Synthetic
Real
Synthetic
Real
Synthetic
1. Phenotyping (25-label)
0 ⋅ 96 (0 ⋅ 95–0 ⋅ 96)
0 ⋅ 91 (0 ⋅ 89–0 ⋅ 94)
0 ⋅ 93
0 ⋅ 90
0 ⋅ 95
0 ⋅ 89
2. Mortality (Binary)
0 ⋅ 76 (0 ⋅ 73–0 ⋅ 78)
0 ⋅ 69 (0 ⋅ 65–0 ⋅ 72)
0 ⋅ 72
0 ⋅ 70
0 ⋅ 75
0 ⋅ 98
3. Length-of-Stay (10-class)
0 ⋅ 77 (0 ⋅ 76–0 ⋅ 79)
0 ⋅ 74 (0 ⋅ 73–0 ⋅ 76)
0 ⋅ 32
0 ⋅ 21
0 ⋅ 34
0 ⋅ 25
4. Readmission (Binary)
0 ⋅ 61 (0 ⋅ 57–0 ⋅ 64)
0 ⋅ 62 (0 ⋅ 58–0 ⋅ 65)
0 ⋅ 51
0 ⋅ 62
0 ⋅ 44
0 ⋅ 66
Table 2: Downstream utility and privacy preservation. Panel A: compares the predictive performance of models trained on Real vs. Synthetic data across four clinical tasks. Panel B: quantifies privacy risks.
Train
Validation
Test
Total
Sample size, n
Patients
146,377 (81%)
16,264 (9%)
18,071 (10%)
180,712
Hospital admissions
348,900
39,432
42,743
431,075
Tokens
121,424,363
13,665,709
14,467,629
149,557,701
Age in years, mean (std)
-
56 ⋅ 20 (20 ⋅ 19)
56 ⋅ 21 (20 ⋅ 08)
56 ⋅ 36 (20 ⋅ 27)
56 ⋅ 22 (20 ⋅ 19)
Gender, n (%)
Female
77,344 (52 ⋅ 8%)
8,719 (53 ⋅ 6%)
9,656 (53 ⋅ 4%)
95,719 (53 ⋅ 0%)
Male
46,202 (45 ⋅ 4%)
7,545 (46 ⋅ 4%)
8,415 (46 ⋅ 6%)
84,993 (47 ⋅ 0%)
Table S1 : Patient characteristics in the MIMIC-IV dataset used in our experiments. The data distribution is consistent across splits, ensuring comparability of training and evaluation cohorts.
Figure S1 : Generating high-fidelity, full-spectrum patient trajectories with Coogee. i , Schematic patient trajectory representation integrating multimodal information: demographics (age, sex, race, marital status, calender year), laboratory tests, diagnoses (ICD-10-CM), procedures (ICD-10-PCS), medications (ATC codes), and structural tokens that encode time gaps and delimiters for records and visits). ii , Knowledge-aware representation of clinical events: each code is linked to its ontology hierarchy (e.g., ICD structure) and mapped to textual descriptions. iii , Transformer-based generative architecture: tokenized medical events are embedded, enriched with knowledge-aware representations, and processed to model longitudinal and multimodal patient trajectories. iv , Example of a synthetic trajectory generated by Coogee: starting from demographics, the model produces diabetes-related codes including labs (glucose, creatinine), diagnoses (type 2 diabetes with CKD), medications (insulin, metformin), and associated procedures, with realistic temporal gaps between visits.
RACE_ { ASIAN, BLACK, HISPANIC, OTHER, UNKNOWN, WHITE }
Marital Status
5
MARITAL_STATUS_ { DIVORCED, MARRIED, SINGLE, UNKNOWN, WIDOWED }
Calendar Year
16
Year of admission start (from YEAR_2005 to YEAR_2020 )
Table S2 : Atomic tokenization and vocabulary composition. The Coogee vocabulary maps 31,901 distinct clinical concepts to single tokens to prevent code fragmentation. The schema integrates static demographics with longitudinal events, utilizing quantile bins to capture continuous laboratory values and discrete interval tokens to preserve temporal resolution.
Figure S2 : Scaling law investigation of model-data trade-offs in Coogee. (a) IsoFLOP curves showing validation loss as a function of model size (parameters) for different computational budgets. Each curve corresponds to a constant training FLOPs constraint, with dashed lines denoting quadratic fits in log–parameter space. (b) Optimal model size (minimizing validation loss per compute budget) follows a power-law relationship with total training FLOPs, consistent with Chinchilla-style scaling trends. Hoffmann et al., 2022 (c) Corresponding optimal token counts also obey a power-law with compute. Together, these analyses quantify the balance between model capacity and dataset size for efficient training. Based on the identified isoFLOP frontier and available computational resources, we selected the 50M-parameter (13M trainable) configuration for our main experiments.
Hyperparameter
Value
Params
Trainable
Frozen
Vocab size
31,901
–
–
–
Context length
2,048
–
–
–
Embedding dim
384
–
–
–
Hidden dim
1,024
–
–
–
Decoder layers
6
–
–
–
Attention heads
6
–
–
–
Table S3 : Architecture and parameter breakdown of the Coogee-50M model. The table details the hyperparameters, embedding dimensions, and per-component parameter counts. Of the total 50M parameters, 13.2M are trainable and 36.7M are frozen. Knowledge-aware embeddings, including structural (hierarchy and semantic) embeddings, are frozen to preserve ontology-derived structure, while trainable parameters lie primarily in the token embedding layer and decoder blocks.
Metric
Real (Truncated)
Synthetic
Trajectory Structure
Visits per Patient
Median: 1 ⋅ 0; IQI: 1 ⋅ 0-2 ⋅ 0
Median:1 ⋅ 0; IQI: 1 ⋅ 0-3 ⋅ 0
Codes per Visit
Median:131 ⋅ 0; IQI: 46 ⋅ 0-316 ⋅ 0
Median:124 ⋅ 0; IQI: 45 ⋅ 0-289 ⋅ 0
Probabilistic Alignment ( R2 )
Unigram Probability
0 ⋅ 997
Same-Visit Co-occurrence
0 ⋅ 988
Table S4 : Comparison of structural metrics controlled for context window limits. When the real baseline is restricted to trajectories that fit within the model’s context window ( ≤ 2,048 tokens), the synthetic data mirrors the real distributions with high precision.
Figure S3 : Preservation of population-level statistics: correlations of unigram (marginal), same-visit bigram (co-occurrence), and sequential-visit bigram (temporal) probabilities.
Figure S4 : Clinician and LLM-based realism scores in Round 1. (a) Distribution of realism scores. Narrative-focused clinicians (Reviewers 1 and 2) show significant distributional overlap between Real (blue) and Synthetic (red) records. In contrast, the rule-checking clinician (Reviewer 3) and LLMs show a clear separation, identifying logical inconsistencies. (b) Inter-rater reliability measured through Intraclass Correlation Coefficients (ICCs). The strong correlation between Reviewer 3 and LLMs (ICCs of 0 ⋅ 34, 0 ⋅ 56 and 0 ⋅ 61) suggests that automated auditors may focus on the validity of clinical rules within the records.
Figure S5 : Validation loss curves of Coogee with and without knowledge-informed embeddings. The plot compares the validation loss trajectories during training for two model variants: Coogee with knowledge_embd (red) and Coogee without knowledge_embd (blue). Both models share identical hyperparameters and training configurations, differing only in the inclusion of auxiliary biomedical knowledge embeddings (hierarchical and semantic). The model with knowledge integration exhibits consistently lower validation loss across training steps.
time
code
numerical_value
code_label
11-29 22:01
START_RECORD
11-29 22:01
AGE_75_80_years
11-29 22:01
SEX_F
11-29 22:01
RACE_UNKNOWN
11-29 22:01
MARITAL_STATUS_MARRIED
11-29 22:01
YEAR_2014
Table S5 : A synthetic patient trajectory achieving a mean realism score of 7.7/10 in combined clinician and automated evaluation (§ 3.2 ). This synthetic record represents an elderly female undergoing Coronary Artery Bypass Grafting (CABG). Evaluators selected this trajectory for its high degree of pathophysiological coherence. Specifically, the human reviewer noted that the severe drop in hemoglobin and hematocrit was physiologically consistent with post-cardiac surgery and logically justified the subsequent diagnosis of acute posthemorrhagic anemia (D62). LLM auditors (Gemini, Qwen) similarly highlighted the highly realistic temporal trends in coagulation monitoring and the use of guideline-concordant discharge medications (e.g., aspirin, statins, beta-blockers) as key factors driving the high realism score.
The generation of high-fidelity synthetic Electronic Health Records (EHR) is crucial for advancing medical research while preserving patient privacy. However, head-to-head comparison of existing generative models is hindered by disjointed codebases, incompatible data loaders, conflicting library dependencies, and inconsistent evaluation protocols. To address these gaps, we introduce a lightweight, end-to-end benchmarking framework for reproducible synthetic EHR evaluation, organized as a unified pipeline spanning data ingestion, standardized model training, and architecture-agnostic evaluation. Our current implementation targets the generation of longitudinal ICD diagnosis codes -- the most commonly studied modality in this literature -- and is built on the community-maintained PyHealth library. We reimplement and unify strong baselines (MedGAN, CorGAN, PromptEHR, HALO) under full ICD-9 vocabulary granularity, and add a lightweight GPT-2 baseline from the general-purpose sequence-modeling literature. We contribute a rigorous, architecture-agnostic privacy-utility evaluation suite that applies identically to GAN- and transformer-based generators, and report bootstrapped confidence intervals across all metrics. We further analyze the poor long-tailed performance of existing models and discuss the extensibility of our framework beyond diagnosis codes. By lowering the engineering barrier to running, extending, and evaluating under a single pipeline, we introduce a starting point for community-driven reproducibility and benchmarking synthetic EHR models.
Synthetic healthcare data are widely proposed as privacy-preserving substitutes for real patient data, yet their evaluation remains dominated by statistical similarity and predictive performance that do not reflect clinical validity. We introduce a multi-dimensional evaluation framework grounded in epidemiology, assessing descriptive fidelity, clinical utility, and structural validity, corresponding to descriptive, predictive, and causal questions. We evaluate four representative generative paradigms - GAN-based, VAE-boosted, diffusion-based, and masked modelling - using PRIME-CVD, a 50,000-person cohort with known ground-truth structure. While all models reproduce marginal distributions, none simultaneously preserve subgroup structure, effect estimates, and dependency structure. Notably, models with strong distributional fidelity can exhibit poor calibration and distorted relationships, leading to unreliable inference. These results show that current evaluation practices can overestimate synthetic data quality and motivate domain-informed assessment based on the ability to support valid clinical and scientific conclusions.
Nicholas I-Hsien Kuo, Blanca Gallego, Louisa Jorm
Centre for Big Data Research in Health, the University of New South Wales, Sydney, Australia
Frontier language models are rarely used in clinical workflows because the realistic, longitudinal benchmarks needed to develop them are scarce. Real electronic health record (EHR) data cannot be openly shared due to privacy, ethics or data use issues and it does not contain verifiable ground truth since the chart records only reflect what clinicians documented. We introduce Synthetic Hospital, an open, fully synthetic, fact-grounded longitudinal EHR benchmark that resolves the open sharing and verifiable ground truth barriers. Built entirely from public medical-education material with no protected health information, it comprises 1,268 longitudinal patients and 5,602 encounters, where every diagnosis, finding, and temporal relation is grounded in standard ontologies (ICD-10-CM, SNOMED CT, LOINC) and with a complete provenance chain back to its source medical education material. Synthetic Hospital is served through a simulated hospital record system that mirrors real EHR infrastructure (standard interoperability APIs, role-based access and function-calling interface). In a blinded review, physicians distinguished its records from real patient charts at near-chance rates (53%). Across 10 frontier and open models, none approaches ceiling: the best model reconstructs a patient's longitudinal problem list with a severity-weighted F1 of 0.73, level with the mean of seven physicians on a matched subset but well below the best of them (0.89), and misses roughly half of clinically relevant findings when summarizing a chart. Overall, these results highlight that Synthetic Hospital is a difficult and realistic test of clinical AI performance.