Constructing and Validating Memorisation Audits for EHR Foundation Models
Organizations: Departments of Computing and Bioengineering, Imperial College London, UK · Department of Surgery & Cancer, Imperial College London, UK · Chair in Digital Health & Data Science, University of Bayreuth, Germany
Abstract
Autoregressive electronic health record (EHR) foundation models may disclose training information at inference, yet clinical predictability makes patient-specific memorisation difficult to distinguish from generalisation. We evaluate fixed-window token reconstruction, sensitive-attribute disclosure, and random patient-level canaries in GPT-2-style models trained on MIMIC-IV, across early, best-validation, and final checkpoints. Reconstruction improves with additional context, but training--held-out differences remain small. Trajectory-unique prompts containing limited clinical information produce no excess sensitive disclosure for training patients. In a matched model with random binary patient-level canaries, training-patient label-recovery AUROC increases from 0.512 to 0.549 while held-out recovery remains at chance. An overfit 1,000-patient positive control achieves near-perfect recovery, confirming sensitivity to strong memorisation. These results reveal weak cohort-level memorisation but no reliable patient-level disclosure under the evaluated attacks. They motivate clinically grounded inference-time audits that can detect patient-specific memorisation in existing models without retraining with canaries.
Figures & tables
| Train | Held-out | ||||||
| Checkpoint | Step | AUROC [95% CI] | AUROC [95% CI] | ||||
| Full-scale model ( members) | |||||||
| Early | 30,000 | 0.512 [0.510, 0.515] | 0.503 [0.496, 0.510] | ||||
| Best | 550,000 | 0.549 [0.547, 0.551] | 0.500 [0.493, 0.507] | ||||
| Late | 730,000 | 0.548 [0.546, 0.551] | 0.498 [0.491, 0.505] | ||||
| Positive control: overfit model ( members) | |||||||
| Checkpoint | Step | Training | Held-out | Gap (pp) | 95% CI |
| Depression-token carriers | |||||
| Early | 30,000 | 932 (15.5%) | 897 (14.9%) | ||
| Best | 550,000 | 982 (16.4%) | 1,010 (16.8%) | ||
| Late | 730,000 | 958 (16.0%) | 1,045 (17.4%) | ||
| Matched negative patients: false disclosure | |||||
| Best | 550,000 | 658 (11.0%) | 719 (12.0%) | ||
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Train | Validation | Test | |
| Patients | 228,508 | 28,560 | 28,538 |
| Total tokens | 289,983,089 | 36,217,252 | 36,496,872 |
| Length, mean sd | |||
| minimum | 8 | 8 | 8 |
| 25th percentile | 128 | 128 | 129 |
| median | 337 | 341 | 346 |
| Architecture | |
|---|---|
| Layers | 6 |
| Attention heads | 12 |
| Embedding dimension | 768 |
| Context window | 2,048 |
| Positional encoding | RoPE ( ) |
| Dropout | 0.1 |
| Checkpoint | Train | Held out | Train | Held out |
|---|---|---|---|---|
| Early | ||||
| Best | ||||
| Late | ||||
| Overfit | ||||
| Benchmark | Prediction point | Terminating tokens |
|---|---|---|
| ICU Mortality | ICU_ADMISSION | MEDS_DEATH , ICU_DISCHARGE |
| ICU Readmission | ICU_DISCHARGE | ICU_ADMISSION , HOSPITAL_DISCHARGE |
| ICU Admission | HOSPITAL_ADMISSION | ICU_ADMISSION , HOSPITAL_DISCHARGE |
| Hospital Mortality | HOSPITAL_ADMISSION | MEDS_DEATH , HOSPITAL_DISCHARGE |
| Hospital Readmission | HOSPITAL_DISCHARGE | HOSPITAL_ADMISSION , MEDS_DEATH |
| AUROC | AUPRC | |||
|---|---|---|---|---|
| Task | Base | Canary | Base | Canary |
| Hospital mortality | 0.858 | 0.848 | 0.318 | 0.297 |
| Hospital readmission | 0.703 | 0.703 | 0.486 | 0.482 |
| ICU admission | 0.912 | 0.911 | 0.767 | 0.766 |
| ICU mortality | 0.878 | 0.887 | 0.447 | 0.486 |
| ICU readmission | 0.735 | 0.724 | 0.400 | 0.388 |
Memorization in Language Models
Memorization in language models studies how training examples and facts are stored, recalled, generalized, and exposed by model parameters.
See all 125 papers