cs.LGNov 11, 2026

Constructing and Validating Memorisation Audits for EHR Foundation Models

Authors: Inigo Stibbe, Joshua Placidi, Padmanabhan Ramnarayan, A. Aldo Faisal

Organizations: Departments of Computing and Bioengineering, Imperial College London, UK · Department of Surgery & Cancer, Imperial College London, UK · Chair in Digital Health & Data Science, University of Bayreuth, Germany

Abstract

Autoregressive electronic health record (EHR) foundation models may disclose training information at inference, yet clinical predictability makes patient-specific memorisation difficult to distinguish from generalisation. We evaluate fixed-window token reconstruction, sensitive-attribute disclosure, and random patient-level canaries in GPT-2-style models trained on MIMIC-IV, across early, best-validation, and final checkpoints. Reconstruction improves with additional context, but training--held-out differences remain small. Trajectory-unique prompts containing limited clinical information produce no excess sensitive disclosure for training patients. In a matched model with random binary patient-level canaries, training-patient label-recovery AUROC increases from 0.512 to 0.549 while held-out recovery remains at chance. An overfit 1,000-patient positive control achieves near-perfect recovery, confirming sensitivity to strong memorisation. These results reveal weak cohort-level memorisation but no reliable patient-level disclosure under the evaluated attacks. They motivate clinically grounded inference-time audits that can detect patient-specific memorisation in existing models without retraining with canaries.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Clinically Grounded Privacy Evaluation of Medical LMs

    Jun 8, 2026Sasha Ronaghi, Sana Tonekaboni, Lena Stempfle +6Data LeakageMemorization in Language Models

  2. Pretraining EHR Foundation Models with Patient-Aware Sampling

    Jul 24, 2026Joshua Placidi, Yuxuan Liu, Jinpei Han +2EHR Foundation ModelsLongitudinal EHR Modeling

  3. Memorisation bias in medical AI

    Sep 15, 2026Moritz A. Knolle, Martin J. Menten, Laurin Lux +5Neural Network MemorizationClinical Prediction