Foresight-England: Development of a National-Scale Generative AI Model of Electronic Health Records for Medical Event Prediction across the COVID-19 Pandemic
Authors: Simon Ellershaw, Christopher Tomlinson, Zeljko Kraljevic, Spiros Denaxas, Harry Hemingway, Cathie Sudlow, Angela M. Wood, Anoop D. Shah, Richard Dobson
Organizations: Institute of Health Informatics, University College London, London, UK · Department of Biostatistics and Health Informatics, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, UK · King’s Institute for Artificial Intelligence, King’s College London, London, UK · University College London Hospitals National Institute for Health Research Biomedical Research Centre, London, UK · Interdisciplinary Transformation University, Linz, Austria · British Heart Foundation Data Science Centre, Health Data Research UK, London, UK · Health Data Research UK, London, UK · Usher Institute, School of Population Health Sciences, The University of Edinburgh · British Heart Foundation Cardiovascular Epidemiology Unit, Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK · Victor Phillip Dahdaleh Heart and Lung Research Institute, University of Cambridge, Cambridge, UK · British Heart Foundation Centre of Research Excellence, University of Cambridge, Cambridge, UK · National Institute for Health and Care Research Blood and Transplant Research Unit in Donor Health and Behaviour, University of Cambridge, Cambridge, UK · Health Data Research UK Cambridge, Wellcome Genome Campus and University of Cambridge, Cambridge, UK · Cambridge Centre of Artificial Intelligence in Medicine, University of Cambridge, Cambridge, UK · Department of Clinical Pharmacology, University College London Hospitals NHS Foundation Trust, London, UK · National Institute for Health Research Biomedical Research Centre at South London and Maudsley NHS Foundation Trust and King’s College London, London, UK
Foresight-England (Foresight-E) is the first national-scale generative foundation model of electronic health records (EHRs), developed as a research pilot strictly for COVID-19 research. We evaluated its ability to model the direct and indirect effects of the pandemic. Trained from scratch entirely within the NHS England Secure Data Environment, Foresight-E is a 243-million-parameter transformer decoder. It was trained and evaluated on de-identified, longitudinal EHRs of approximately 61 million individuals, integrating primary/secondary care, death registrations, and COVID-19 data. Training and validation used a 90% subset (54.9 million) spanning November 2018 to December 2022; the remaining 10% (6.1 million) was held out for evaluation. Foresight-E models patient timelines autoregressively, predicting the next medical event given their prior history. At inference, it operates zero-shot, predicting any concept in its ~40,000-code vocabulary without task-specific training. Our tokenisation scheme retains the clinical granularity of ICD-10, OPCS-4, and SNOMED CT codes, jointly representing absolute and relative timing. We designed an evaluation framework for 30-day COVID-19 hospitalisation and mortality, including subgroup analyses by demographic factors and vaccination status. To assess generalisation to unseen future data and the pandemic's indirect effects, we tested the model on medical events from 2023 (beyond its training period), benchmarking against logistic regression and XGBoost. As detailed in the Project Status section, NHS England has paused access to data for the Foresight-E project, meaning quantitative results are currently unavailable. Instead, we share our strategy for tokenisation, architecture, training, inference, and evaluation as a methodological template and case study in the challenges of building population-scale EHR foundation models.
Figures & tables
Train
Validation
Test: 30-day COVID-19
Test: 1 Year
Sex
Female
_ ( _ %)
_ ( _ %)
_ ( _ %)
_ ( _ %)
Male
_ ( _ %)
_ ( _ %)
_ ( _ %)
_ ( _ %)
Ethnicity
Asian
_ ( _ %)
_ ( _ %)
_ ( _ %)
_ ( _ %)
Black
_ ( _ %)
_ ( _ %)
_ ( _ %)
_ ( _ %)
Table 1: PLACEHOLDER Selected characteristics of the training, validation, and test sets (used in Sections 5.1 and 5.2 ). Ages are computed at the time of the last input event in each patient’s timeline. Counts are rounded to the nearest five to comply with the NHSE SDE’s disclosure control requirements. Total sample sizes for each split are derived from previously reported counts [ 14 ] . The rationale for placeholders is provided in Section 5 .
Figure 1: Datasets and cohort splits used in the training and evaluation of Foresight-E. Total sample sizes for each split are derived from previously reported counts [ 14 ] .
Figure 2: Schematic of next token prediction training of Foresight-E on a synthetic patient timeline. Tokens at position IDs 2–7 are omitted for clarity. Given the sequence of tokens up to position i , the model predicts the probability distribution over the vocabulary for the token at position i+1 . These probabilities are compared with the true next token to compute the cross-entropy loss for self-supervised training. Note that the initial static demographic tokens are masked ([Ignore]) during the loss calculation to prevent the model from learning to predict sequence progression solely from demographic features. The final and padding tokens are also ignored because they have no label to compare against.
Figure 3: Schematic of idealised zero-shot inference on a synthetic patient timeline. Given a history ending with a positive SARS-CoV-2 test, Foresight-E predicts a hospital admission within the next 30 days. Tokens 2–7 are omitted for clarity.
Figure 4: Visualisation of an idealised Foresight-E zero-shot inference for the task of 30-day hospitalisation following a positive SARS-CoV-2 test. A tokenised input timeline, shortened for visualisation, is used to prompt three independent predicted timelines in this example. ‘Time Diff‘ tokens are in units of days. In two of the predicted timelines, hospitalisation is predicted within the next 30 days (at 20 and 25 days, respectively). The remaining timeline predicts the next event at 100 days, falling outside the target window. Therefore, the predicted probability is 2/3.
Figure 5: PLACEHOLDER. ROC, PR, and calibration curves showing Foresight-E’s prediction performance for 30-day mortality and hospitalisation following a positive SARS-CoV-2 test. Shaded regions represent 95% confidence intervals estimated via 1,000 bootstrap iterations. Counts are rounded to the nearest five to comply with the NHSE SDE’s disclosure control requirements.
Figure 6: PLACEHOLDER. Temporal evaluation of 30-day mortality prediction across the COVID-19 pandemic. Monthly performance of Foresight-E in predicting death within 30 days of a positive SARS-CoV-2 test, measured by AUROC (top panel) and AUPRC (second panel). Error bars represent 95% confidence intervals estimated via 1,000 bootstrap iterations. Also shown are monthly mortality occurrence rates (third panel) and the number of positive SARS-CoV-2 tests (bottom panel), illustrating changes in disease dynamics over time. Shaded background regions denote the predominant SARS-CoV-2 variants [ 60 ] with labels shown along the upper x-axis. Counts are rounded to the nearest five to comply with the NHSE SDE’s disclosure control requirements.
Figure 7: PLACEHOLDER. Variation in AUROC and AUPRC metrics for the Foresight-E model predicting 30-day mortality after a positive SARS-CoV-2 test, stratified by age, ethnicity, sex, and COVID-19 vaccination status. Error bars show 95% confidence intervals calculated by 1,000 bootstrap iterations. Also shown is the mortality occurrence rate, as well as the count of patients for each group. Performance is consistent across ethnicity and sex, but varies notably by age group and vaccination status. Counts are rounded to the nearest five to comply with the NHSE SDE’s disclosure control requirements.
Figure 8: PLACEHOLDER. Variation in AUROC and AUPRC metrics for the Foresight-E model predicting 30-day mortality after a positive SARS-CoV-2 test with the number of past medical events recorded in a patient’s timeline. Also shown are the binned frequency of past events and corresponding mortality occurrence rates. A bin size of 16 events was used, and the final bin includes all patients with more than 498 events. Counts are rounded to the nearest five to comply with the NHSE SDE’s disclosure control requirements.
Figure 9: PLACEHOLDER. Prediction performance of Foresight-E in 2023 using ROC, PR, and calibration curves. Shaded regions represent 95% confidence intervals estimated via 1,000 bootstrap iterations. Counts are rounded to the nearest five to comply with the NHSE SDE’s disclosure control requirements.
Figure 10: PLACEHOLDER. Relationship between AUROC and event occurrence rate for Foresight-E’s 1-year prediction in 2023. _ AUROCs were observed for more _ Phecodes, such as _ .
While large-scale pretraining has revolutionized language modeling, its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present RAVEN, a novel generative pretraining strategy for sequential EHR data based on Recurrence-Aware next-Visit EveNt prediction. Leveraging a dataset of over one million unique individuals, our model learns to autoregressively generate tokenized clinical events for the next visit conditioned on patient history. We introduce regularization on predicting repeated events and highlight a key pitfall in EHR-based foundation model evaluations: repeated event tokens can inflate performance metrics when new onsets are not distinguished from subsequent occurrences. Furthermore, we empirically investigate the scaling behaviors in a data-constrained, compute-saturated regime, showing that simply increasing model size is suboptimal without commensurate increases in data volume. We evaluate our model via zero-shot prediction for forecasting the incidence of a diverse set of diseases, where it rivals fully fine-tuned representation-based Transformer models and outperforms both standard simulation-based next-token approaches and a prompted medical large language model baseline. Finally, without additional parameter updates, we show that RAVEN can generalize to an external patient cohort under lossy clinical code mappings and feature coverage gaps.
Center for Data Science, New York University, New York, NY, USA · Department of Radiology, NYU Grossman School of Medicine, New York, NY, USA · Center for Biomedical Imaging, NYU Grossman School of Medicine, New York, NY, USA +1
Accurate disease trajectory prediction is critical for early intervention, resource allocation, and improving long-term outcomes. While electronic health records (EHRs) provide a rich longitudinal view of patient health in clinical environments, models trained on curated research cohorts may not reflect routine deployment settings, and those trained on single-hospital datasets capture only fragments of each patient's trajectory. This highlights the importance of leveraging large, multi-hospital health systems for training and validation to better reflect real-world clinical complexity. In this work, we develop DT-Transformer, a foundation model trained on 57.1M structured EHR entries over 1.7M patients from Mass General Brigham (MGB), spanning 11 hospitals and a broad network of outpatient clinics. DT-Transformer achieves strong discrimination in both held-out and prospective validation settings. Next-event prediction achieves a median age- and sex-stratified AUC of 0.871 across 896 disease categories, with all categories exceeding AUC 0.5. These results support health system-scale training as a path toward foundation models suited to real-world clinical forecasting.
Yunying Zhu, Andrew R Weckstein, Kueiyu Joshua Lin +1
Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA · Harvard T.H. Chan School of Public Health, Harvard University, Boston, MA, USA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, MA, USA +2
Autoregressive electronic health record (EHR) foundation models predict the next event token in patient trajectories: sequences of structured medical events. A typical pretraining strategy is to concatenate all trajectories into a global stream and sample token windows from it. This gives patients with longer sequences more training exposure and allows tokens to attend across patient boundaries. We argue that these choices create a misalignment between pretraining and downstream clinical tasks. To explore this, we investigate how the composition of pretraining sequences affects clinical prediction performance. Using the global-stream approach as a baseline, we propose alternative sequence composition methods that control how patient trajectories contribute to pretraining sequences. Across four MIMIC-IV v2.2 tasks, our best method improves macro AUROC, AUPRC, F1, and balanced accuracy, with relative gains of 9.3% in F1 and 2.7% in AUPRC under a fixed training recipe. We find that these gains generalize across four additional evaluation settings, including the INSPIRE and eICU-CRD datasets, with improvements in macro AUROC, AUPRC, and F1 score. Our results show that simply changing how EHR data are presented to the model during pretraining can improve downstream performance, without changing the model or collecting additional data.
Joshua Placidi, Inigo Stibbe, Alfred J. Balston +2