Autoregressive foundation models for electronic health records can make predictions for diverse clinical tasks without finetuning. These methods work by generating possible "synthetic futures" for patients, and then inferring downstream predictions over these simulated trajectories. Despite their strong capabilities, these methods have several flaws. (1) Inference is extremely computationally expensive, as each prediction requires simulating many trajectories, each with many observations. (2) Their performance is noisy due to the variance inherent to simulation, which in particular reduces efficacy on rare events, which are often of high importance clinically. (3) They are not promptable, as their only input is the patient's medical history, requiring indirect aggregation at inference time. We introduce EveryQuery, a promptable foundation model for structured EHR data. Building on the insight that clinical prediction tasks can be written in a structured query language, EveryQuery takes a query as a prompt alongside the patient's history and estimates the outcome directly. This allows EveryQuery to make predictions across diverse clinical tasks, such as classification and survival modeling, without simulation or finetuning. Across three EHR datasets, EveryQuery outperforms a competitive autoregressive baseline on clinically relevant classification and time-to-event tasks, with mean AUROC gains of 14.3% and 12.7% on MIMIC-IV and a large academic medical center and comparable performance on the smaller NWICU dataset. Its advantage is largest for rare outcomes, and it requires 580x less inference compute.
Figures & tables
Figure 1: Overview of EveryQuery. Both models answer a query q during inference (§ 3.1 ), estimating whether a code occurs before a given time, for a patient history x . (a) An autoregressive model generates K future trajectories of up to T events each, and estimates the probability as the fraction of trajectories in which the code occurs. This yields quantized, high-variance estimates. (b) EveryQuery takes q as input alongside x and predicts p(y=1∣x,q) directly.
Task
Conditional Clauses (qi,ai)
Target qn
In-hospital mortality
–
(death, (0, discharge))
30-day readmission
–
(admission, (discharge, 30d))
Post-discharge mortality
((death, (0, discharge)), 0)
(death, (discharge, 90d))
Table 1: Example prompts in the query language.
Figure 2: EveryQuery outperforms the autoregressive baseline across clinical prediction tasks. Each point is a task, comparing EveryQuery with the AR baseline on AUROC for classification tasks (a) and Harrell’s C for time-to-event tasks (b). Points above the diagonal favor EveryQuery.
Figure 3: EveryQuery’s advantage over the autoregressive baseline is largest for rare events. Each row is a category of clinical prediction tasks, and each dot shows the mean Δ AUROC for the tasks in that category grouped by prevalence. ICU tasks (a) are evaluated on MIMIC-IV and NWICU; and longitudinal care tasks (b) on AMC. EveryQuery’s gains grow as outcomes become rarer.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Subjects
Dataset
Train
Val
Test
Total Events
Vocab
MIMIC
200,773
25,059
15,059
779,733,629
11,476
AMC
1,651,030
291,538
27,019
2,250,292,868
21,289
NWICU
20,723
2,590
772
54,785,364
2,159
Appendix
Table 2: Dataset statistics per split after preprocessing.
Hyperparameter
Value
Optimization
Optimizer
AdamW
Learning rate
5×10−4
Betas
(0.9,0.98)
Epsilon
10−8
Weight decay
0.05
Appendix
Table 3: EveryQuery training and architecture hyperparameters.
Task
Prev.
AUROC
EveryQuery
Autoregressive
Visit/OP within 365d | no TIMELINE//END within 365d, no MEDS_DEATH within 365d
0.730
0.543
0.720
Visit/ER within 365d | no TIMELINE//END within 365d, no MEDS_DEATH within 365d
0.114
0.728
0.821
MEDS_DEATH within 1825d
0.028
0.887
0.838
Visit/ER within 30d | no TIMELINE//END within 30d, no MEDS_DEATH within 30d
0.025
0.662
0.762
MEDS_DEATH within 365d
0.010
0.921
0.855
Appendix
Table 4: AMC outpatient visit, general cohort ( n=5 tasks).
Task
Prev.
AUROC
EveryQuery
Autoregressive
LOINC/2823-3//ANY within 90d | no TIMELINE//END within 90d, no MEDS_DEATH within 90d
0.623
0.799
0.827
LOINC/2823-3//UCUM/mmol/L//value_[5.0,inf) within 365d | no TIMELINE//END within 365d, no MEDS_DEATH within 365d
0.354
0.769
0.757
ICD10CM/I48.91 within 1095d | no TIMELINE//END within 1095d
0.284
0.828
0.799
Visit/IP within 365d | no TIMELINE//END within 365d, no MEDS_DEATH within 365d, no RxNorm/1373473 within 30d
0.222
0.741
0.697
Visit/IP within 365d | no TIMELINE//END within 365d, no MEDS_DEATH within 365d, no RxNorm/1545658 within 30d
ICU_DISCHARGE within 7d | no TIMELINE//END within 7d, no MEDS_DEATH before ICU_DISCHARGE
0.882
0.782
0.710
ICU_DISCHARGE within 3d | no TIMELINE//END within 3d, no MEDS_DEATH before ICU_DISCHARGE
0.712
0.796
0.629
ICU_DISCHARGE within 2d | no TIMELINE//END within 2d, no MEDS_DEATH before ICU_DISCHARGE
0.599
0.798
0.595
ICU_DISCHARGE within 1d | no TIMELINE//END within 1d, no MEDS_DEATH before ICU_DISCHARGE
0.390
0.798
0.564
MEDS_DEATH within 365d
0.293
0.831
0.727
Appendix
Table 7: MIMIC-IV ICU, 24h after admission ( n=38 tasks).
Task
Prev.
AUROC
EveryQuery
Autoregressive
ICU_DISCHARGE within 7d | no TIMELINE//END within 7d, no MEDS_DEATH before ICU_DISCHARGE
0.875
0.721
0.528
ICU_DISCHARGE within 3d | no TIMELINE//END within 3d, no MEDS_DEATH before ICU_DISCHARGE
0.686
0.706
0.619
ICU_DISCHARGE within 2d | no TIMELINE//END within 2d, no MEDS_DEATH before ICU_DISCHARGE
0.565
0.688
0.661
ICU_DISCHARGE within 1d | no TIMELINE//END within 1d, no MEDS_DEATH before ICU_DISCHARGE
0.369
0.676
0.693
MEDS_DEATH within 365d
0.203
0.817
0.737
Appendix
Table 8: NWICU ICU, 24h after admission ( n=27 tasks).
Dataset
Target code
MIMIC-IV
MEDS_DEATH
PROCEDURE//START//225792
ICU_ADMISSION//Medical Intensive Care Unit (MICU)
MEDICATION//START//Norepinephrine
HOSPITAL_DISCHARGE//HOME
INFUSION_START//221289//value_[0.35491893,inf)
Appendix
Table 9: Target codes for time-to-event evaluation. Codes are written as they appear in each dataset’s vocabulary.
Figure 4: Survival curves on MIMIC-IV. Each panel shows one target code (Table 9 ), comparing the Kaplan–Meier estimate on held-out patients (black, dashed) with the survival curves estimated by the autoregressive baseline (blue) and EveryQuery (orange) at each prediction horizon in days.
Figure 5: Survival curves on NWICU. Each panel shows one target code (Table 9 ), comparing the Kaplan–Meier estimate on held-out patients (black, dashed) with the survival curves estimated by the autoregressive baseline (blue) and EveryQuery (orange) at each prediction horizon in days.
Figure 6: Survival curves on AMC. Each panel shows one target code (Table 9 ), comparing the Kaplan–Meier estimate on held-out patients (black, dashed) with the survival curves estimated by the autoregressive baseline (blue) and EveryQuery (orange) at each prediction horizon in days.
Figure 7: On randomly sampled tasks, EveryQuery’s advantage over the autoregressive baseline grows as events become rarer. Each point is one randomly sampled task (§ 5.5 ), plotted by prevalence (log scale) against Δ AUROC (EveryQuery minus autoregressive). Red lines show linear fits, and each panel reports the Spearman correlation between prevalence and Δ AUROC.
Electronic Health Records (EHR) contain rich longitudinal patient information and are widely used in predictive modeling applications. However, effectively leveraging historical data remains challenging due to long trajectories, heterogeneous events, temporal irregularity, and the varying relevance of past clinical context. Existing approaches often rely on fixed windows or uniform aggregation, which can obscure clinically important signals. In this work, we introduce EHR-RAGp, a retrieval-based framework that dynamically integrates the most relevant patient history consisting of diverse clinical event types. We propose a prototype-guided retrieval module that acts as an alignment mechanism and estimates the relevance of retrieved historical chunks with respect to a given prediction task, guiding the model towards the most informative context. Across multiple clinical prediction tasks and two benchmark datasets, EHR-RAGp consistently outperforms state-of- the-art EHR-based and transformer-based baselines. Furthermore, EHR-RAGp is model-agnostic, as integrating it with various backbone models yields substantial performance gains. Overall, EHR-RAGp establishes a novel direction for modeling long-range clinical context to improve downstream performance via retrieval.
Saeed Shurrab, Mariam Al-Omari, Dana El Samad +1
New York University · New York University Abu Dhabi
Autoregressive foundation models for electronic health records (EHRs) typically inherit pretraining methods from language modeling, where patient trajectories are concatenated into a single token stream and windows are sampled from that stream. In EHR data, this choice is consequential: windows may mix multiple patients, and patients with longer records contribute more optimization updates, potentially introducing bias. We propose Patient Sampling, a pretraining sequence-construction method that allows us to control how training signal is distributed across patients. We compare this method to the standard approach, which we refer to as Global Stream. We show that stochastic Patient Sampling with controllable weighting improves performance on real-world EHR data. Across downstream clinical tasks on MIMIC-IV v2.2 and v3.1, Patient Sampling improves Macro AUROC and AUPRC over the Global Stream baseline. These results identify training and validation sequence construction as important and underexplored design choices for autoregressive EHR foundation models.
Joshua Placidi, Yuxuan Liu, Jinpei Han +2
Dept. of Computing, Imperial College London, London, United Kingdom · Dept. of Bioengineering, Imperial College London, London, United Kingdom · UKRI Centres in AI for Health, United Kingdom +1
Autoregressive electronic health record (EHR) foundation models predict the next event token in patient trajectories: sequences of structured medical events. A typical pretraining strategy is to concatenate all trajectories into a global stream and sample token windows from it. This gives patients with longer sequences more training exposure and allows tokens to attend across patient boundaries. We argue that these choices create a misalignment between pretraining and downstream clinical tasks. To explore this, we investigate how the composition of pretraining sequences affects clinical prediction performance. Using the global-stream approach as a baseline, we propose alternative sequence composition methods that control how patient trajectories contribute to pretraining sequences. Across four MIMIC-IV v2.2 tasks, our best method improves macro AUROC, AUPRC, F1, and balanced accuracy, with relative gains of 9.3% in F1 and 2.7% in AUPRC under a fixed training recipe. We find that these gains generalize across four additional evaluation settings, including the INSPIRE and eICU-CRD datasets, with improvements in macro AUROC, AUPRC, and F1 score. Our results show that simply changing how EHR data are presented to the model during pretraining can improve downstream performance, without changing the model or collecting additional data.
Joshua Placidi, Inigo Stibbe, Alfred J. Balston +2