Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction
Authors: Haresh Rengaraj Rajamohan, Xiang Gao, Weicheng Zhu, Shih-Lun Huang, Long Chen, Gabe Schulman, Huizhen Jin, Shengduo Li, +5 more
Organizations: Center for Data Science, New York University, New York, NY, USA · Department of Radiology, NYU Grossman School of Medicine, New York, NY, USA · Center for Biomedical Imaging, NYU Grossman School of Medicine, New York, NY, USA · Department of Population Health, NYU Grossman School of Medicine, New York, NY, USA
While large-scale pretraining has revolutionized language modeling, its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present RAVEN, a novel generative pretraining strategy for sequential EHR data based on Recurrence-Aware next-Visit EveNt prediction. Leveraging a dataset of over one million unique individuals, our model learns to autoregressively generate tokenized clinical events for the next visit conditioned on patient history. We introduce regularization on predicting repeated events and highlight a key pitfall in EHR-based foundation model evaluations: repeated event tokens can inflate performance metrics when new onsets are not distinguished from subsequent occurrences. Furthermore, we empirically investigate the scaling behaviors in a data-constrained, compute-saturated regime, showing that simply increasing model size is suboptimal without commensurate increases in data volume. We evaluate our model via zero-shot prediction for forecasting the incidence of a diverse set of diseases, where it rivals fully fine-tuned representation-based Transformer models and outperforms both standard simulation-based next-token approaches and a prompted medical large language model baseline. Finally, without additional parameter updates, we show that RAVEN can generalize to an external patient cohort under lossy clinical code mappings and feature coverage gaps.
Figures & tables
Figure 1: Study overview of recurrence-aware next-visit foundation modeling on longitudinal EHRs. a, Patient trajectories are represented as temporally ordered visits containing unordered clinical events. During pretraining, a time-coded separator token [SEP] predicts the full event set at the next visit; during zero-shot inference, the same interface queries disease risk at future horizons by appending a separator token at time t+H and pooling logits over condition-specific code sets. b, Repeated chronic targets are downweighted during pretraining according to their prior count so that rare first onsets retain training signal. c, In a finite EHR corpus revisited for many epochs, model size must match the corresponding available data as larger models eventually overfit, and the selected 144M model lies near the full-data optimum. d, Model development uses a patient-level 70%/15%/15% split on patient data from NYU Langone; a schematic external zero-shot transfer setting based on Stanford EHRSHOT highlights ontology harmonization between benchmark concepts and the institutional token space, which can introduce information loss and drop of certain features.
Condition
RAVEN
Multiclass
SeqLoss
EGE
MedGemma
BERT (FT)
Dementia
0.789 / 0.037
0.687 / 0.026
0.704 / 0.031
0.677 / 0.026
0.620 / 0.011
0.731 / 0.050
Knee OA
0.726 / 0.057
0.627 / 0.039
0.672 / 0.053
0.606 / 0.038
0.642 / 0.039
0.744 / 0.064
COPD
0.691 / 0.052
0.544 / 0.036
0.629 / 0.044
0.602 / 0.040
0.589 / 0.030
0.704 / 0.055
CHF
0.857 / 0.078
0.708 / 0.047
0.733 / 0.048
0.656 / 0.045
0.736 / 0.036
0.862 / 0.085
Acute MI
0.793 / 0.031
0.603 / 0.015
0.579 / 0.013
0.604 / 0.017
0.748 / 0.026
0.818 / 0.049
Pancreatic Cancer
0.693 / 0.002
0.496 / 0.000
0.494 / 0.000
0.502 / 0.001
0.620 / 0.001
0.607 / 0.001
Table 1: Zero-shot forecasting at the 2-year horizon (AUROC / AUPRC). All methods except BERT (FT) operate without task-specific fine-tuning. RAVEN uses a single forward pass; simulation-based models aggregate R=100 rollouts. Baseline prevalence levels and task-specific statistics can be found in the Supplementary Table S4. Bold denotes the best AUROC per condition.
Condition
RAVEN
Multiclass
SeqLoss
EGE
MedGemma
BERT (FT)
Dementia
0.773 / 0.108
0.669 / 0.089
0.693 / 0.102
0.670 / 0.088
0.609 / 0.041
0.721 / 0.146
Knee OA
0.697 / 0.200
0.635 / 0.166
0.671 / 0.190
0.620 / 0.166
0.617 / 0.146
0.718 / 0.221
COPD
0.676 / 0.164
0.531 / 0.124
0.640 / 0.169
0.615 / 0.149
0.586 / 0.115
0.699 / 0.189
CHF
0.821 / 0.184
0.718 / 0.114
0.723 / 0.134
0.666 / 0.127
0.718 / 0.101
0.838 / 0.220
Acute MI
0.754 / 0.080
0.613 / 0.060
0.569 / 0.047
0.623 / 0.059
0.699 / 0.067
0.782 / 0.106
Pancreatic Cancer
0.639 / 0.004
0.516 / 0.002
0.513 / 0.004
0.498 / 0.001
0.597 / 0.002
0.642 / 0.003
Table 2: Zero-shot forecasting at the 5-year horizon (AUROC / AUPRC). The same evaluation protocol is used as in Table 1 . Bold denotes the best AUROC per condition.
Figure 2: Zero-shot generalization of RAVEN for new diagnosis predictions on EHRSHOT. All baseline models (CLMBR-T, GBM, Logistic Regression, Random Forest) are trained on varying numbers of labeled examples per class ( K ) drawn from the training dataset, whereas RAVEN is evaluated in a zero-shot setting with no target-domain supervision. a , AUROC as a function of the number of training examples per class. Solid lines denote baseline models with s.d. shading across random seeds; dashed horizontal lines indicate RAVEN performance at K=0 for three regularization strengths ( λ=1.0,0.5,0.25 ), with 95% confidence intervals shown at the zero-shot axis. RAVEN with different regularization strengths matches or exceeds baselines trained on different number of labeled examples for conditions including acute MI, hyperlipidemia, and hypertension. b , Comparison to baselines at K=all (full EHRSHOT training set). RAVEN can be competitive on certain conditions with fully fine-tuned models despite having seen zero training examples. Error bars denote 95% confidence intervals.
Figure 3: Effect of history-dependent regularization strength. We sweep the decay parameter λ that downweights repeated target events during training and report on-time rate and AUPRC across conditions. All main downstream results use a single global setting with intermediate level λ⋆=0.5 .
Figure 4: Effect of recurrence-aware regularization on zero-shot disease onset prediction. a, Per-condition AUROC as a function of the decay parameter λ for the 2-year and 5-year horizons, with shaded 95% confidence intervals. The macro-average panel (bottom right) summarizes the overall trend. All results use 144M model trained on the full dataset; lower λ corresponds to stronger penalization of repeated clinical events. b, Change in AUROC from no regularization ( λ = 1.0) across seven conditions at 2-year and 5-year horizons. Bars above zero indicate improved discrimination under regularization. c, Corresponding change in F1 score. Dashed vertical lines separate the 2-year and 5-year evaluations.
Figure 5: Compute-saturated scaling of standard and RAVEN pretraining. a, Validation loss and test loss under the standard next-visit objective as a function of model size across multiple dataset budgets. The fitted minima shift toward larger models as the data budget increases, indicating that the optimal capacity depends strongly on the amount of available training data. b, The same scaling evidence with full RAVEN history-dependent regularization enabled during pretraining, showing consistent behaviors.
Figure 6: Scaling downstream zero-shot disease forecasting performance. a., b. Macro-average AUROC and AUPRC for 2-year ( a ) and 5-year ( b ) zero-shot disease onset prediction with shaded 95% confidence intervals, averaged across seven conditions. At each data budget, the optimal model size from the compute-saturated scaling analysis (Fig. 5) is used: 47M parameters for the 10%, 25%, and 50% subsets, and 144M parameters for the full dataset. All models are trained with recurrence-aware regularization ( λ=0.5 ). c., d. Same zero-shot task metrics 2-year ( c ) and 5-year ( d ) plotted with respect to model sizes. All models are trained with full data and regularization ( λ=0.5 ).
NYU Langone (N=1,288,242)
EHRSHOT (N=6,739)
Attribute
Train
Val
Test
All
Train
Val
Test
All
Gender
Male
347,497
74,128
74,291
495,916
1,122
1,090
1,086
3,298
Female
531,280
114,160
113,995
759,435
1,173
1,142
1,126
3,441
Age
< 20
73,478
15,743
15,597
104,818
8
3
2
13
21–40
228,578
49,002
48,939
326,519
412
457
431
1,300
41–60
291,009
62,040
62,348
415,397
648
597
576
1,821
Table 3: Patient demographics for NYU Langone and EHRSHOT cohorts. Patient demographics in the train, validation, and test splits for the two cohorts. For EHRSHOT, we only used the test set.
Disease
Type
Description
Code
Dementia
Diagnosis
Vascular dementia without behavioral disturbance
F01.50
Dementia
Diagnosis
Vascular dementia with behavioral disturbance
F01.51
Dementia
Diagnosis
Dementia in other diseases w/o behavioral disturbance
F02.80
Dementia
Diagnosis
Dementia in other diseases w/ behavioral disturbance
F02.81
Dementia
Diagnosis
Unspecified dementia without behavioral disturbance
F03.90
Dementia
Diagnosis
Unspecified dementia with behavioral disturbance
F03.91
Table S1 : Diagnosis and medication codes for disease definition and labeling.
Hyperparameter
Value
n_embd
1024
n_head
8
n_layer
8
n_tokens
42,337
bias
false
dropout
0
Table S2 : Training hyperparameters and configuration settings for the main 144M model.
Layers
Heads
Embedding Dim
Parameters
4
4
16
0.69M
2
2
64
2.81M
2
2
128
5.81M
4
4
256
13.99M
8
8
512
46.85M
8
8
1024
144.04M
Table S3 : Model configurations used in the scaling experiments.
Condition
Horizon
Total ( n )
Positive
Negative
Prevalence (%)
COPD
2-year
239,464
5,255
234,209
2.19
COPD
5-year
47,628
4,251
43,377
8.93
CHF
2-year
263,220
2,874
260,346
1.09
CHF
5-year
52,072
2,168
49,904
4.16
Dementia
2-year
271,172
2,021
269,151
0.75
Dementia
5-year
53,520
1,685
51,835
3.15
Table S4 : Evaluation cohort statistics for zero-shot disease-onset forecasting tasks on the held-out test set. Prevalence denotes the fraction of positive examples in each task. We note, that for zero shot evaluations, we exclude those who already have the condition by prediction time and by prediction time + 1 year gap window, and therefore these rates represent fractions of "new onset" within the prediction window.
Condition
Hor.
AUROC
AUPRC
F1
Precision
Recall
CHF
2y
0.854 [0.848, 0.861]
0.078 [0.072, 0.085]
0.140 [0.132, 0.149]
0.097 [0.091, 0.103]
0.254 [0.238, 0.271]
5y
0.826 [0.818, 0.834]
0.189 [0.177, 0.203]
0.267 [0.255, 0.279]
0.190 [0.182, 0.199]
0.445 [0.424, 0.465]
COPD
2y
0.687 [0.680, 0.695]
0.050 [0.048, 0.053]
0.096 [0.091, 0.102]
0.066 [0.062, 0.070]
0.179 [0.168, 0.189]
5y
0.659 [0.651, 0.668]
0.155 [0.149, 0.163]
0.222 [0.215, 0.231]
0.153 [0.148, 0.159]
0.404 [0.389, 0.420]
Dementia
2y
0.781 [0.770, 0.792]
0.035 [0.032, 0.039]
0.082 [0.076, 0.090]
0.051 [0.047, 0.055]
0.213 [0.196, 0.233]
5y
0.787 [0.776, 0.797]
0.121 [0.110, 0.133]
0.180 [0.167, 0.194]
0.136 [0.126, 0.146]
0.268 [0.246, 0.289]
Table S5 : Zero-shot forecasting with λ=0.25 for the 144M model trained on full data.
Condition
Hor.
AUROC
AUPRC
F1
Precision
Recall
CHF
2y
0.857 [0.850, 0.864]
0.078 [0.072, 0.085]
0.147 [0.137, 0.157]
0.111 [0.104, 0.119]
0.216 [0.201, 0.233]
5y
0.821 [0.812, 0.829]
0.184 [0.172, 0.198]
0.257 [0.244, 0.268]
0.186 [0.177, 0.195]
0.414 [0.394, 0.434]
COPD
2y
0.691 [0.684, 0.698]
0.052 [0.049, 0.055]
0.100 [0.095, 0.106]
0.068 [0.064, 0.071]
0.193 [0.184, 0.204]
5y
0.676 [0.668, 0.685]
0.164 [0.157, 0.171]
0.238 [0.230, 0.246]
0.163 [0.158, 0.168]
0.441 [0.426, 0.456]
Dementia
2y
0.789 [0.778, 0.799]
0.037 [0.034, 0.042]
0.083 [0.075, 0.090]
0.053 [0.049, 0.058]
0.182 [0.165, 0.199]
5y
0.773 [0.762, 0.784]
0.108 [0.101, 0.118]
0.182 [0.169, 0.195]
0.136 [0.127, 0.146]
0.274 [0.254, 0.296]
Table S6 : Zero-shot forecasting with λ⋆=0.5 for the 144M model trained on full data.
Condition
Hor.
AUROC
AUPRC
F1
Precision
Recall
CHF
2y
0.855 [0.848, 0.861]
0.075 [0.070, 0.082]
0.138 [0.129, 0.146]
0.096 [0.089, 0.102]
0.246 [0.230, 0.262]
5y
0.824 [0.816, 0.833]
0.189 [0.177, 0.204]
0.273 [0.259, 0.284]
0.202 [0.192, 0.211]
0.418 [0.398, 0.440]
COPD
2y
0.686 [0.679, 0.693]
0.050 [0.048, 0.053]
0.097 [0.092, 0.103]
0.072 [0.067, 0.076]
0.152 [0.143, 0.161]
5y
0.668 [0.659, 0.676]
0.162 [0.155, 0.169]
0.231 [0.225, 0.237]
0.146 [0.142, 0.150]
0.551 [0.537, 0.567]
Dementia
2y
0.766 [0.754, 0.776]
0.034 [0.031, 0.038]
0.081 [0.072, 0.089]
0.057 [0.051, 0.063]
0.142 [0.126, 0.156]
5y
0.761 [0.749, 0.772]
0.109 [0.100, 0.122]
0.182 [0.170, 0.195]
0.129 [0.121, 0.138]
0.309 [0.287, 0.331]
Table S7 : Zero-shot forecasting with λ=0.75 for the 144M model trained on full data.
Condition
Hor.
AUROC
AUPRC
F1
Precision
Recall
CHF
2y
0.856 [0.850, 0.862]
0.073 [0.069, 0.080]
0.139 [0.129, 0.148]
0.104 [0.097, 0.111]
0.207 [0.192, 0.222]
5y
0.830 [0.822, 0.838]
0.186 [0.175, 0.199]
0.268 [0.255, 0.280]
0.202 [0.193, 0.212]
0.396 [0.376, 0.417]
COPD
2y
0.692 [0.684, 0.699]
0.052 [0.049, 0.054]
0.102 [0.096, 0.107]
0.071 [0.067, 0.075]
0.181 [0.170, 0.191]
5y
0.676 [0.667, 0.684]
0.173 [0.166, 0.181]
0.240 [0.232, 0.248]
0.167 [0.162, 0.173]
0.423 [0.409, 0.438]
Dementia
2y
0.772 [0.761, 0.782]
0.036 [0.033, 0.041]
0.087 [0.079, 0.096]
0.059 [0.054, 0.065]
0.167 [0.151, 0.184]
5y
0.770 [0.759, 0.781]
0.110 [0.101, 0.121]
0.179 [0.167, 0.191]
0.127 [0.118, 0.135]
0.305 [0.284, 0.326]
Table S8 : Zero-shot forecasting with λ=1.0 (no reg.) for the 144M model trained on full data.
Model size
Horizon
AUROC
AUPRC
F1
Precision
Recall
0.69M
2y
0.732 [0.719, 0.745]
0.0274 [0.0260, 0.0294]
0.057 [0.052, 0.062]
0.037 [0.034, 0.040]
0.128 [0.116, 0.140]
5y
0.706 [0.689, 0.722]
0.0927 [0.0879, 0.0992]
0.147 [0.136, 0.159]
0.101 [0.093, 0.111]
0.288 [0.270, 0.308]
2.81M
2y
0.764 [0.752, 0.775]
0.0345 [0.0325, 0.0371]
0.072 [0.065, 0.078]
0.048 [0.044, 0.052]
0.158 [0.143, 0.174]
5y
0.748 [0.733, 0.763]
0.1151 [0.1090, 0.1226]
0.173 [0.164, 0.184]
0.120 [0.113, 0.127]
0.333 [0.312, 0.354]
5.81M
2y
0.755 [0.743, 0.768]
0.0345 [0.0325, 0.0374]
0.071 [0.065, 0.077]
0.049 [0.045, 0.053]
0.150 [0.137, 0.163]
5y
0.725 [0.708, 0.741]
0.1094 [0.1035, 0.1168]
0.165 [0.155, 0.175]
0.117 [0.110, 0.124]
0.311 [0.285, 0.338]
Table S9 : Macro-averaged zero-shot forecasting across model sizes (full data, λ=0.5 ).
Data budget
Horizon
AUROC
AUPRC
F1
Precision
Recall
10%
2y
0.755 [0.741, 0.768]
0.0339 [0.0320, 0.0368]
0.069 [0.063, 0.076]
0.046 [0.042, 0.051]
0.154 [0.140, 0.171]
5y
0.723 [0.708, 0.738]
0.1028 [0.0972, 0.1100]
0.159 [0.151, 0.167]
0.108 [0.102, 0.114]
0.314 [0.296, 0.332]
25%
2y
0.771 [0.758, 0.784]
0.0368 [0.0345, 0.0399]
0.076 [0.068, 0.084]
0.052 [0.048, 0.059]
0.157 [0.141, 0.173]
5y
0.739 [0.722, 0.755]
0.1064 [0.1009, 0.1135]
0.170 [0.160, 0.179]
0.114 [0.108, 0.121]
0.372 [0.347, 0.400]
50%
2y
0.764 [0.750, 0.777]
0.0378 [0.0356, 0.0409]
0.077 [0.072, 0.084]
0.051 [0.047, 0.055]
0.183 [0.168, 0.199]
5y
0.740 [0.722, 0.756]
0.1126 [0.1063, 0.1204]
0.171 [0.162, 0.179]
0.115 [0.110, 0.121]
0.338 [0.320, 0.356]
Table S10 : Macro-averaged zero-shot forecasting across data budgets ( λ=0.5 ).
Autoregressive electronic health record (EHR) foundation models predict the next event token in patient trajectories: sequences of structured medical events. A typical pretraining strategy is to concatenate all trajectories into a global stream and sample token windows from it. This gives patients with longer sequences more training exposure and allows tokens to attend across patient boundaries. We argue that these choices create a misalignment between pretraining and downstream clinical tasks. To explore this, we investigate how the composition of pretraining sequences affects clinical prediction performance. Using the global-stream approach as a baseline, we propose alternative sequence composition methods that control how patient trajectories contribute to pretraining sequences. Across four MIMIC-IV v2.2 tasks, our best method improves macro AUROC, AUPRC, F1, and balanced accuracy, with relative gains of 9.3% in F1 and 2.7% in AUPRC under a fixed training recipe. We find that these gains generalize across four additional evaluation settings, including the INSPIRE and eICU-CRD datasets, with improvements in macro AUROC, AUPRC, and F1 score. Our results show that simply changing how EHR data are presented to the model during pretraining can improve downstream performance, without changing the model or collecting additional data.
Joshua Placidi, Inigo Stibbe, Alfred J. Balston +2
Autoregressive foundation models for electronic health records (EHRs) typically inherit pretraining methods from language modeling, where patient trajectories are concatenated into a single token stream and windows are sampled from that stream. In EHR data, this choice is consequential: windows may mix multiple patients, and patients with longer records contribute more optimization updates, potentially introducing bias. We propose Patient Sampling, a pretraining sequence-construction method that allows us to control how training signal is distributed across patients. We compare this method to the standard approach, which we refer to as Global Stream. We show that stochastic Patient Sampling with controllable weighting improves performance on real-world EHR data. Across downstream clinical tasks on MIMIC-IV v2.2 and v3.1, Patient Sampling improves Macro AUROC and AUPRC over the Global Stream baseline. These results identify training and validation sequence construction as important and underexplored design choices for autoregressive EHR foundation models.
Joshua Placidi, Yuxuan Liu, Jinpei Han +2
Dept. of Computing, Imperial College London, London, United Kingdom · Dept. of Bioengineering, Imperial College London, London, United Kingdom · UKRI Centres in AI for Health, United Kingdom +1
Electronic Health Records (EHR) contain rich longitudinal patient information and are widely used in predictive modeling applications. However, effectively leveraging historical data remains challenging due to long trajectories, heterogeneous events, temporal irregularity, and the varying relevance of past clinical context. Existing approaches often rely on fixed windows or uniform aggregation, which can obscure clinically important signals. In this work, we introduce EHR-RAGp, a retrieval-based framework that dynamically integrates the most relevant patient history consisting of diverse clinical event types. We propose a prototype-guided retrieval module that acts as an alignment mechanism and estimates the relevance of retrieved historical chunks with respect to a given prediction task, guiding the model towards the most informative context. Across multiple clinical prediction tasks and two benchmark datasets, EHR-RAGp consistently outperforms state-of- the-art EHR-based and transformer-based baselines. Furthermore, EHR-RAGp is model-agnostic, as integrating it with various backbone models yields substantial performance gains. Overall, EHR-RAGp establishes a novel direction for modeling long-range clinical context to improve downstream performance via retrieval.
Saeed Shurrab, Mariam Al-Omari, Dana El Samad +1
New York University · New York University Abu Dhabi