Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction

Haresh Rengaraj Rajamohan; Xiang Gao; Weicheng Zhu; Shih-Lun Huang; Long Chen; Gabe Schulman; Huizhen Jin; Shengduo Li

cs.LGMar 25, 2026

Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction

Authors: Haresh Rengaraj Rajamohan, Xiang Gao, Weicheng Zhu, Shih-Lun Huang, Long Chen, Gabe Schulman, Huizhen Jin, Shengduo Li, +5 more

Organizations: Center for Data Science, New York University, New York, NY, USA · Department of Radiology, NYU Grossman School of Medicine, New York, NY, USA · Center for Biomedical Imaging, NYU Grossman School of Medicine, New York, NY, USA · Department of Population Health, NYU Grossman School of Medicine, New York, NY, USA

Abstract

While large-scale pretraining has revolutionized language modeling, its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present RAVEN, a novel generative pretraining strategy for sequential EHR data based on Recurrence-Aware next-Visit EveNt prediction. Leveraging a dataset of over one million unique individuals, our model learns to autoregressively generate tokenized clinical events for the next visit conditioned on patient history. We introduce regularization on predicting repeated events and highlight a key pitfall in EHR-based foundation model evaluations: repeated event tokens can inflate performance metrics when new onsets are not distinguished from subsequent occurrences. Furthermore, we empirically investigate the scaling behaviors in a data-constrained, compute-saturated regime, showing that simply increasing model size is suboptimal without commensurate increases in data volume. We evaluate our model via zero-shot prediction for forecasting the incidence of a diverse set of diseases, where it rivals fully fine-tuned representation-based Transformer models and outperforms both standard simulation-based next-token approaches and a prompted medical large language model baseline. Finally, without additional parameter updates, we show that RAVEN can generalize to an external patient cohort under lossy clinical code mappings and feature coverage gaps.

Figures & tables

Figure 1: Study overview of recurrence-aware next-visit foundation modeling on longitudinal EHRs. a, Patient trajectories are represented as temporally ordered visits containing unordered clinical events. During pretraining, a time-coded separator token [SEP] predicts the full event set at the next visit; during zero-shot inference, the same interface queries disease risk at future horizons by appending a separator token at time $t+H$ and pooling logits over condition-specific code sets. b, Repeated chronic targets are downweighted during pretraining according to their prior count so that rare first onsets retain training signal. c, In a finite EHR corpus revisited for many epochs, model size must match the corresponding available data as larger models eventually overfit, and the selected 144M model lies near the full-data optimum. d, Model development uses a patient-level 70%/15%/15% split on patient data from NYU Langone; a schematic external zero-shot transfer setting based on Stanford EHRSHOT highlights ontology harmonization between benchmark concepts and the institutional token space, which can introduce information loss and drop of certain features.

Condition	RAVEN	Multiclass	SeqLoss	EGE	MedGemma	BERT (FT)
Dementia	0.789 / 0.037	0.687 / 0.026	0.704 / 0.031	0.677 / 0.026	0.620 / 0.011	0.731 / 0.050
Knee OA	0.726 / 0.057	0.627 / 0.039	0.672 / 0.053	0.606 / 0.038	0.642 / 0.039	0.744 / 0.064
COPD	0.691 / 0.052	0.544 / 0.036	0.629 / 0.044	0.602 / 0.040	0.589 / 0.030	0.704 / 0.055
CHF	0.857 / 0.078	0.708 / 0.047	0.733 / 0.048	0.656 / 0.045	0.736 / 0.036	0.862 / 0.085
Acute MI	0.793 / 0.031	0.603 / 0.015	0.579 / 0.013	0.604 / 0.017	0.748 / 0.026	0.818 / 0.049
Pancreatic Cancer	0.693 / 0.002	0.496 / 0.000	0.494 / 0.000	0.502 / 0.001	0.620 / 0.001	0.607 / 0.001

Table 1: Zero-shot forecasting at the 2-year horizon (AUROC / AUPRC). All methods except BERT (FT) operate without task-specific fine-tuning. RAVEN uses a single forward pass; simulation-based models aggregate

R{=}100

rollouts. Baseline prevalence levels and task-specific statistics can be found in the Supplementary Table S4. Bold denotes the best AUROC per condition.

Condition	RAVEN	Multiclass	SeqLoss	EGE	MedGemma	BERT (FT)
Dementia	0.773 / 0.108	0.669 / 0.089	0.693 / 0.102	0.670 / 0.088	0.609 / 0.041	0.721 / 0.146
Knee OA	0.697 / 0.200	0.635 / 0.166	0.671 / 0.190	0.620 / 0.166	0.617 / 0.146	0.718 / 0.221
COPD	0.676 / 0.164	0.531 / 0.124	0.640 / 0.169	0.615 / 0.149	0.586 / 0.115	0.699 / 0.189
CHF	0.821 / 0.184	0.718 / 0.114	0.723 / 0.134	0.666 / 0.127	0.718 / 0.101	0.838 / 0.220
Acute MI	0.754 / 0.080	0.613 / 0.060	0.569 / 0.047	0.623 / 0.059	0.699 / 0.067	0.782 / 0.106
Pancreatic Cancer	0.639 / 0.004	0.516 / 0.002	0.513 / 0.004	0.498 / 0.001	0.597 / 0.002	0.642 / 0.003

Table 2: Zero-shot forecasting at the 5-year horizon (AUROC / AUPRC). The same evaluation protocol is used as in Table 1 . Bold denotes the best AUROC per condition.

Panel of Figure 2 — Figure 2: Zero-shot generalization of RAVEN for new diagnosis predictions on EHRSHOT. All baseline models (CLMBR-T, GBM, Logistic Regression, Random Forest) are trained on varying numbers of labeled examples per class ( $K$ ) drawn from the training dataset, whereas RAVEN is evaluated in a zero-shot setting with no target-domain supervision. a , AUROC as a function of the number of training examples per class. Solid lines denote baseline models with s.d. shading across random seeds; dashed horizontal lines indicate RAVEN performance at $K=0$ for three regularization strengths ( $\lambda=1.0,0.5,0.25$ ), with 95% confidence intervals shown at the zero-shot axis. RAVEN with different regularization strengths matches or exceeds baselines trained on different number of labeled examples for conditions including acute MI, hyperlipidemia, and hypertension. b , Comparison to baselines at $K=\text{all}$ (full EHRSHOT training set). RAVEN can be competitive on certain conditions with fully fine-tuned models despite having seen zero training examples. Error bars denote 95% confidence intervals.

Figure 3: Effect of history-dependent regularization strength. We sweep the decay parameter

\lambda

that downweights repeated target events during training and report on-time rate and AUPRC across conditions. All main downstream results use a single global setting with intermediate level

\lambda^{\star}=0.5

Figure 4: Effect of recurrence-aware regularization on zero-shot disease onset prediction. a, Per-condition AUROC as a function of the decay parameter

\lambda

for the 2-year and 5-year horizons, with shaded 95% confidence intervals. The macro-average panel (bottom right) summarizes the overall trend. All results use 144M model trained on the full dataset; lower

\lambda

corresponds to stronger penalization of repeated clinical events. b, Change in AUROC from no regularization (

\lambda

= 1.0) across seven conditions at 2-year and 5-year horizons. Bars above zero indicate improved discrimination under regularization. c, Corresponding change in F1 score. Dashed vertical lines separate the 2-year and 5-year evaluations.

Figure 5: Compute-saturated scaling of standard and RAVEN pretraining. a, Validation loss and test loss under the standard next-visit objective as a function of model size across multiple dataset budgets. The fitted minima shift toward larger models as the data budget increases, indicating that the optimal capacity depends strongly on the amount of available training data. b, The same scaling evidence with full RAVEN history-dependent regularization enabled during pretraining, showing consistent behaviors.

Figure 6: Scaling downstream zero-shot disease forecasting performance. a., b. Macro-average AUROC and AUPRC for 2-year ( a ) and 5-year ( b ) zero-shot disease onset prediction with shaded 95% confidence intervals, averaged across seven conditions. At each data budget, the optimal model size from the compute-saturated scaling analysis (Fig. 5) is used: 47M parameters for the 10%, 25%, and 50% subsets, and 144M parameters for the full dataset. All models are trained with recurrence-aware regularization (

\lambda=0.5

). c., d. Same zero-shot task metrics 2-year ( c ) and 5-year ( d ) plotted with respect to model sizes. All models are trained with full data and regularization (

\lambda=0.5

		NYU Langone (N=1,288,242)				EHRSHOT (N=6,739)
Attribute		Train	Val	Test	All	Train	Val	Test	All
Gender	Male	347,497	74,128	74,291	495,916	1,122	1,090	1,086	3,298
	Female	531,280	114,160	113,995	759,435	1,173	1,142	1,126	3,441
Age	< 20	73,478	15,743	15,597	104,818	8	3	2	13
	21–40	228,578	49,002	48,939	326,519	412	457	431	1,300
	41–60	291,009	62,040	62,348	415,397	648	597	576	1,821

Table 3: Patient demographics for NYU Langone and EHRSHOT cohorts. Patient demographics in the train, validation, and test splits for the two cohorts. For EHRSHOT, we only used the test set.

Disease	Type	Description	Code
Dementia	Diagnosis	Vascular dementia without behavioral disturbance	F01.50
Dementia	Diagnosis	Vascular dementia with behavioral disturbance	F01.51
Dementia	Diagnosis	Dementia in other diseases w/o behavioral disturbance	F02.80
Dementia	Diagnosis	Dementia in other diseases w/ behavioral disturbance	F02.81
Dementia	Diagnosis	Unspecified dementia without behavioral disturbance	F03.90
Dementia	Diagnosis	Unspecified dementia with behavioral disturbance	F03.91

Table S1 : Diagnosis and medication codes for disease definition and labeling.

Hyperparameter	Value
n_embd	1024
n_head	8
n_layer	8
n_tokens	42,337
bias	false
dropout	0

Table S2 : Training hyperparameters and configuration settings for the main 144M model.

Layers	Heads	Embedding Dim	Parameters
4	4	16	0.69M
2	2	64	2.81M
2	2	128	5.81M
4	4	256	13.99M
8	8	512	46.85M
8	8	1024	144.04M

Table S3 : Model configurations used in the scaling experiments.

Condition	Horizon	Total ( $n$ )	Positive	Negative	Prevalence (%)
COPD	2-year	239,464	5,255	234,209	2.19
COPD	5-year	47,628	4,251	43,377	8.93
CHF	2-year	263,220	2,874	260,346	1.09
CHF	5-year	52,072	2,168	49,904	4.16
Dementia	2-year	271,172	2,021	269,151	0.75
Dementia	5-year	53,520	1,685	51,835	3.15

Table S4 : Evaluation cohort statistics for zero-shot disease-onset forecasting tasks on the held-out test set. Prevalence denotes the fraction of positive examples in each task. We note, that for zero shot evaluations, we exclude those who already have the condition by prediction time and by prediction time + 1 year gap window, and therefore these rates represent fractions of "new onset" within the prediction window.

Condition	Hor.	AUROC	AUPRC	F1	Precision	Recall
CHF	2y	0.854 [0.848, 0.861]	0.078 [0.072, 0.085]	0.140 [0.132, 0.149]	0.097 [0.091, 0.103]	0.254 [0.238, 0.271]
	5y	0.826 [0.818, 0.834]	0.189 [0.177, 0.203]	0.267 [0.255, 0.279]	0.190 [0.182, 0.199]	0.445 [0.424, 0.465]
COPD	2y	0.687 [0.680, 0.695]	0.050 [0.048, 0.053]	0.096 [0.091, 0.102]	0.066 [0.062, 0.070]	0.179 [0.168, 0.189]
	5y	0.659 [0.651, 0.668]	0.155 [0.149, 0.163]	0.222 [0.215, 0.231]	0.153 [0.148, 0.159]	0.404 [0.389, 0.420]
Dementia	2y	0.781 [0.770, 0.792]	0.035 [0.032, 0.039]	0.082 [0.076, 0.090]	0.051 [0.047, 0.055]	0.213 [0.196, 0.233]
	5y	0.787 [0.776, 0.797]	0.121 [0.110, 0.133]	0.180 [0.167, 0.194]	0.136 [0.126, 0.146]	0.268 [0.246, 0.289]

Table S5 : Zero-shot forecasting with

\lambda=0.25

for the 144M model trained on full data.

Condition	Hor.	AUROC	AUPRC	F1	Precision	Recall
CHF	2y	0.857 [0.850, 0.864]	0.078 [0.072, 0.085]	0.147 [0.137, 0.157]	0.111 [0.104, 0.119]	0.216 [0.201, 0.233]
	5y	0.821 [0.812, 0.829]	0.184 [0.172, 0.198]	0.257 [0.244, 0.268]	0.186 [0.177, 0.195]	0.414 [0.394, 0.434]
COPD	2y	0.691 [0.684, 0.698]	0.052 [0.049, 0.055]	0.100 [0.095, 0.106]	0.068 [0.064, 0.071]	0.193 [0.184, 0.204]
	5y	0.676 [0.668, 0.685]	0.164 [0.157, 0.171]	0.238 [0.230, 0.246]	0.163 [0.158, 0.168]	0.441 [0.426, 0.456]
Dementia	2y	0.789 [0.778, 0.799]	0.037 [0.034, 0.042]	0.083 [0.075, 0.090]	0.053 [0.049, 0.058]	0.182 [0.165, 0.199]
	5y	0.773 [0.762, 0.784]	0.108 [0.101, 0.118]	0.182 [0.169, 0.195]	0.136 [0.127, 0.146]	0.274 [0.254, 0.296]

Table S6 : Zero-shot forecasting with

\lambda^{\star}=0.5

for the 144M model trained on full data.

Condition	Hor.	AUROC	AUPRC	F1	Precision	Recall
CHF	2y	0.855 [0.848, 0.861]	0.075 [0.070, 0.082]	0.138 [0.129, 0.146]	0.096 [0.089, 0.102]	0.246 [0.230, 0.262]
	5y	0.824 [0.816, 0.833]	0.189 [0.177, 0.204]	0.273 [0.259, 0.284]	0.202 [0.192, 0.211]	0.418 [0.398, 0.440]
COPD	2y	0.686 [0.679, 0.693]	0.050 [0.048, 0.053]	0.097 [0.092, 0.103]	0.072 [0.067, 0.076]	0.152 [0.143, 0.161]
	5y	0.668 [0.659, 0.676]	0.162 [0.155, 0.169]	0.231 [0.225, 0.237]	0.146 [0.142, 0.150]	0.551 [0.537, 0.567]
Dementia	2y	0.766 [0.754, 0.776]	0.034 [0.031, 0.038]	0.081 [0.072, 0.089]	0.057 [0.051, 0.063]	0.142 [0.126, 0.156]
	5y	0.761 [0.749, 0.772]	0.109 [0.100, 0.122]	0.182 [0.170, 0.195]	0.129 [0.121, 0.138]	0.309 [0.287, 0.331]

Table S7 : Zero-shot forecasting with

\lambda=0.75

for the 144M model trained on full data.

Condition	Hor.	AUROC	AUPRC	F1	Precision	Recall
CHF	2y	0.856 [0.850, 0.862]	0.073 [0.069, 0.080]	0.139 [0.129, 0.148]	0.104 [0.097, 0.111]	0.207 [0.192, 0.222]
	5y	0.830 [0.822, 0.838]	0.186 [0.175, 0.199]	0.268 [0.255, 0.280]	0.202 [0.193, 0.212]	0.396 [0.376, 0.417]
COPD	2y	0.692 [0.684, 0.699]	0.052 [0.049, 0.054]	0.102 [0.096, 0.107]	0.071 [0.067, 0.075]	0.181 [0.170, 0.191]
	5y	0.676 [0.667, 0.684]	0.173 [0.166, 0.181]	0.240 [0.232, 0.248]	0.167 [0.162, 0.173]	0.423 [0.409, 0.438]
Dementia	2y	0.772 [0.761, 0.782]	0.036 [0.033, 0.041]	0.087 [0.079, 0.096]	0.059 [0.054, 0.065]	0.167 [0.151, 0.184]
	5y	0.770 [0.759, 0.781]	0.110 [0.101, 0.121]	0.179 [0.167, 0.191]	0.127 [0.118, 0.135]	0.305 [0.284, 0.326]

Table S8 : Zero-shot forecasting with

\lambda=1.0

(no reg.) for the 144M model trained on full data.

Model size	Horizon	AUROC	AUPRC	F1	Precision	Recall
0.69M	2y	0.732 [0.719, 0.745]	0.0274 [0.0260, 0.0294]	0.057 [0.052, 0.062]	0.037 [0.034, 0.040]	0.128 [0.116, 0.140]
	5y	0.706 [0.689, 0.722]	0.0927 [0.0879, 0.0992]	0.147 [0.136, 0.159]	0.101 [0.093, 0.111]	0.288 [0.270, 0.308]
2.81M	2y	0.764 [0.752, 0.775]	0.0345 [0.0325, 0.0371]	0.072 [0.065, 0.078]	0.048 [0.044, 0.052]	0.158 [0.143, 0.174]
	5y	0.748 [0.733, 0.763]	0.1151 [0.1090, 0.1226]	0.173 [0.164, 0.184]	0.120 [0.113, 0.127]	0.333 [0.312, 0.354]
5.81M	2y	0.755 [0.743, 0.768]	0.0345 [0.0325, 0.0374]	0.071 [0.065, 0.077]	0.049 [0.045, 0.053]	0.150 [0.137, 0.163]
	5y	0.725 [0.708, 0.741]	0.1094 [0.1035, 0.1168]	0.165 [0.155, 0.175]	0.117 [0.110, 0.124]	0.311 [0.285, 0.338]

Table S9 : Macro-averaged zero-shot forecasting across model sizes (full data,

\lambda=0.5

Data budget	Horizon	AUROC	AUPRC	F1	Precision	Recall
10%	2y	0.755 [0.741, 0.768]	0.0339 [0.0320, 0.0368]	0.069 [0.063, 0.076]	0.046 [0.042, 0.051]	0.154 [0.140, 0.171]
	5y	0.723 [0.708, 0.738]	0.1028 [0.0972, 0.1100]	0.159 [0.151, 0.167]	0.108 [0.102, 0.114]	0.314 [0.296, 0.332]
25%	2y	0.771 [0.758, 0.784]	0.0368 [0.0345, 0.0399]	0.076 [0.068, 0.084]	0.052 [0.048, 0.059]	0.157 [0.141, 0.173]
	5y	0.739 [0.722, 0.755]	0.1064 [0.1009, 0.1135]	0.170 [0.160, 0.179]	0.114 [0.108, 0.121]	0.372 [0.347, 0.400]
50%	2y	0.764 [0.750, 0.777]	0.0378 [0.0356, 0.0409]	0.077 [0.072, 0.084]	0.051 [0.047, 0.055]	0.183 [0.168, 0.199]
	5y	0.740 [0.722, 0.756]	0.1126 [0.1063, 0.1204]	0.171 [0.162, 0.179]	0.115 [0.110, 0.121]	0.338 [0.320, 0.356]

Table S10 : Macro-averaged zero-shot forecasting across data budgets (

\lambda=0.5

Condition	RAVEN	Multiclass	SeqLoss	EGE	BERT (FT)	MedGemma-27B
Dementia	0.789 [0.778, 0.799]	0.687 [0.642, 0.730]	0.704 [0.660, 0.745]	0.677 [0.665, 0.690]	0.731 [0.730, 0.733]	0.620 [0.592, 0.647]
Knee OA	0.726 [0.720, 0.732]	0.627 [0.601, 0.654]	0.672 [0.649, 0.697]	0.606 [0.599, 0.613]	0.744 [0.738, 0.749]	0.642 [0.625, 0.656]
COPD	0.691 [0.684, 0.698]	0.544 [0.514, 0.573]	0.629 [0.606, 0.655]	0.602 [0.594, 0.610]	0.704 [0.697, 0.711]	0.589 [0.572, 0.606]
CHF	0.857 [0.850, 0.864]	0.708 [0.674, 0.742]	0.733 [0.703, 0.766]	0.656 [0.646, 0.666]	0.862 [0.855, 0.868]	0.736 [0.716, 0.754]
Acute MI	0.793 [0.784, 0.801]	0.603 [0.570, 0.639]	0.579 [0.547, 0.614]	0.604 [0.594, 0.613]	0.818 [0.810, 0.826]	0.748 [0.728, 0.768]
Pancreatic Cancer	0.693 [0.649, 0.735]	0.496 [0.495, 0.496]	0.494 [0.493, 0.495]	0.502 [0.498, 0.509]	0.607 [0.561, 0.648]	0.620 [0.473, 0.746]

Table S11 : Zero-shot forecasting at 2-year horizon (AUROC, 95% confidence intervals).

Condition	RAVEN	Multiclass	SeqLoss	EGE	BERT (FT)	MedGemma-27B
Dementia	0.037 [0.034, 0.042]	0.026 [0.019, 0.041]	0.031 [0.020, 0.057]	0.026 [0.023, 0.030]	0.050 [0.050, 0.051]	0.011 [0.009, 0.015]
Knee OA	0.057 [0.054, 0.060]	0.039 [0.035, 0.046]	0.053 [0.046, 0.063]	0.038 [0.037, 0.040]	0.064 [0.061, 0.067]	0.039 [0.037, 0.043]
COPD	0.052 [0.049, 0.055]	0.036 [0.029, 0.048]	0.044 [0.037, 0.056]	0.040 [0.038, 0.043]	0.055 [0.053, 0.058]	0.030 [0.028, 0.033]
CHF	0.078 [0.072, 0.085]	0.047 [0.037, 0.066]	0.048 [0.038, 0.067]	0.045 [0.040, 0.051]	0.085 [0.079, 0.092]	0.036 [0.032, 0.043]
Acute MI	0.031 [0.029, 0.033]	0.015 [0.012, 0.021]	0.013 [0.011, 0.018]	0.017 [0.015, 0.019]	0.049 [0.044, 0.055]	0.026 [0.023, 0.031]
Pancreatic Cancer	0.002 [0.001, 0.003]	0.000 [0.000, 0.000]	0.000 [0.000, 0.000]	0.001 [0.001, 0.001]	0.001 [0.001, 0.001]	0.001 [0.000, 0.002]

Table S12 : Zero-shot forecasting at 2-year horizon (AUPRC, 95% confidence intervals).

Condition	RAVEN	Multiclass	SeqLoss	EGE	BERT (FT)	MedGemma-27B
Dementia	0.773 [0.762, 0.784]	0.669 [0.646, 0.691]	0.693 [0.670, 0.714]	0.670 [0.656, 0.684]	0.721 [0.720, 0.722]	0.609 [0.596, 0.621]
Knee OA	0.697 [0.690, 0.704]	0.635 [0.623, 0.648]	0.671 [0.658, 0.683]	0.620 [0.611, 0.629]	0.718 [0.711, 0.725]	0.617 [0.610, 0.625]
COPD	0.676 [0.668, 0.685]	0.531 [0.516, 0.545]	0.640 [0.626, 0.653]	0.615 [0.606, 0.624]	0.699 [0.690, 0.707]	0.586 [0.578, 0.595]
CHF	0.821 [0.812, 0.829]	0.718 [0.698, 0.737]	0.723 [0.705, 0.743]	0.666 [0.655, 0.677]	0.838 [0.830, 0.846]	0.718 [0.707, 0.728]
Acute MI	0.754 [0.742, 0.764]	0.613 [0.593, 0.633]	0.569 [0.550, 0.589]	0.623 [0.610, 0.635]	0.782 [0.772, 0.792]	0.699 [0.686, 0.710]
Pancreatic Cancer	0.639 [0.580, 0.699]	0.516 [0.491, 0.553]	0.513 [0.488, 0.551]	0.498 [0.498, 0.499]	0.642 [0.591, 0.695]	0.597 [0.539, 0.651]

Table S13 : Zero-shot forecasting at 5-year horizon (AUROC, 95% confidence intervals).

Condition	RAVEN	Multiclass	SeqLoss	EGE	BERT (FT)	MedGemma-27B
Dementia	0.108 [0.101, 0.118]	0.089 [0.075, 0.106]	0.102 [0.087, 0.121]	0.088 [0.080, 0.098]	0.146 [0.145, 0.147]	0.041 [0.039, 0.044]
Knee OA	0.200 [0.193, 0.207]	0.166 [0.157, 0.177]	0.190 [0.179, 0.202]	0.166 [0.160, 0.173]	0.221 [0.213, 0.229]	0.146 [0.142, 0.151]
COPD	0.164 [0.157, 0.171]	0.124 [0.114, 0.136]	0.169 [0.157, 0.184]	0.149 [0.143, 0.157]	0.189 [0.181, 0.197]	0.115 [0.111, 0.119]
CHF	0.184 [0.172, 0.198]	0.114 [0.103, 0.128]	0.134 [0.120, 0.153]	0.127 [0.117, 0.140]	0.220 [0.205, 0.236]	0.101 [0.095, 0.110]
Acute MI	0.080 [0.075, 0.086]	0.060 [0.052, 0.071]	0.047 [0.043, 0.054]	0.059 [0.054, 0.065]	0.106 [0.098, 0.117]	0.067 [0.062, 0.073]
Pancreatic Cancer	0.004 [0.003, 0.014]	0.002 [0.002, 0.005]	0.004 [0.002, 0.019]	0.001 [0.001, 0.001]	0.003 [0.002, 0.005]	0.002 [0.002, 0.003]

Table S14 : Zero-shot forecasting at 5-year horizon (AUPRC, 95% confidence intervals).

Explore similar work

Sequence Composition in EHR Foundation Model Pretraining
Oct 12, 2026Joshua Placidi, Inigo Stibbe, Alfred J. Balston +2Electronic Health Records Longitudinal EHR Modeling
Department of Computing, Imperial College London
Pretraining EHR Foundation Models with Patient-Aware Sampling
Jul 24, 2026Joshua Placidi, Yuxuan Liu, Jinpei Han +2Longitudinal EHR Modeling Electronic Health Records
Dept. of Computing, Imperial College London, London, United Kingdom · Dept. of Bioengineering, Imperial College London, London, United Kingdom · UKRI Centres in AI for Health, United Kingdom +1
EHR-RAGp: Prototype-Guided Retrieval of Longitudinal Electronic Health Records for Clinical Prediction Models
May 12, 2026Saeed Shurrab, Mariam Al-Omari, Dana El Samad +1Long-Context Modeling Clinical Prediction
New York University · New York University Abu Dhabi

Oct 12, 2026J/K move · Enter open · S save

Sequence Composition in EHR Foundation Model Pretraining

Joshua Placidi, Inigo Stibbe, Alfred J. Balston, Marek Rei+1

Department of Computing, Imperial College London

Electronic Health Records Longitudinal EHR Modeling Self-Supervised Pre-Training Clinical Outcome Prediction

Open paper AskPDF arXiv

Autoregressive electronic health record (EHR) foundation models predict the next event token in patient trajectories: sequences of structured medical events. A typical pretraining strategy is to concatenate all trajectories into a global stream and sample token windows from it. This gives patients with longer sequences more training exposure and allows tokens to attend across patient boundaries. We argue that these choices create a misalignment between pretraining and downstream clinical tasks. To explore this, we investigate how the composition of pretraining sequences affects clinical prediction performance. Using the global-stream approach as a baseline, we propose alternative sequence composition methods that control how patient trajectories contribute to pretraining sequences. Across four MIMIC-IV v2.2 tasks, our best method improves macro AUROC, AUPRC, F1, and balanced accuracy, with relative gains of 9.3% in F1 and 2.7% in AUPRC under a fixed training recipe. We find that these gains generalize across four additional evaluation settings, including the INSPIRE and eICU-CRD datasets, with improvements in macro AUROC, AUPRC, and F1 score. Our results show that simply changing how EHR data are presented to the model during pretraining can improve downstream performance, without changing the model or collecting additional data.

Abstract

Explore similar work

Sequence Composition in EHR Foundation Model Pretraining

Pretraining EHR Foundation Models with Patient-Aware Sampling

EHR-RAGp: Prototype-Guided Retrieval of Longitudinal Electronic Health Records for Clinical Prediction Models