Sequence Composition in EHR Foundation Model Pretraining

Joshua Placidi; Inigo Stibbe; Alfred J. Balston; Marek Rei; A. Aldo Faisal

cs.LGOct 12, 2026

Sequence Composition in EHR Foundation Model Pretraining

Authors: Joshua Placidi, Inigo Stibbe, Alfred J. Balston, Marek Rei, A. Aldo Faisal

Organizations: Department of Computing, Imperial College London

Abstract

Autoregressive electronic health record (EHR) foundation models predict the next event token in patient trajectories: sequences of structured medical events. A typical pretraining strategy is to concatenate all trajectories into a global stream and sample token windows from it. This gives patients with longer sequences more training exposure and allows tokens to attend across patient boundaries. We argue that these choices create a misalignment between pretraining and downstream clinical tasks. To explore this, we investigate how the composition of pretraining sequences affects clinical prediction performance. Using the global-stream approach as a baseline, we propose alternative sequence composition methods that control how patient trajectories contribute to pretraining sequences. Across four MIMIC-IV v2.2 tasks, our best method improves macro AUROC, AUPRC, F1, and balanced accuracy, with relative gains of 9.3% in F1 and 2.7% in AUPRC under a fixed training recipe. We find that these gains generalize across four additional evaluation settings, including the INSPIRE and eICU-CRD datasets, with improvements in macro AUROC, AUPRC, and F1 score. Our results show that simply changing how EHR data are presented to the model during pretraining can improve downstream performance, without changing the model or collecting additional data.

Figures & tables

Figure 1: Autoregressive prediction from a patient trajectory. Medical events are encoded as discrete tokens, and the EHR-FM predicts the next token from the preceding clinical context.

Panel of Figure 2 — Figure 2: Trajectory token length and clinical content in the training set of MIMIC-IV v2.2. (a) Distribution of patient trajectory lengths and tokens. Each token is represented by the length of its corresponding trajectory. (b) Distributions of hospital mortality and ICU readmission task instances by trajectory length. (c) Recorded outcome prevalence by trajectory length. (d) Token composition by trajectory length.

Figure 3: Sequence composition for four patients with unequal trajectory lengths. In the Global Stream example, each token appears once, so longer trajectories contribute more tokens. Ordered Global Stream ( $\alpha=0$ ) samples windows repeatedly to give patients approximately equal expected contributions.

Dataset	Tokenization	Patients	Tokens	Vocab. size	Median length (tokens)	Length $>2{,}048$ (% of patients)
MIMIC-IV v2.2	ETHOS	285,606	362,697,213	4,509	338	14.2%
MIMIC-IV v3.1	ETHOS	346,262	500,700,081	4,598	365	16.1%
MIMIC-IV v2.2	Ours	287,511	576,657,770	6,431	512	22.2%
INSPIRE	Ours	99,900	176,560,774	1,313	840	18.3%
eICU-CRD	Ours	138,963	461,065,895	8,000	1,719	43.1%

Table 1: Dataset and tokenization statistics. Token counts and trajectory lengths are calculated after tokenization. The final column gives the percentage of patients whose complete trajectory exceeds 2,048 tokens, which is the size of the context window,

L

, we use in our experiments.

Figure 4: Macro downstream performance across four clinical prediction tasks on MIMIC-IV v2.2 test set, reported as mean

\pm

standard deviation over five seeds.

Figure 5: Cross-patient attention following a patient boundary in GS. Cross-patient attention is the fraction of attention assigned to tokens belonging to preceding patients. The line shows the mean across examples, with shading indicating

\pm 1

standard deviation.

Figure 6: Local patient-weighting sweeps on MIMIC-IV v2.2. Panels show next-token validation loss and macro validation performance across four clinical tasks as

\alpha

varies. Dotted lines indicate the GS+M baseline.

Figure 7: Generalization across datasets and tokenizations. Points show macro test-set performance across the indicated tasks, averaged over five seeds for MIMIC-IV v2.2 with ETHOS tokenization and reported for one seed in each additional setting. Annotations report relative changes from GS+M to OGS. Markers beside dataset labels identify tokenizers. Horizontal scales differ between metrics.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Expected training-target exposure by patient trajectory length under Patient Sampling. (a) Distributions of patients and tokens. (b–d) Expected target shares for

\alpha=0.2

0.5

, and

1

. Each distribution sums to 100% across length bins.

Figure 9: Expected training-data coverage on MIMIC-IV v2.2 as a function of

\alpha

. Let

u_{k}

be the probability that corpus token position

k

appears in one sampled window. After

B

independent draws, its probability of being encountered at least once is

1-(1-u_{k})^{B}

; coverage averages this quantity over all corpus positions, regardless of loss eligibility. We use

B\approx 141{,}593

windows with context length 2,048, giving a nominal token budget equal to the training-corpus size. The dotted line shows GS, which is independent of

\alpha

Parameter	Value
Architecture	GPT-2-style decoder ( Radford et al., 2019 )
Parameters	49,454,592
Transformer layers	6
Hidden dimension	768
Attention heads	12
Feed-forward dimension	3,072

Appendix

Table 2: Model architecture and pretraining configuration. The parameter count is for MIMIC-IV v2.2 with ETHOS tokenization; it varies with vocabulary size in the other settings.

Panel of Figure 10 — Figure 10: Initial patient-weighting sweep on MIMIC-IV v2.2 with ETHOS tokenization (seed 42). Each column shows one clinical prediction task; rows show F1, AUROC, AUPRC, and balanced accuracy. Curves compare PS, RPS, and OGS across values of the sampling exponent $\alpha$ . Dotted lines show the GS+M baseline, which does not depend on $\alpha$ .

Method	$\alpha$	Best validation loss $\downarrow$	Macro AUROC $\uparrow$	Macro AUPRC $\uparrow$	Macro F1 $\uparrow$	Macro BAcc $\uparrow$
PS	$0.0$	$0.9310$	$0.8095$	$0.4238$	$0.3459$	$0.6215$
	$0.2$	$0.9048$	$0.8307$	$0.4552$	$0.4050$	$0.6550$
	$0.4$	$0.8826$	$0.8463$	$0.4811$	$0.4328$	$0.6677$
	$0.6$	$0.8673$	$\mathbf{0.8471}$	$\mathbf{0.4957}$	$\mathbf{0.4553}$	$\mathbf{0.6769}$
	$0.8$	$0.8588$	$0.8399$	$0.4896$	$0.3992$	$0.6422$
	$1.0$	$\mathbf{0.8556}$	$0.8343$	$0.4895$	$0.4046$	$0.6451$

Appendix

Table 3: Initial single-seed patient-weighting sweep on MIMIC-IV v2.2 (seed 42). Downstream macro metrics average four clinical tasks. Best validation loss is the lowest loss reached during each run. Bold indicates the best value within each method for each metric.

Panel of Figure 11 — Figure 11: Local patient-weighting sweep on MIMIC-IV v2.2 with ETHOS tokenization (seed 42), following the initial coarse sweep. Each column shows one clinical prediction task; rows show F1, AUROC, AUPRC, and balanced accuracy. Curves compare PS, RPS, and OGS across local values of the sampling exponent $\alpha$ . Dotted lines show the GS+M baseline, which does not depend on $\alpha$ .

Method	$\alpha$	Best validation loss $\downarrow$	Macro AUROC $\uparrow$	Macro AUPRC $\uparrow$	Macro F1 $\uparrow$	Macro BAcc $\uparrow$
PS	$0.50$	$0.8733$	$\mathbf{0.8543}$	$0.4874$	$\mathbf{0.4599}$	$\mathbf{0.6926}$
	$0.55$	$0.8703$	$0.8466$	$0.4945$	$0.4456$	$0.6648$
	$0.60$	$0.8673$	$0.8471$	$0.4957$	$0.4553$	$0.6769$
	$0.65$	$0.8641$	$0.8397$	$0.4893$	$0.3860$	$0.6338$
	$0.70$	$\mathbf{0.8623}$	$0.8512$	$\mathbf{0.4987}$	$0.4436$	$0.6714$
RPS	$0.70$	$0.8641$	$0.8467$	$0.4976$	$0.4332$	$0.6584$

Appendix

Table 4: Local patient-weighting search on MIMIC-IV v2.2 following the initial coarse sweep (seed 42). Downstream macro metrics average four clinical tasks. Best validation loss is the lowest loss reached during each run. Bold indicates the best value within each method for each metric.

Method	$\alpha$	Macro F1	Macro AUROC	Macro AUPRC	Macro BAcc
GS	–	$0.420\pm 0.017$	$0.844\pm 0.004$	$0.486\pm 0.010$	$0.655\pm 0.010$
GS+M	–	$0.430\pm 0.015$	$0.845\pm 0.004$	$\mathit{0.492\pm 0.004}$	$0.660\pm 0.009$
PS	$0.70$	$0.419\pm 0.032$	$0.839\pm 0.012$	$0.481\pm 0.018$	$0.654\pm 0.018$
RPS	$0.85$	$\mathit{0.439\pm 0.014}$	$\mathit{0.847\pm 0.005}$	$0.492\pm 0.007$	$\mathit{0.665\pm 0.010}$
OGS	$0.80$	$\mathbf{0.459\pm 0.017}$	$\mathbf{0.849\pm 0.008}$	$\mathbf{0.499\pm 0.007}$	$\mathbf{0.679\pm 0.014}$

Appendix

Table 5: Macro downstream performance across four clinical prediction tasks on MIMIC-IV v2.2, reported as mean

\pm

standard deviation over five seeds. GS denotes Global Stream, GS+M Global Stream with masking, PS Patient Sampling, RPS Recursive Patient Sampling, and OGS Ordered Global Stream. Best mean results are shown in bold and second-best results in italics. Higher values are better.

MIMIC-IV table	ETHOS	Unified
Demographics and encounters
patients	✓	✓
admissions	✓	✓
transfers	✓	✓
triage	✓	✓
Diagnoses and billing procedures

Appendix

Table 6: Selected MIMIC-IV source tables included in the ETHOS and unified tokenization configurations. The unified column reflects the repository’s current default MIMIC-IV configuration.

Benchmark	Prediction point	Positive / negative outcome	Horizon
Hospital mortality	HOSPITAL_ADMISSION	MEDS_DEATH (+) HOSPITAL_DISCHARGE ( $-$ )	–
ICU mortality	ICU_ADMISSION	MEDS_DEATH (+) ICU_DISCHARGE ( $-$ )	–
ICU readmission	ICU_DISCHARGE	ICU_ADMISSION (+) HOSPITAL_DISCHARGE ( $-$ )	–
ICU admission	HOSPITAL_ADMISSION	ICU_ADMISSION (+) HOSPITAL_DISCHARGE ( $-$ )	–
Hospital readmission	HOSPITAL_DISCHARGE	HOSPITAL_ADMISSION (+) MEDS_DEATH or horizon ( $-$ )	30 days
Postoperative ICU admission	OPERATION_END	ICU_ADMISSION (+) HOSPITAL_DISCHARGE ( $-$ )	–

Appendix

Table 7: Prediction points and outcomes for the rollout benchmarks. The positive outcome must precede the competing negative outcome and, where specified, occur within the time horizon. For horizon-based tasks, TIMELINE_END also stops generated rollouts; observed instances ending before the horizon without a definitive outcome are excluded.

Dataset	Tokenization	Maximum tokens
MIMIC-IV v2.2	ETHOS	4,096
MIMIC-IV v2.2	Unified	8,192
MIMIC-IV v3.1	ETHOS	4,096
INSPIRE	Unified	4,096
eICU-CRD	Unified	8,192

Appendix

Table 8: Maximum generated tokens per rollout. Methods compared within each dataset and tokenization setting use the same limit.

Task	Instances	Positive	Prevalence
MIMIC-IV v2.2, ETHOS tokenization
Hospital mortality	42,973	836	1.95%
ICU admission	42,974	6,636	15.44%
ICU mortality	7,365	532	7.22%
ICU readmission	6,469	850	13.14%
MIMIC-IV v2.2, unified tokenization

Appendix

Table 9: Test-set prediction instances for the indicated dataset and tokenization settings. A patient may contribute multiple instances. Prevalence is the percentage of instances with a positive label.

	AUROC		AUPRC		F1		BAcc
Task	GS+M	OGS	GS+M	OGS	GS+M	OGS	GS+M	OGS
MIMIC-IV v2.2 ETHOS tokenization
Hospital mortality	$0.836$	$\mathbf{0.868}$	$0.310$	$\mathbf{0.329}$	$0.265$	$\mathbf{0.353}$	$0.585$	$\mathbf{0.634}$
ICU admission	$0.911$	$\mathbf{0.913}$	$0.769$	$\mathbf{0.770}$	$\mathbf{0.690}$	$\mathbf{0.690}$	$\mathbf{0.784}$	$\mathbf{0.784}$
ICU mortality	$0.886$	$\mathbf{0.888}$	$0.462$	$\mathbf{0.514}$	$0.431$	$\mathbf{0.497}$	$0.662$	$\mathbf{0.723}$
ICU readmission	$0.735$	$\mathbf{0.748}$	$\mathbf{0.420}$	$0.416$	$\mathbf{0.312}$	$0.311$	$\mathbf{0.593}$	$\mathbf{0.593}$

Appendix

Table 10: Task-level generalization results across datasets and tokenizations for seed 42. GS+M and OGS are compared on each task; BAcc denotes balanced accuracy. Dataset macro scores are unweighted means over the tasks displayed within each dataset. The overall mean weights the five dataset settings equally, and OGS gain is the difference in percentage points, calculated from full-precision macro scores. Bold indicates the better displayed result in each pair; values tied after rounding to three decimal places are both bolded. Task definitions may differ across datasets, so absolute scores should not be compared directly between them.

Explore similar work

Jul 24, 2026

Pretraining EHR Foundation Models with Patient-Aware Sampling

Autoregressive foundation models for electronic health records (EHRs) typically inherit pretraining methods from language modeling, where patient trajectories are concatenated into a single token stream and windows are sampled from that stream. In EHR data, this choice is consequential: windows may mix multiple patients, and patients with longer records contribute more optimization updates, potentially introducing bias. We propose Patient Sampling, a pretraining sequence-construction method that allows us to control how training signal is distributed across patients. We compare this method to the standard approach, which we refer to as Global Stream. We show that stochastic Patient Sampling with controllable weighting improves performance on real-world EHR data. Across downstream clinical tasks on MIMIC-IV v2.2 and v3.1, Patient Sampling improves Macro AUROC and AUPRC over the Global Stream baseline. These results identify training and validation sequence construction as important and underexplored design choices for autoregressive EHR foundation models.

Joshua Placidi, Yuxuan Liu, Jinpei Han +2

Dept. of Computing, Imperial College London, London, United Kingdom · Dept. of Bioengineering, Imperial College London, London, United Kingdom · UKRI Centres in AI for Health, United Kingdom +1

Longitudinal EHR Modeling Electronic Health Records Autoregressive Modeling

Ask PDF

Mar 25, 2026

Scaling Recurrence-aware Foundation Models for Clinical Records via Next-Visit Prediction

While large-scale pretraining has revolutionized language modeling, its potential remains underexplored in healthcare with structured electronic health records (EHRs). We present RAVEN, a novel generative pretraining strategy for sequential EHR data based on Recurrence-Aware next-Visit EveNt prediction. Leveraging a dataset of over one million unique individuals, our model learns to autoregressively generate tokenized clinical events for the next visit conditioned on patient history. We introduce regularization on predicting repeated events and highlight a key pitfall in EHR-based foundation model evaluations: repeated event tokens can inflate performance metrics when new onsets are not distinguished from subsequent occurrences. Furthermore, we empirically investigate the scaling behaviors in a data-constrained, compute-saturated regime, showing that simply increasing model size is suboptimal without commensurate increases in data volume. We evaluate our model via zero-shot prediction for forecasting the incidence of a diverse set of diseases, where it rivals fully fine-tuned representation-based Transformer models and outperforms both standard simulation-based next-token approaches and a prompted medical large language model baseline. Finally, without additional parameter updates, we show that RAVEN can generalize to an external patient cohort under lossy clinical code mappings and feature coverage gaps.

Haresh Rengaraj Rajamohan, Xiang Gao, Weicheng Zhu +10

Center for Data Science, New York University, New York, NY, USA · Department of Radiology, NYU Grossman School of Medicine, New York, NY, USA · Center for Biomedical Imaging, NYU Grossman School of Medicine, New York, NY, USA +1

Longitudinal EHR Modeling Electronic Health Records Scaling Laws Clinical Outcome Prediction

Ask PDF

Mar 9, 2026

EveryQuery: A Promptable Foundation Model for Clinical Prediction Tasks over Electronic Health Records

Autoregressive foundation models for electronic health records can make predictions for diverse clinical tasks without finetuning. These methods work by generating possible "synthetic futures" for patients, and then inferring downstream predictions over these simulated trajectories. Despite their strong capabilities, these methods have several flaws. (1) Inference is extremely computationally expensive, as each prediction requires simulating many trajectories, each with many observations. (2) Their performance is noisy due to the variance inherent to simulation, which in particular reduces efficacy on rare events, which are often of high importance clinically. (3) They are not promptable, as their only input is the patient's medical history, requiring indirect aggregation at inference time. We introduce EveryQuery, a promptable foundation model for structured EHR data. Building on the insight that clinical prediction tasks can be written in a structured query language, EveryQuery takes a query as a prompt alongside the patient's history and estimates the outcome directly. This allows EveryQuery to make predictions across diverse clinical tasks, such as classification and survival modeling, without simulation or finetuning. Across three EHR datasets, EveryQuery outperforms a competitive autoregressive baseline on clinically relevant classification and time-to-event tasks, with mean AUROC gains of 14.3% and 12.7% on MIMIC-IV and a large academic medical center and comparable performance on the smaller NWICU dataset. Its advantage is largest for rare outcomes, and it requires 580x less inference compute.

Payal Chandak, Gregory Kondas, Liat Antwarg Friedman +2

Harvard-MIT HST · Columbia University · Harvard Medical School

Clinical Outcome Prediction Clinical Prediction Longitudinal EHR Modeling Electronic Health Records

Ask PDF