cs.LGNov 5, 2026

How Deep is the Signal? Time-Context up to 30 Years for Colorectal Cancer Prediction in a Primary-Care EHR Model

Authors: Achille Raviez, Lan Wang, Brendan Delaney

Organizations: I-X, Imperial College London, United Kingdom · Sorbonne Université, AP-HP, France

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

EHR foundation models read medical events within a fixed context window, and long context has so far been measured only in tokens. How far back in time a model must look to capture the earliest predictive signal is unknown. We introduce time-context, the calendar lookback window before the prediction point, and provide, to our knowledge, the first systematic evaluation of decade-scale time-context for an EHR foundation model. We pretrained a decoder-only transformer (34.6M parameters) on 1.4B coded events from 3.6M primary-care patients in North West London and fine-tuned it to predict incident colorectal cancer under a 12-month pre-diagnostic buffer. A frozen cohort of 1,008 cases and 9,582 matched controls, all registered for at least 30 years, was evaluated at 15 nested lookback windows from 2 to 30 years, so that every truncation is compared within the same patients. Discrimination rose from an AUROC of 0.551 at 2 years to 0.590 at 18 years, then plateaued, with about 80% of the gain reached within 10 years; an XGBoost baseline converged to the same level (peak AUROC 0.589). A simulation further suggests that rotary position embeddings (RoPE) over elapsed time alias beyond about 20 years at the default base frequency, motivating a retuned θθ. Time-context is a measurable design axis for clinical prediction models, distinct from token-context, and should be reported.

Figures & tables

Figure 1
Figure 2
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3
Figure 4
Figure 5
Figure 6
CohortCasesControlsRatioInput
Pretraining3.6M patients–Full
Fine-tuning13,49037,3211:2.77Full
Validation2842,7781:9.782–22y (6)
Evaluation1,0089,5821:9.512–30y (15)
Appendix
Table 7
Window (y)TransformerXGBoost
20.5505 (0.5320–0.5691)0.5281 (0.5101–0.5481)
40.5625 (0.5433–0.5812)0.5445 (0.5266–0.5650)
60.5730 (0.5530–0.5922)0.5553 (0.5359–0.5731)
80.5756 (0.5555–0.5950)0.5625 (0.5437–0.5809)
100.5825 (0.5637–0.6003)0.5718 (0.5533–0.5902)
120.5840 (0.5643–0.6020)0.5795 (0.5596–0.5981)
Appendix
Table 8
Window (y)TransformerXGBoost
20.1128 (0.1026–0.1245)0.1082 (0.0982–0.1200)
40.1169 (0.1062–0.1289)0.1133 (0.1022–0.1261)
60.1206 (0.1098–0.1323)0.1162 (0.1060–0.1289)
80.1239 (0.1118–0.1367)0.1176 (0.1074–0.1308)
100.1259 (0.1141–0.1385)0.1193 (0.1093–0.1322)
120.1265 (0.1144–0.1393)0.1225 (0.1120–0.1346)
Appendix
Table 9
More in this topic

Longitudinal EHR Modeling

Longitudinal electronic health record modeling learns from patient histories over time to forecast outcomes, characterize disease trajectories, and guide care.

See all 169 papers

Explore similar work

CardsList
Oct 29, 2026

TRACER, a time-to-event EHR foundation model for disease detection across health systems

Foundation models trained on longitudinal electronic health records (EHRs) can predict clinical diagnoses years in the future, making them a potential tool for disease screening. We developed TRACER (Temporal Risk Assessment with Clinical EHR Representations), a decoder-based foundation model pretrained on data from over 4.5 million patients from four health systems, and evaluated it on 16 incident diseases at 1-, 2-, and 5-year horizons in more than 2 million patients from two health systems excluded from model development. In contrast to typical evaluations that focus on global metrics (e.g., AUROC), we measured performance at the low false-positive rates that are realistic for screening. We found that TRACER retained its discrimination in external health systems, and at 95% specificity, the patients it flagged were up to eight times more likely to be diagnosed within a year than unselected patients. Evaluating at this operating point also revealed what global metrics alone did not: detection and false-alert burden varied across diseases and health systems, even when discrimination appeared stable. Our results illustrate the potential of foundation models for screening and underscore the need for evaluation in health systems excluded from development to assess disease detection and false-alert burden at levels appropriate for intended clinical use.}

Mara Pleasure, Ekaterina Redekop, Vedrana Ivezic +4

Biomedical AI Research Lab, University of California, Los Angeles, Los Angeles, CA, USA · Department of Radiology, University of California, Los Angeles, Los Angeles, CA, USA · Bioinformatics, University of California, Los Angeles, Los Angeles, CA, USA +4

EHR Foundation ModelsLongitudinal EHR ModelingClinical Risk PredictionMedical DiagnosisEvent Time Prediction
AskPDF
May 12, 2026

EHR-RAGp: Prototype-Guided Retrieval of Longitudinal Electronic Health Records for Clinical Prediction Models

Electronic Health Records (EHR) contain rich longitudinal patient information and are widely used in predictive modeling applications. However, effectively leveraging historical data remains challenging due to long trajectories, heterogeneous events, temporal irregularity, and the varying relevance of past clinical context. Existing approaches often rely on fixed windows or uniform aggregation, which can obscure clinically important signals. In this work, we introduce EHR-RAGp, a retrieval-based framework that dynamically integrates the most relevant patient history consisting of diverse clinical event types. We propose a prototype-guided retrieval module that acts as an alignment mechanism and estimates the relevance of retrieved historical chunks with respect to a given prediction task, guiding the model towards the most informative context. Across multiple clinical prediction tasks and two benchmark datasets, EHR-RAGp consistently outperforms state-of- the-art EHR-based and transformer-based baselines. Furthermore, EHR-RAGp is model-agnostic, as integrating it with various backbone models yields substantial performance gains. Overall, EHR-RAGp establishes a novel direction for modeling long-range clinical context to improve downstream performance via retrieval.

Saeed Shurrab, Mariam Al-Omari, Dana El Samad +1

New York University · New York University Abu Dhabi

Long-Context ModelingClinical PredictionLongitudinal EHR Modeling
AskPDF
May 14, 2026

DT-Transformer: A Foundation Model for Disease Trajectory Prediction on a Real-world Health System

Accurate disease trajectory prediction is critical for early intervention, resource allocation, and improving long-term outcomes. While electronic health records (EHRs) provide a rich longitudinal view of patient health in clinical environments, models trained on curated research cohorts may not reflect routine deployment settings, and those trained on single-hospital datasets capture only fragments of each patient's trajectory. This highlights the importance of leveraging large, multi-hospital health systems for training and validation to better reflect real-world clinical complexity. In this work, we develop DT-Transformer, a foundation model trained on 57.1M structured EHR entries over 1.7M patients from Mass General Brigham (MGB), spanning 11 hospitals and a broad network of outpatient clinics. DT-Transformer achieves strong discrimination in both held-out and prospective validation settings. Next-event prediction achieves a median age- and sex-stratified AUC of 0.871 across 896 disease categories, with all categories exceeding AUC 0.5. These results support health system-scale training as a path toward foundation models suited to real-world clinical forecasting.

Yunying Zhu, Andrew R Weckstein, Kueiyu Joshua Lin +1

Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, USA · Harvard T.H. Chan School of Public Health, Harvard University, Boston, MA, USA · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, MA, USA +2

Clinical PredictionTime Series TransformersTime-Series Foundation ModelsHealthcareLongitudinal EHR ModelingEHR Foundation Models+1
AskPDF