Temporal information in structured electronic health records (EHRs) is often lost in sparse one-hot or count-based representations, while sequence models can be costly and data-hungry. We propose PaReGTA, an LLM-based encoding framework that (i) converts longitudinal EHR events into visit-level templated text with explicit temporal cues, (ii) learns domain-adapted visit embeddings via lightweight contrastive fine-tuning of a sentence-embedding model, and (iii) aggregates visit embeddings into a fixed-dimensional patient representation using hybrid temporal pooling that captures both recency and globally informative visits. The resulting fixed-dimensional patient representations can be used with conventional downstream machine-learning models. To examine factor-level sensitivity, we use PaReGTA-RSS (Representation Shift Score), a prespecified factor-removal analysis that recomputes patient representations after removing clinically defined factor groups and quantifies the resulting change in the fitted logit of a fixed logistic-regression model. We evaluated PaReGTA in a cohort of 39,088 patients with migraine from the All of Us Research Program (AoU) on a retrospective patient-level classification task with an EHR-derived target. In an exploratory comparison on the fixed held-out test cohort of 7,818 patients, PaReGTA-Gap + LightGBM, used as a post hoc analytical reference, had the highest reported AUC, accuracy, and F1 values among the evaluated sparse, BERT-based EHR, and recurrent approaches in this cohort.
Figures & tables
Figure 1: Overview of PaReGTA.
Figure 2: PaReGTA-RSS factor-removal sensitivity for (A) medication classes and (B) comorbidities in the full cohort. The panels use independent horizontal scales.
Figure 3: PaReGTA-RSS factor-removal sensitivity for (A) medication classes and (B) comorbidities among participants recorded as male at birth. The panels use independent horizontal scales.
Figure 4: PaReGTA-RSS factor-removal sensitivity among participants recorded as female at birth: (A) medication classes, (B) comorbidities in the full female subgroup, and (C) comorbidities in the subgroup without temporomandibular disorders or fibromyalgia. The panels use independent horizontal scales.
Figure 5: Temporal PaReGTA-RSS sensitivity after removing retained medication and comorbidity records within nested 30-, 90-, 180-, and 365-day recency windows in a fixed common cohort: (A) the target-negative group ( n=701 ) and (B) the target-positive group ( n=266 ). Values represent mean absolute fitted-logit changes, and the panels use independent horizontal scales.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Category name
ICD
Code
Migraine (all)
ICD-9-CM
346
Migraine (all)
ICD-10-CM
G43
Chronic migraine without aura
ICD-9-CM
346.7
Chronic migraine without aura
ICD-9-CM
346.70
Chronic migraine without aura
ICD-9-CM
346.71
Chronic migraine without aura
ICD-9-CM
346.72
Appendix
Table 1: Prespecified ICD codes used for the operational migraine categories.
Category name
ICD
Code
Depression
ICD-10-CM
F32.A
Depression
ICD-10-CM
Z13.31
Depression
ICD-10-CM
Z13.32
Depression
ICD-10-CM
F53.0
Depression
ICD-9-CM
290.13
Depression
ICD-9-CM
V79.0
Appendix
Table 2: Prespecified ICD codes used for the operational comorbidity categories.
Feature engineering for Electronic Health Records (EHR) is complicated by irregular observation intervals, variable measurement frequencies, and structural sparsity inherent to clinical time series. Existing automated methods either lack clinical domain awareness or assume clean, regularly sampled inputs, limiting their applicability to real-world EHR data. We present FeatEHR-LLM, a framework that leverages Large Language Models (LLMs) to generate clinically meaningful tabular features from irregularly sampled EHR time series. To limit patient privacy exposure, the LLM operates exclusively on dataset schemas and task descriptions rather than raw patient records. A tool-augmented generation mechanism equips the LLM with specialized routines for querying irregular temporal data, enabling it to produce executable feature-extraction code that explicitly handles uneven observation patterns and informative sparsity. FeatEHR-LLM supports both univariate and multivariate feature generation through an iterative, validation-in-the-loop pipeline. Evaluated on eight clinical prediction tasks across four ICU datasets, our framework achieves the highest mean AUROC on 7 out of 8 tasks, with improvements of up to 6 percentage points over strong baselines. Code is available at github.com/hojjatkarami/FeatEHR-LLM.
Hojjat Karami, David Atienza, Jean-Philippe Thiran +1
1École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
Large Language Models (LLMs) have shown strong promise for mining Electronic Health Records (EHRs) by reasoning over longitudinal clinical information to capture context-rich patient trajectories. However, leveraging LLMs for structured EHRs (e.g., standardized diagnosis and medication codes) presents two key challenges. First, translating time-stamped EHR sequences into plain text can obscure both temporal structure and code identities, weakening the ability to capture code co-occurrence and longitudinal regularities. Second, unlike cohort-trained predictive models that learn a shared, task-aligned representation space across patients, LLMs are often applied in a case-isolated inference setting where each patient is processed independently without leveraging population-level patterns. To address these challenges, we introduce RePrompT, a time-aware LLM framework that integrates structured EHR encoders through prompt tuning, without modifying underlying architectures. Specifically, RePrompT recurrently incorporates latent states from prior visits to preserve longitudinal information, and injects population-level information through trainable prompt tokens derived from a cohort-trained, task-aligned EHR encoder. Experiments on MIMIC-III and MIMIC-IV demonstrate that RePrompT consistently outperforms both EHR-based and LLM-based baselines across multiple clinical prediction tasks.
Electronic Health Records (EHR) contain rich longitudinal patient information and are widely used in predictive modeling applications. However, effectively leveraging historical data remains challenging due to long trajectories, heterogeneous events, temporal irregularity, and the varying relevance of past clinical context. Existing approaches often rely on fixed windows or uniform aggregation, which can obscure clinically important signals. In this work, we introduce EHR-RAGp, a retrieval-based framework that dynamically integrates the most relevant patient history consisting of diverse clinical event types. We propose a prototype-guided retrieval module that acts as an alignment mechanism and estimates the relevance of retrieved historical chunks with respect to a given prediction task, guiding the model towards the most informative context. Across multiple clinical prediction tasks and two benchmark datasets, EHR-RAGp consistently outperforms state-of- the-art EHR-based and transformer-based baselines. Furthermore, EHR-RAGp is model-agnostic, as integrating it with various backbone models yields substantial performance gains. Overall, EHR-RAGp establishes a novel direction for modeling long-range clinical context to improve downstream performance via retrieval.
Saeed Shurrab, Mariam Al-Omari, Dana El Samad +1
New York University · New York University Abu Dhabi