EHRAdapt: Adapting Pretrained Language Models to Electronic Health Records with Semantic Priors for Rare Clinical Events

Andre R Goncalves; Vincent Liu; Priyadip Ray

cs.AISep 27, 2026

EHRAdapt: Adapting Pretrained Language Models to Electronic Health Records with Semantic Priors for Rare Clinical Events

Authors: Andre R Goncalves, Vincent Liu, Priyadip Ray

Organizations: Lawrence Livermore National Laboratory · Kaiser Permanente

Abstract

Electronic health records (EHRs) encode clinical histories as (time, modality, code) tuples, whereas pretrained language models expect text tokens. Serializing them as text inflates sequence length and redundantly encodes structure. We introduce EHRAdapt, an adapter that maps tuples directly into a frozen language model's embedding space. Modality receives a learned embedding, time gaps enter through learned attention biases, and event codes receive dedicated vectors. Learning event vectors is the central challenge: clinical vocabularies are long-tailed, leaving rare events too few observations for reliable estimates. EHRAdapt therefore represents each event vector as the sum of a semantic prior and an evidence residual. The prior is a frozen embedding of the event's clinical description from a biomedical language model trained on clinical ontologies, mapped into the model's input space by a shared learned projection, so it supplies clinical meaning even when observations are scarce. The residual, a learned low-rank event-specific correction, refines it as evidence accumulates. We run continued pretraining on about 4 million patients' records with three frozen LLM backbones (OLMo2 1B, Llama3.2 1B, and OLMo2 7B), training only the adapter (0.1--0.6% of all parameters). The full adapter outperforms all ablations in held-out next-event prediction on every backbone. Removing the semantic pathway hurts rare events over ten times more than the most frequent ones, whereas removing the residual hurts overall prediction but improves it for the rarest events. On reportable infectious-disease and syndromic downstream classification tasks, EHRAdapt outperforms text-based LLM and count-based baselines, and both pathways improve rare-disease discrimination. The two pathways therefore play complementary roles, visible only when results are broken down by event frequency rather than averaged.

Figures & tables

Configuration	OLMo2-1B	Llama3.2-1B	OLMo2-7B
Full model	$\mathbf{2.847}\pm 0.006$	$\mathbf{2.806}\pm 0.011$	$\mathbf{2.799}\pm 0.003$
Semantic prior only	$3.217\pm 0.019$	$3.129\pm 0.006$	$3.090\pm 0.008$
Residual embeddings only	$2.959\pm 0.012$	$2.937\pm 0.014$	$2.976\pm 0.050$
Frozen isometry + residual	$3.158\pm 0.027$	$2.974\pm 0.015$	$3.057\pm 0.001$
Frozen isometry only	$4.689\pm 0.005$	$4.500\pm 0.017$	$4.513\pm 0.014$

Table 1: Global CPT test CE (nats, lower is better), mean

\pm

sample SD across three seeds.

Model	All 14	Diseases 13	Frequent 7	Rare 6
BioLORD (mean)	$0.297$	$0.254$	$0.345$	$0.148$
BioLORD (decay)	$0.308$	$0.266$	$0.356$	$0.162$
OLMo 1B, native text	$0.266$	$0.221$	$0.303$	$0.126$
Llama 1B, native text	$0.261$	$0.215$	$0.302$	$0.113$
OLMo 7B, native text	$0.255$	$0.209$	$0.290$	$0.114$
RF, tuned (count-based)	$0.330$	$0.288$	$0.373$	$0.190$

Table 2: Downstream AUPRC. Multiclass tasks use macro-AUPRC. Scores are mean

\pm

sample SD over seeds (each seed averaged over 2020–2022).

OLMo 1B
Configuration	$\Delta$ CE	Freq. 7	Rare 6	GI/Resp.
Semantic prior only	$0.371$	$+0.38$	$+1.91$	$+0.30$
Residual embeddings only	$0.112$	$+1.30$	$+3.60$	$+0.66$
Frozen isometry + residual	$0.312$	$+2.69$	$+3.30$	$+6.69$
Frozen isometry only	$1.842$	$+5.95$	$+8.48$	$+6.78$
Llama 1B

Table 3: Full model improvement over ablations (positive favors Full model ): reduction in CPT test cross-entropy for

\Delta

CE (nats) and gain in AUPRC for downstream columns (percentage points), paired by seed and year. GI/Resp. averages the two class AUPRCs. Full tables in Appendix H .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Fraction	Patients	Chunks	Clinical events
Train	98%	3.95M	4.46M	717M
Validation	1%	40,197	45,484	7.32M
Test	1%	39,734	45,144	7.32M

Appendix

Table 4: Continued-pretraining cohort and splits. Patients with at least one clinical event. Chunking retains all 732M clinical events. Event types by modality: diagnosis 44,079; procedure 28,401; laboratory test 2,853; chief complaint 2,174; medication 817; encounter type 59; vital signs 1.

Backbone	Checkpoint	$H$	Layers	Heads	KV heads	Emb. norm
OLMo2-1B	allenai/OLMo-2-0425-1B	2,048	16	16	16	10.724
Llama3.2-1B	meta-llama/Llama-3.2-1B	2,048	16	32	8	0.987
OLMo2-7B	allenai/OLMo-2-1124-7B	4,096	32	32	32	7.423

Appendix

Table 5: Frozen backbones.

H

: hidden size; Emb. norm: mean norm of the pretrained token embeddings, which sets the initialization scale of

P_{\mathrm{sem}}

A. Totals
	Size	OLMo2-1B	Llama3.2-1B	OLMo2-7B
Original text LM, including LM head		1,484,916,736	1,235,814,400	7,298,617,344
Frozen backbone used by EHRAdapt		1,279,395,840	1,235,814,400	6,887,575,552
Trainable EHRAdapt parameters		6,821,685	6,825,781	8,556,341
Complete EHRAdapt model		1,286,217,525	1,242,640,181	6,896,131,893
Trainable fraction		0.53%	0.55%	0.12%

Appendix

Table 6: Parameter counts for the full EHRAdapt configuration (78,388-event vocabulary, linear semantic projection, rank-64 projected residual, tied prediction head, frozen backbone). The original OLMo2 text models have separate language-model heads, which EHRAdapt replaces; Llama3.2 ties its head to its input embeddings, so its two backbone counts are identical. In panel B,

V

is the vocabulary size,

r

the residual rank,

H

the hidden size,

L

and

A

the numbers of layers and attention heads, and

K

the number of elapsed-time buckets; 768 is the BioLORD embedding dimension.

Setting	OLMo2 1B	Llama3.2 1B	OLMo2 7B
Backbone optimization	Frozen
Optimizer	AdamW
Precision; attention	BF16; sdpa
Effective batch	128 chunks (no gradient accumulation)
Learning rate: output/temporal	$1\mathrm{e}{-4}$ for vocabulary bias, logit scale, and temporal-attention bias
Learning rate: event encoder	$3\mathrm{e}{-4}$ for semantic projection, residual parameters, and modality embeddings

Appendix

Table 7: Training configuration. Settings in the upper block are shared by all backbones.

Class	2019	2020	2021	2022	Total
Gastrointestinal	1,420	742	808	1,150	4,120
Respiratory	9,474	6,943	3,765	6,899	27,081
Known	10,894	7,685	4,573	8,049	31,201
Control	19,447	13,524	8,261	14,376	55,608
All positive cases	21,788	15,370	9,146	16,098	62,402
Total	41,235	28,894	17,407	30,474	118,010

Appendix

Table 8: Syndromic classification case counts per year. Each test year uses all preceding years for training.

Class	2019	2020	2021	2022	Total
Campylobacteriosis	825	448	479	634	2,386
Salmonellosis (nontyphoidal)	590	357	298	481	1,726
Chickenpox (varicella)	386	255	178	218	1,037
Shigellosis	419	180	246	360	1,205
Pertussis	486	97	28	30	641
Invasive H. influenzae	190	118	63	107	478

Appendix

Table 9: Infectious-disease classification case counts per year. Each test year uses all preceding years for training. Rare-6 diseases are identified by † .

	Test 2020		Test 2021		Test 2022
Task	Train	Test	Train	Test	Train	Test
Syndromic	41,235	28,894	70,129	17,407	87,536	30,474
Infectious	7,192	3,850	11,042	3,523	14,565	4,564

Appendix

Table 10: Full-refit training and held-out annual test counts for syndromic and infectious disease tasks.

Class	ICD-9-CM	ICD-10-CM
Syndromic classification
Gastrointestinal	–	A02.9, A03.0–A03.3, A03.8–A03.9, A04.5, A05.1, A05.3, A05.5, A32.11, A32.12, A32.7, A32.89, A32.9, B96.21
Respiratory	460, 461.9, 465.8, 465.9, 466.0, 486, 487.0, 487.1, 487.8, 488.01, 488.02, 488.09, 488.11, 488.12, 488.81, 488.82, 488.89, 490	J00, J01.90, J06.9, J09.X1–J09.X3, J09.X9, J10.00, J10.01, J10.08, J10.1, J10.2, J10.81, J10.89, J11.00, J11.08, J11.1, J11.2, J11.89, J12.89, J12.9, J18.1, J18.8, J18.9, J20.9, J40
Known	Any case diagnosis code outside the two syndromic code lists
Control	Matched patients (same demographics and geographic location) with none of the codes above
Infectious-disease classification

Appendix

Table 11: ICD codes defining the classes in the two downstream tasks. Ranges are inclusive.

OLMo2-1B
Configuration	CE	Top-1 (%)	Top-10 (%)
Full model	$2.847\pm 0.006$	$40.201\pm 0.151$	$75.433\pm 0.077$
Semantic prior only	$3.217\pm 0.019$	$36.653\pm 0.269$	$71.378\pm 0.246$
Residual embeddings only	$2.959\pm 0.012$	$39.431\pm 0.126$	$73.649\pm 0.154$
Frozen isometry + residual	$3.158\pm 0.027$	$36.568\pm 0.412$	$70.913\pm 0.419$
Frozen isometry only	$4.689\pm 0.005$	$23.342\pm 0.083$	$50.138\pm 0.108$

Appendix

Table 12: Global held-out CPT test metrics for Full model and ablations. Mean

\pm

sample SD over three seeds per configuration

OLMo2-1B
Configuration	Frequent 7	Rare 6	GI/Resp.
Semantic prior only	$+0.38\pm 0.16$	$+1.91\pm 1.14$	$+0.30\pm 0.53$
Residual embeddings only	$+1.30\pm 0.84$	$+3.60\pm 1.18$	$+0.66\pm 0.79$
Frozen isometry + residual	$+2.69\pm 2.01$	$+3.30\pm 2.70$	$+6.69\pm 2.03$
Frozen isometry only	$+5.95\pm 0.22$	$+8.48\pm 1.35$	$+6.78\pm 2.33$
Llama3.2-1B

Appendix

Table 13: Paired downstream advantages of Full model over ablations in AUPRC percentage points (mean

\pm

sample SD of three seed-level differences).

OLMo2-1B
Configuration	All 14	Diseases 13	Frequent 7	Rare 6
Full model	$0.366\pm 0.004$	$0.326\pm 0.005$	$0.405\pm 0.002$	$0.233\pm 0.010$
Semantic prior only	$0.355\pm 0.002$	$0.315\pm 0.003$	$0.402\pm 0.001$	$0.214\pm 0.005$
Residual embeddings only	$0.344\pm 0.005$	$0.302\pm 0.006$	$0.392\pm 0.007$	$0.197\pm 0.004$
Frozen isometry + residual	$0.338\pm 0.020$	$0.296\pm 0.021$	$0.379\pm 0.019$	$0.200\pm 0.024$
Frozen isometry only	$0.298\pm 0.001$	$0.255\pm 0.001$	$0.346\pm 0.001$	$0.148\pm 0.004$

Appendix

Table 14: Infectious-disease classification: absolute macro-AUPRC values for the Full model and its ablations. Values are mean

\pm

sample SD across three seed-level annual averages.

OLMo2-1B
Configuration	All 4	GI	Respiratory
Full model	$0.791\pm 0.005$	$0.681\pm 0.005$	$0.730\pm 0.009$
Semantic prior only	$0.787\pm 0.003$	$0.679\pm 0.008$	$0.726\pm 0.004$
Residual embeddings only	$0.786\pm 0.000$	$0.670\pm 0.003$	$0.727\pm 0.001$
Frozen isometry + residual	$0.742\pm 0.022$	$0.649\pm 0.008$	$0.628\pm 0.046$
Frozen isometry only	$0.742\pm 0.013$	$0.596\pm 0.020$	$0.679\pm 0.020$

Appendix

Table 15: Syndromic classification: absolute macro-AUPRC (multiclass) and AUPRC (GI and Respiratory) values for the Full model and its ablations. Values are mean

\pm

sample SD across three seed-level annual averages.

Explore similar work

PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Models
Jun 23, 2026Lin Lawrence Guo, Adam Paul Yan, Emily Vettese +1Medical World Model Electronic Health Records
Child Health Evaluative Sciences · The Hospital for Sick Children · Toronto, Canada +1
ChatHealthAI: Aligning Electronic Health Record Representations with Large Language Models for Grounded Clinical Reasoning
Jun 1, 2026Bo-Hong Wang, Baicheng Peng, Ruilin Wang +3Medical World Model Clinical Reasoning Training
School of Computer Science, McGill University, Montreal, QC, Canada · Mila - Quebec AI Institute, Montreal, QC, Canada
RePrompT: Recurrent Prompt Tuning for Integrating Structured EHR Encoders with Large Language Models
Apr 20, 2026Arya Hadizadeh Moghaddam, Drew Ross, Mohsen Nayebi Kerdabadi +2Electronic Health Records Longitudinal Patient Records
University of Kansas, USA

Jun 23, 2026, cs.CLJ/K move · Enter open · S save

PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Models

Lin Lawrence Guo, Adam Paul Yan, Emily Vettese, Lillian Sung

Child Health Evaluative Sciences · The Hospital for Sick Children · Toronto, Canada+1

Medical World Model Electronic Health Records Clinical

Open paper AskPDF arXiv

Most electronic health record (EHR) foundation models encode clinical events as discrete event tokens from a fixed vocabulary and therefore cannot directly represent events containing unseen concepts or new combinations of concepts and attributes such as numeric values. This limits transfer across institutions and even across deployment pipelines within the same institution. We introduce PORTER, a language-grounded structured EHR foundation model that decouples event representation from this fixed vocabulary. PORTER represents events through their descriptions using a frozen text encoder, integrates numeric values through a dedicated pathway, and learns clinical dynamics over patient timelines with an autoregressively pretrained temporal backbone. Across 74 clinical prediction tasks at a pediatric hospital, PORTER matched the mean AUROC of a fixed-vocabulary model with the same temporal backbone and pretraining objective. When the same patient timelines were rendered using event descriptions not seen during pretraining, PORTER transferred without retraining or vocabulary mapping, recovering 97.1% of the mean AUROC of a model trained directly on the target vocabulary. When transferred to MIMIC, PORTER outperformed the fixed-vocabulary model, which dropped 69% of events because their tokens were unseen. Mechanistic analyses showed cross-vocabulary transfer tracked preservation of patient-level representation geometry rather than the scale of the text encoder, and the numeric pathway improved sensitivity to magnitude without disrupting clinical concept identity. PORTER also achieved higher AUROC than a task-specific text serialization comparator, at 329-fold lower amortized compute. PORTER is a step toward vocabulary-independent EHR foundation models that reduce the need for vocabulary harmonization while preserving in-domain performance and enabling efficient cross-task reuse.

Abstract

Explore similar work

PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Models

ChatHealthAI: Aligning Electronic Health Record Representations with Large Language Models for Grounded Clinical Reasoning

RePrompT: Recurrent Prompt Tuning for Integrating Structured EHR Encoders with Large Language Models