Discrete Neural Interlingua for Ontology-Agnostic EHR Modeling

Tong Ding; Caiwei Tian; Dongmin Bang; Ming Y. Lu; Sophia J. Wagner; Andrew Zhang; Rowland Pettit; Long Phi Le

cs.CLOct 19, 2026

Discrete Neural Interlingua for Ontology-Agnostic EHR Modeling

Authors: Tong Ding, Caiwei Tian, Dongmin Bang, Ming Y. Lu, Sophia J. Wagner, Andrew Zhang, Rowland Pettit, Long Phi Le,

Organizations: Mass General Brigham AI Institute · Harvard University · Massachusetts Institute of Technology

Abstract

EHR foundation models promise scalable clinical prediction across health systems, yet their deployment is constrained by a tokenization bottleneck. The same clinical concept can appear as different ontology codes, local identifiers, synonyms, or free-text descriptions, which standard EHR tokenizers treat as unrelated symbols. A model trained on MIMIC-III with a closed-vocabulary tokenizer can use only 13% of eICU events. We introduce MEDIATOR, a discrete ontology-agnostic tokenizer that maps any clinical concept expressed as text into a shared vector-quantized codebook, turning EHR harmonization into open-vocabulary tokenization without manual mappings or a fixed vocabulary. Trained with synonym supervision and guided by enriched concept descriptions, code co-occurrence, and relation graphs, MEDIATOR collapses equivalent expressions while organizing distinct concepts in a medically meaningful discrete space. Used as the token interface of an EHR Transformer, MEDIATOR improves mean AUPRC over the strongest prior tokenizer by 4.0 points (8% relative) on 20 MIMIC-III tasks and by 8.6 points (29% relative) in zero-shot transfer to 14 eICU tasks, where it uses all events. The transfer gain combines coverage and representation, as MEDIATOR remains the best tokenizer under identical inputs (+1.0 and +1.9 points on operational and subcohort-mortality tasks). Same-encoder controls show that the text encoder drives in-domain accuracy, while the learned objectives of MEDIATOR drive transfer robustness and entity-linking quality. MEDIATOR also enables transferable attribution across hospitals.

Figures & tables

Figure 1: Motivation for Mediator . a . The vocabularies of Hospital A (light blue) and Hospital B (orange) only partly overlap, so models trained at A fail on B-specific codes and drop free text. b . Mediator maps equivalent surface forms from both hospitals, colored by source hospital, to one shared vector-quantized (VQ) token (dark blue). c . Shared tokens let one EHR model transfer across hospitals, with predictions and attributions defined over the same tokens.

Figure 2: Mediator architecture. A text encoder maps any clinical concept description to a dense embedding, a trainable adaptor projects it into the quantization space, and the result is assigned to a learned VQ codebook. Training combines same-concept contrastive objectives before and after quantization, multi-view assignment agreement, semantic teachers from enriched descriptions, code co-occurrence, and relation graphs, and reconstruction to the encoder embedding space.

Figure 3: UMAP visualization of the Mediator codebook. Each point is a source concept, shown by its assigned VQ code embedding projected with UMAP. Colors indicate independent ontology-derived categories for (a) diagnosis codes, (b) measurements, (c) SNOMED clinical concepts, and (d) procedures. Dashed boxes mark example categories. Full category legends are in Appendix E .

Task	Code Emb	ClinVec	MedTok	$\text{MedTok}_{\text{text}}$ -top5	$\text{MedTok}_{\text{text}}$ -top1	Mediator
Operational tasks
LOS $<$ 7	0.819(0.012)	0.807(0.004)	0.827(0.007)	0.830(0.005)	0.762(0.011)	0.829(0.007)
ICU Mortality	0.571(0.015)	0.484(0.006)	0.556(0.008)	0.563(0.007)	0.418(0.014)	0.573(0.010)
Readmission	0.336(0.026)	0.342(0.018)	0.366(0.021)	0.347(0.021)	0.336(0.022)	0.374(0.023)
Mean	0.576	0.544	0.583	0.580	0.505	0.592
Subcohort mortality tasks

Table 1: EHR prediction performance on MIMIC-III, reported as mean (std) AUPRC across folds. Across the 20 tasks, Mediator improves over MedTok (one-sided Wilcoxon signed-rank test,

p=9.5\times 10^{-7}

). Patient-level bootstrap tests (5,000 resamples) show significant gains (

p<0.05

) on 7/20 tasks over MedTok and 17/20 tasks over Code Emb.

Task	Code Emb	ClinVec	MedTok	$\text{MedTok}_{\text{text}}$ -top5	$\text{MedTok}_{\text{text}}$ -top1	Mediator
Operational tasks
LOS $<$ 7	0.933(0.003)	0.939(0.002)	0.934(0.001)	0.941(0.026)	0.939(0.001)	0.942(0.004)
ICU Mortality	0.259(0.018)	0.211(0.010)	0.282(0.009)	0.162(0.066)	0.145(0.048)	0.349(0.005)
Mean	0.596	0.575	0.608	0.552	0.542	0.645
Subcohort mortality tasks
Acute Kidney Injury	0.318(0.032)	0.237(0.019)	0.347(0.013)	0.233(0.031)	0.213(0.002)	0.408(0.011)

Table 2: EHR prediction performance on eICU, reported as mean (std) AUPRC across folds. Across the 14 tasks, Mediator improves over MedTok (one-sided Wilcoxon signed-rank test,

p=6.1\times 10^{-5}

). Patient-level bootstrap tests (5,000 resamples) show significant gains (

p<0.05

) on 12/14 tasks over MedTok and 14/14 tasks over Code Emb.

	NCBI			MedMentions			COMETA
Model	Acc.	Pur.	F1	Acc.	Pur.	F1	Acc.	Pur.	F1
Qwen3-8B + $k$ -means	0.487	0.928	0.639	0.450	0.543	0.492	0.648	0.807	0.718
MedTok text -top5	0.195	0.986	0.326	0.045	0.885	0.085	0.231	0.913	0.368
MedTok text -top1	0.269	0.360	0.308	0.131	0.082	0.101	0.384	0.101	0.160
Mediator	0.524	0.907	0.664	0.471	0.516	0.493	0.682	0.794	0.734

Table 3: Direct open-vocabulary tokenization evaluation. Tests whether each tokenizer maps unseen mentions and their ground-truth concept descriptions to one code while preserving purity.

Figure 4: Transferable attribution. Top-15 tokens by IG attribution to cirrhosis mortality on MIMIC-III (left) and zero-shot eICU (right). Shared VQ indices are blue, others gray.

Table 8

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Table 9

		MIMIC-III		eICU
Group	Task	Total Stays	Events	Total Stays	Events
Hospital Operation	LOS $<$ 7 days	40757	18931	83166	21818
	ICU Mortality	31650	2954	83166	7003
	Readmission	9515	2613	–	–
Subcohort ICU Mortality	Acute Kidney Injury	6757	1370	8309	1367
	Acute MI	3681	439	5671	414

Appendix

Table A.3 : Dataset statistics for prediction tasks on MIMIC-III and eICU. Total Stays is the number of ICU stays in each task cohort (one per patient); Events is the number of positive labels.

Figure A.1 : UMAP visualization of the Mediator codebook with full category legends. Each point denotes a source concept represented by its assigned VQ code embedding and projected to two dimensions with UMAP. Colors indicate ontology-derived categories, with the number of concepts per category in parentheses, for (a) diagnosis codes (ICD-9 and ICD-10), (b) measurement concepts (LOINC), (c) clinical concepts (SNOMED CT), and (d) procedure codes (ICD-9-PCS, ICD-10-PCS, and CPT-4).

Figure A.2 : Effect of codebook size

K

. We vary the VQ codebook size and report mean BEL (VQ Evaluation) F1 and operational-task mean AUPRC on MIMIC-III and zero-shot eICU. We additionally report codebook usage on the training corpus.

Codebook size	NCBI	MedMentions	COMETA	Mean
10k	0.663	0.343	0.669	0.558
20k	0.670	0.409	0.707	0.595
40k	0.681	0.451	0.723	0.619
60k	0.637	0.469	0.732	0.613
80k	0.673	0.484	0.738	0.632
100k	0.664	0.493	0.734	0.630

Appendix

Table A.4 : Biomedical entity linking F1 across codebook sizes.

	MIMIC-III				eICU
Method	Oper.	Mort.	Surv.	All	Oper.	Mort.	All
$-\mathcal{L}_{\mathrm{agree}}$	0.582 $\pm$ 0.010	0.524 $\pm$ 0.010	0.610 $\pm$ 0.023	0.554 $\pm$ 0.003	0.642 $\pm$ 0.006	0.334 $\pm$ 0.010	0.378 $\pm$ 0.008
$-\mathcal{L}_{\mathrm{co}}$ (PPMI)	0.585 $\pm$ 0.011	0.530 $\pm$ 0.013	0.628 $\pm$ 0.054	0.562 $\pm$ 0.008	0.644 $\pm$ 0.005	0.324 $\pm$ 0.013	0.370 $\pm$ 0.011
$-\mathcal{L}_{\mathrm{rel}}$	0.590 $\pm$ 0.007	0.549 $\pm$ 0.006	0.585 $\pm$ 0.045	0.564 $\pm$ 0.010	0.644 $\pm$ 0.005	0.330 $\pm$ 0.010	0.375 $\pm$ 0.009
$-\mathcal{L}_{\mathrm{desc}}$	0.588 $\pm$ 0.009	0.537 $\pm$ 0.022	0.601 $\pm$ 0.051	0.561 $\pm$ 0.018	0.640 $\pm$ 0.008	0.337 $\pm$ 0.007	0.380 $\pm$ 0.006
Mediator full	0.592 $\pm$ 0.007	0.533 $\pm$ 0.012	0.639 $\pm$ 0.037	0.568 $\pm$ 0.011	0.645 $\pm$ 0.003	0.340 $\pm$ 0.009	0.384 $\pm$ 0.007

Appendix

Table A.5 : Leave-one-out component ablation by task family. Mean AUPRC

\pm

standard deviation across folds for task-family means on MIMIC-III and zero-shot eICU transfer.

Objective	MIMIC-III Oper.	eICU Oper.
Baseline	0.585	0.641
Baseline $+\,\mathcal{L}_{\mathrm{desc}}$	0.586	0.642
Baseline $+\,\mathcal{L}_{\mathrm{co}}$ (PPMI)	0.580	0.645
Baseline $+\,\mathcal{L}_{\mathrm{rel}}$	0.584	0.635
Baseline $+\,\mathcal{L}_{\mathrm{agree}}$	0.583	0.644
Baseline $+\,\mathcal{L}_{\mathrm{agree}}+\mathcal{L}_{\mathrm{desc}}$	0.584	0.637

Appendix

Table A.6 : Additive component ablation. Mean AUPRC over operational tasks on MIMIC-III and zero-shot eICU transfer, adding components to a baseline trained with only the equivalence, VQ, and reconstruction losses.

Analysis	Variant	Dim.	MIMIC-III	eICU
A. Alternative frozen encoders for Mediator
Encoder	BioLORD-2023	768	0.551	0.312
Encoder	Qwen3-Embedding-0.6B	1024	0.550	0.372
Encoder	Qwen3-Embedding-8B	4096	0.568	0.384
B. Same-encoder clustering controls
Tokenizer	Qwen embeddings + $k$ -means	4096	0.568	0.378

Appendix

Table A.7 : Encoder and tokenizer ablations. Panel A varies the frozen text encoder used by Mediator . Panel B keeps the same frozen Qwen3-Embedding-8B encoder fixed and compares Mediator against direct clustering controls. All results report mean AUPRC on 20 MIMIC-III tasks and 14 zero-shot eICU transfer tasks.

Tokenizer	MIMIC-III	eICU
GT-BEHRT
Code Emb	0.572	0.313
ClinVec	0.535	0.280
MedTok	0.573	0.313
MedTok text -top1	0.511	0.219
MedTok text -top5	0.534	0.224

Appendix

Table A.8 : Tokenizer comparison under stronger EHR architectures. We evaluate Mediator as the token interface for GT-BEHRT and TransformEHR. Results are mean AUPRC on 20 MIMIC-III tasks and 14 zero-shot eICU transfer tasks.

Task	Code Emb	ClinVec	MedTok	$\text{MedTok}_{\text{text}}$ -top5	$\text{MedTok}_{\text{text}}$ -top1	Mediator
Operational tasks
LOS $<$ 7	0.933(0.003)	0.939(0.002)	0.934(0.001)	0.937(0.004)	0.926(0.003)	0.938(0.002)
ICU Mortality	0.259(0.018)	0.211(0.010)	0.282(0.009)	0.138(0.022)	0.130(0.010)	0.298(0.021)
Mean	0.596	0.575	0.608	0.537	0.528	0.618
Subcohort mortality tasks
Acute Kidney Injury	0.318(0.032)	0.237(0.019)	0.347(0.013)	0.184(0.023)	0.185(0.011)	0.345(0.010)

Appendix

Table A.9 : Matched-input zero-shot transfer on eICU. All models are evaluated on the overlapping subset of eICU codes available to closed-vocabulary baselines. We report mean AUPRC with standard deviation across folds. Best results are bolded and second-best results are underlined.

Input string $x$	$h=f_{\eta}(x)$	$z=p_{\theta}(h)$	$k^{\star}(z)$
“shortness of breath” (clinical phrase)	$h_{1}$	$z_{1}$	$i$
“can’t breathe” (colloquial)	$h_{2}$	$z_{2}$	$i$
“dyspnea” (SNOMED CT term)	$h_{3}$	$z_{3}$	$i$
“diabetes mellitus” (distinct concept)	$h_{4}$	$z_{4}$	$j\neq i$

Appendix

Table A.10 : Worked example of Mediator tokenization. Each string is encoded by the frozen text encoder

f_{\eta}

, projected into the quantization space by

p_{\theta}

, and assigned to its nearest codebook entry. Three surface forms of dyspnea, from a clinical phrase, a colloquial expression, and the SNOMED CT term, are assigned the same code

i

, whereas the distinct concept diabetes mellitus is assigned a different code

j

. Because the codebook is fixed after training, strings from new code systems are tokenized in the same way without retraining.

Tokenization in Language Models

See all 47 papers

Tokenization Tradeoffs in Structured EHR Foundation Models
Mar 3, 2026Lin Lawrence Guo, Santiago Eduardo Arciniegas, Joseph Jihyung Lee +4EHR Foundation Models Longitudinal EHR Modeling
Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, Canada · Division of Haematology/Oncology, The Hospital for Sick Children, Toronto, Canada · Department of Medicine, University Health Network, Toronto, Canada +1
PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Models
Jun 23, 2026Lin Lawrence Guo, Adam Paul Yan, Emily Vettese +1EHR Foundation Models Clinical Outcome Prediction
Child Health Evaluative Sciences · The Hospital for Sick Children · Toronto, Canada +1
INTERVenE: Temporal-Abstraction-Interval Based Transformers for Short-Horizon Medical Event Prediction
Aug 30, 2026Shahar Oded, Yuval ShaharEvent Time Prediction Interpretable ML
The Stein Faculty of Computer and Information Science, Ben-Gurion University of the Negev, Beer-Sheva, Israel

Mar 3, 2026J/K move · Enter open · S save

Tokenization Tradeoffs in Structured EHR Foundation Models

Lin Lawrence Guo, Santiago Eduardo Arciniegas, Joseph Jihyung Lee, Adam Paul Yan+3

Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, Canada · Division of Haematology/Oncology, The Hospital for Sick Children, Toronto, Canada · Department of Medicine, University Health Network, Toronto, Canada+1

EHR Foundation Models Longitudinal EHR Modeling Electronic Health Records

Open paper AskPDF arXiv

Foundation models for structured electronic health records (EHRs) are pretrained on longitudinal sequences of timestamped clinical events to learn adaptable patient representations. Tokenization -- how these timelines are converted into discrete model inputs -- determines what information is preserved, how efficiently it is encoded, and which relationships must be learned versus precomputed. Yet the impact of tokenization design choices on downstream performance and computational efficiency remains largely unexplored. Here, we pretrained a transformer on pediatric EHR data under a factorial design, varying tokenization along event encoding, time encoding, and workflow annotation. We evaluated area-under-the-receiver-operating-characteristic curve across 74 clinical prediction tasks. Joint event encoding and positional time encoding outperformed their alternatives (73/74 and 71/74 tasks) while requiring 39.5% and 9.6% fewer pretraining floating-point operations, respectively. Targeted ablations traced the joint encoding advantage to local binding efficiency, that is, code-attribute pairs are combined into single tokens, rather than split across tokens that the model must learn to associate during pretraining. External evaluation on an adult intensive care unit cohort demonstrated that this advantage generalizes despite substantial vocabulary mismatch, while temporal and workflow effects remain institution-specific. These results establish tokenization as a tractable lever for improving both the performance and efficiency of EHR foundation models.

Abstract

Tokenization in Language Models

Explore similar work

Tokenization Tradeoffs in Structured EHR Foundation Models

PORTER: Language-Grounded Event Representations for Portable Structured EHR Foundation Models

INTERVenE: Temporal-Abstraction-Interval Based Transformers for Short-Horizon Medical Event Prediction