cs.CLNov 12, 2026

Evaluating Pretrained Language Semantics for Structured EHR Prediction Tasks

Authors: Ashvin Gupta, Brendan Delaney, Alessandra Russo

Organizations: Imperial College London, United Kingdom

Abstract

Structured Electronic Health Record (EHR) models typically represent clinical concepts as dataset-specific tokens, discarding their associated natural-language semantics. Consequently, relationships between concepts must largely be learned from EHR co-occurrence without directly exploiting pretrained linguistic knowledge. We investigate the contribution of pretrained language semantics to longitudinal EHR prediction through tEHR, a framework that converts clinical events into natural-language descriptions and adapts pretrained autoregressive language models through EHR-specific continued pretraining and task-specific discriminative fine-tuning. We evaluate tEHR across four MIMIC-IV prediction tasks, where Mistral-7B achieves the strongest overall performance against structured-data, code-based, and LLM-based baselines. Controlled ablations demonstrate the importance of lexical grounding, numerical values, and both adaptation stages. A capacity-controlled experiment further shows that replacing clinical descriptions with atomic code tokens substantially reduces performance despite retaining the same LLM backbone. Alternative approaches incorporating pretrained semantic embeddings offer lower computational costs but do not match full textualisation. These results highlight the importance of both semantic representation and downstream adaptation when applying pretrained language models to structured EHR prediction.

Explore similar work

CardsList
  1. ChatHealthAI: Aligning Electronic Health Record Representations with Large Language Models for Grounded Clinical Reasoning

    Jun 1, 2026Bo-Hong Wang, Baicheng Peng, Ruilin Wang +3LLM GroundingLLM Alignment

  2. EHRAdapt: Adapting Pretrained Language Models to Electronic Health Records with Semantic Priors for Rare Clinical Events

    Sep 27, 2026Andre R Goncalves, Vincent Liu, Priyadip RayDomain Adaptation for LLMsLong-Tail Learning

  3. RePrompT: Recurrent Prompt Tuning for Integrating Structured EHR Encoders with Large Language Models

    Apr 20, 2026Arya Hadizadeh Moghaddam, Drew Ross, Mohsen Nayebi Kerdabadi +2Representation LearningPrompt Tuning