Challenges in Explaining Pretrained Clinical Text Classifiers

Kristian Miok; Matej Klemen; Blaz Škrlj; Marko Robnik Šikonja

cs.CLMay 27, 2026

Challenges in Explaining Pretrained Clinical Text Classifiers

Authors: Kristian Miok, Matej Klemen, Blaz Škrlj, Marko Robnik Šikonja

Organizations: Faculty of Computer and Information Science, University of Ljubljana, Slovenia · ICAM - Advanced Environmental Research Institute, West University of Timisoara, Romania · Jožef Stefan Institute

Abstract

Explaining the predictions of neural models in clinical NLP remains a significant challenge, especially for complex tasks involving long, unstructured medical texts. While post-hoc methods like LIME and SHAP are widely used, they often fall short when applied to clinical narratives. In this paper, we identify core limitations of token-level and perturbation-based explanation techniques through targeted demonstra- tions on a hospital length-of-stay prediction task. Our findings reveal issues such as overemphasis on non-informative tokens, instability in at- tributions, and high-confidence predictions for incoherent input variants. These results underscore the need for explanation strategies that are clin- ically meaningful, semantically grounded, and robust to linguistic noise.

Explore similar work

X-FEMR: A Token-level Explainable Approach for Electronic Health Records Foundation Models using Transformer-based Models
Jul 7, 2026Jie Huang, Pengfei Yin, Zihan Xu +3Electronic Health Record Foundation Models Explainability
NEURON: A Neuro-symbolic System for Grounded Clinical Explainability
May 2, 2026Anuradha Chandrasekaran, Dimitrios Zikos, Mutlu Mete +3Explainability Model Interpretability
Explaining BiomedCLIP with Weighted Banzhaf Interactions Supported by Tree-Gram Parsing
Jul 25, 2026Jakub Rymarski, Adam Rempała, Bartłomiej Sobieski +1Medical Vision-Language Models

Jul 7, 2026 · cs.LGJ/K move · Enter open · S save

X-FEMR: A Token-level Explainable Approach for Electronic Health Records Foundation Models using Transformer-based Models

Jie Huang, Pengfei Yin, Zihan Xu, Daniel Capurro+2

School of Computing and Information Systems, University of Melbourne, Melbourne, Australia · Department of General Medicine, Royal Melbourne Hospital Melbourne, Australia

Foundation Models for Electronic Health Records (FEMRs) are pretrained on large-scale structured patient data, enabling them to convert longitudinal patient trajectories into generalizable representations for diverse clinical prediction tasks. Despite their effectiveness, FEMRs remain black-box models, raising concerns about bias, interpretability, and clinical trust. To address this, we propose the first token-level explainability approach for FEMRs. We train a Transformer-based surrogate model on input-output pairs from the FEMR across two prediction tasks, approximating its behavior while preserving temporal dynamics. We identify the most influential tokens, providing insights into how FEMRs leverage different aspects of patient history for predictions. To evaluate clinical relevance, we introduce a novel clinical alignment metric that quantifies the correspondence between the surrogate model's key tokens and clinically validated features. Our results demonstrate that the surrogate closely approximates FEMR predictions and that token-level explanations align well with clinical knowledge, offering a practical framework for interpretable and trustworthy clinical AI.

Electronic Health Record Foundation Models Explainability Surrogate Models Finite Element Method

Open paper Ask about it PDF

Abstract

Explore similar work

X-FEMR: A Token-level Explainable Approach for Electronic Health Records Foundation Models using Transformer-based Models

NEURON: A Neuro-symbolic System for Grounded Clinical Explainability

Explaining BiomedCLIP with Weighted Banzhaf Interactions Supported by Tree-Gram Parsing