GraphWalker: Patient Analogy Meets Information Gain for Clinical Reasoning with Large Language Models

Yue Fang; Weibin Liao; Yuxin Guo; Jiaran Gao; Hongxin Ding; Jinyang Zhang; Xinke Jiang; Zhibang Yang

cs.LGApr 8, 2026

GraphWalker: Patient Analogy Meets Information Gain for Clinical Reasoning with Large Language Models

Authors: Yue Fang, Weibin Liao, Yuxin Guo, Jiaran Gao, Hongxin Ding, Jinyang Zhang, Xinke Jiang, Zhibang Yang,

Organizations: School of Computer Science, Peking University, Beijing, China · National Engineering Research Center for Software Engineering, Peking University, Beijing, China

Abstract

Clinical reasoning over electronic health records (EHRs) is a fundamental yet challenging task in modern healthcare. While large language models (LLMs) offer a promising paradigm via in-context demonstrations that requires no task-specific parameter updates, existing methods for reasoning by patient analogy in EHR settings suffer from three core limitations: (1) Perspective Limitation, where data-driven similarity misaligns with LLM reasoning needs while model-driven signals are constrained by limited clinical competence; (2) Cohort Awareness, as demonstrations are selected independently without modeling population-level structure; and (3) Information Aggregation, where redundancy and interaction effects among demonstrations are ignored. We propose GraphWalker, a training-free framework that lets frozen LLMs reason by analogy over retrieved patient cases. GraphWalker (i) jointly leverages data-driven and model-driven perspectives, (ii) discovers patient cohorts to ground retrieval in population-level structure, and (iii) employs a lazy greedy search with frontier expansion to compose demonstrations with high marginal information gain. Extensive experiments on multiple real-world EHR benchmarks show that GraphWalker consistently outperforms state-of-the-art demonstration selection baselines, and remains substantially more robust under cross-dataset distribution shift, without task-specific parameter updates. GraphWalker further generalizes to black-box LLMs and composes naturally with agentic reasoning frameworks, positioning it as a pluggable patient-analogy skill in LLM-based clinical workflows. Our code is available at https://github.com/PuppyKnightUniversity/GraphWalker.

Figures & tables

Figure 1: Illustration of existing demonstration selection methods (top) and their three core challenges (bottom).

Figure 2: Analysis of the Limitations of a Single Perspective.

Figure 3: Analysis of Cohort Discovery vs. Instance Retrieval.

Figure 4: Analysis of information aggregation across multiple demonstrations under different demonstration selection algorithms.

Qwen3-14B
Method		MIMIC-III Mortality			MIMIC-III LOS		MIMIC-IV Readmission
Paradigm	Approach	AUROC $\uparrow$	AUPRC $\uparrow$	F1 $\uparrow$	ma-ROC $\uparrow$	mi-ROC $\uparrow$	AUROC $\uparrow$	AUPRC $\uparrow$	F1 $\uparrow$
Vanilla	Zero-shot	64.09 $\pm$ 5.26	17.58 $\pm$ 3.98	31.62 $\pm$ 5.44	65.22 $\pm$ 3.00	37.49 $\pm$ 2.48	41.11 $\pm$ 4.77	27.11 $\pm$ 4.51	41.66 $\pm$ 3.80
Vanilla	Random	66.33 $\pm$ 5.12	31.04 $\pm$ 6.89	43.00 $\pm$ 5.88	58.12 $\pm$ 1.98	61.28 $\pm$ 3.35	49.98 $\pm$ 4.47	24.81 $\pm$ 3.63	40.77 $\pm$ 4.05
Data Perspective	Semantic-emb	64.43 $\pm$ 6.71	21.78 $\pm$ 6.09	33.60 $\pm$ 6.16	57.06 $\pm$ 3.77	66.60 $\pm$ 2.60	45.12 $\pm$ 4.13	26.91 $\pm$ 4.99	42.01 $\pm$ 3.94
	EHR-emb	75.24 $\pm$ 5.86	29.52 $\pm$ 8.20	42.49 $\pm$ 8.88	62.03 $\pm$ 3.13	66.98 $\pm$ 2.53	57.79 $\pm$ 4.37	32.73 $\pm$ 5.01	44.87 $\pm$ 4.01

Table 1: Performance comparison on MIMIC-III and MIMIC-IV datasets using Qwen3-14B and Llama3.1-8B-Instruct as backbone LLMs. All metrics are higher-is-better (

\uparrow

); values are mean

\pm

std over three random seeds. Bold : best; underlined : second-best.

Method	MIMIC-III Mortality			MIMIC-III LOS		MIMIC-IV Readmission
	AUROC $\uparrow$	AUPRC $\uparrow$	F1 $\uparrow$	ma-ROC $\uparrow$	mi-ROC $\uparrow$	AUROC $\uparrow$	AUPRC $\uparrow$	F1 $\uparrow$
GraphWalker	84.19 $\pm$ 3.35	59.46 $\pm$ 7.64	58.24 $\pm$ 5.89	69.51 $\pm$ 3.16	84.64 $\pm$ 2.14	75.91 $\pm$ 4.11	60.61 $\pm$ 6.19	58.01 $\pm$ 5.33
w/o EHR-emb	61.09 $\pm$ 5.32	31.13 $\pm$ 6.81	37.50 $\pm$ 5.73	62.03 $\pm$ 3.40	72.63 $\pm$ 2.46	45.30 $\pm$ 4.55	26.69 $\pm$ 4.20	41.74 $\pm$ 3.79
w/o Cohort	81.19 $\pm$ 3.78	48.27 $\pm$ 8.01	55.06 $\pm$ 6.18	65.85 $\pm$ 3.75	81.16 $\pm$ 2.27	75.65 $\pm$ 4.04	59.61 $\pm$ 6.09	56.87 $\pm$ 5.19
w/o Greedy & Frontier Expansion	69.43 $\pm$ 5.80	46.72 $\pm$ 8.24	55.34 $\pm$ 7.22	60.93 $\pm$ 3.30	53.33 $\pm$ 2.52	62.43 $\pm$ 4.72	43.69 $\pm$ 6.25	47.24 $\pm$ 4.90
w/o Early Stop	81.34 $\pm$ 3.74	48.66 $\pm$ 7.91	55.99 $\pm$ 6.18	66.31 $\pm$ 3.23	82.77 $\pm$ 2.14	75.53 $\pm$ 4.09	59.13 $\pm$ 6.12	57.38 $\pm$ 5.22

Table 2: Ablation study of GraphWalker using Qwen3-14B as backbone LLM on MIMIC-III and MIMIC-IV datasets. All evaluation metrics are defined such that higher values indicate better performance (

\uparrow

). The best results are highlighted in Bold .

Figure 6: Performance of GraphWalker over different hyper-parameters on MIMIC-IV.

Panel of Figure 7 — Figure 7: In-domain (hatched) vs. cross-dataset OOD (solid) on MIMIC-III mortality. Left: AUPRC; right: F1. Shaded regions denote the relative drop. AUROC: Appendix C.3 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Qwen3-32B
Method		MIMIC-III Mortality			MIMIC-III LOS		MIMIC-IV Readmission
Paradigm	Approach	AUROC $\uparrow$	AUPRC $\uparrow$	F1 $\uparrow$	ma-ROC $\uparrow$	mi-ROC $\uparrow$	AUROC $\uparrow$	AUPRC $\uparrow$	F1 $\uparrow$
Vanilla	Zero-shot	65.67 $\pm$ 5.75	30.54 $\pm$ 7.56	38.43 $\pm$ 6.99	55.79 $\pm$ 3.33	23.53 $\pm$ 2.46	44.19 $\pm$ 4.57	25.06 $\pm$ 3.88	42.57 $\pm$ 3.83
Vanilla	Random	59.90 $\pm$ 5.06	24.63 $\pm$ 5.29	36.22 $\pm$ 4.94	59.50 $\pm$ 3.51	80.44 $\pm$ 2.00	48.25 $\pm$ 4.65	28.13 $\pm$ 4.51	42.22 $\pm$ 3.77
Data Perspective	Semantic-emb	60.62 $\pm$ 5.80	19.87 $\pm$ 4.61	32.08 $\pm$ 5.59	57.65 $\pm$ 3.65	74.16 $\pm$ 2.38	47.86 $\pm$ 4.71	26.07 $\pm$ 3.89	42.47 $\pm$ 3.85
	EHR-emb	85.02 $\pm$ 3.21	58.56 $\pm$ 8.02	58.84 $\pm$ 6.01	63.02 $\pm$ 3.33	67.00 $\pm$ 2.71	61.32 $\pm$ 4.45	44.25 $\pm$ 5.92	46.24 $\pm$ 4.06

Appendix

Table 3: Additional results on MIMIC-III and MIMIC-IV datasets using Qwen3-32B as the backbone LLM. All metrics are higher-is-better (

\uparrow

); values are mean

\pm

std over three random seeds. Bold : best; underlined : second-best.

Method	MIMIC-III Mortality			MIMIC-IV Readmission
	AUROC $\uparrow$	AUPRC $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$	AUPRC $\uparrow$	F1 $\uparrow$
SMART (fine-tuned predictor)	80.64	45.40	47.18	74.41	54.62	57.38
GraphWalker (with SMART encoder)	84.19	59.46	58.24	75.91	60.61	58.01
$\Delta$	+3.55	+14.06	+11.06	+1.50	+5.99	+0.63
ConCare (fine-tuned predictor)	69.19	36.01	40.65	70.30	41.87	50.26
GraphWalker (with ConCare encoder)	70.46	36.93	45.90	77.01	59.49	59.44

Appendix

Table 4: Performance of GraphWalker with different EHR encoders on MIMIC-III mortality and MIMIC-IV readmission, using Qwen3-14B as the backbone LLM. For each encoder, we compare two settings: (i) the encoder fine-tuned end-to-end as a task-specific supervised predictor, and (ii) GraphWalker using the same pretrained encoder (without fine-tuning) for graph-based demonstration selection.

\Delta

denotes the absolute improvement (in percentage points) of GraphWalker over the supervised fine-tuned baseline. Note that this differs from EHR-emb in Table 1 , which uses SMART embeddings for ICL retrieval without fine-tuning.

Figure 8: Shot scaling performance on MIMIC-III Mortality with Qwen3-14B as the backbone LLM.

Figure 9: Shot scaling performance on MIMIC-IV Readmission with Qwen3-14B as the backbone LLM.

Figure 10: Performance–latency tradeoff on MIMIC-III mortality across AUROC, AUPRC, and F1 (left to right). Each bubble represents one case retrieval method, with position showing mean performance (x-axis) and average inference time per case (y-axis); bubble color identifies the method. GraphWalker (red) achieves the strongest performance across all three metrics while keeping latency comparable to prior LLM-based case selection methods.

Method	AUROC			AUPRC			F1
	In-Dom.	OOD	Drop	In-Dom.	OOD	Drop	In-Dom.	OOD	Drop
SFT	68.34 $\pm$ 6.29	51.04 $\pm$ 5.47	$\downarrow$ 25.3%	17.48 $\pm$ 4.88	11.34 $\pm$ 4.11	$\downarrow$ 35.1%	30.43 $\pm$ 6.35	17.38 $\pm$ 5.12	$\downarrow$ 42.9%
RAM-EHR Xu et al. (2024a)	80.58 $\pm$ 3.48	59.26 $\pm$ 3.35	$\downarrow$ 26.5%	53.02 $\pm$ 3.87	39.23 $\pm$ 4.47	$\downarrow$ 26.0%	51.26 $\pm$ 3.44	37.26 $\pm$ 3.67	$\downarrow$ 27.3%
GraphCare Jiang et al. (2024)	80.21 $\pm$ 3.55	62.34 $\pm$ 5.57	$\downarrow$ 22.3%	54.77 $\pm$ 3.19	39.35 $\pm$ 3.67	$\downarrow$ 28.2%	52.22 $\pm$ 3.75	36.04 $\pm$ 4.36	$\downarrow$ 31.0%
DearLLM Xu et al. (2025)	82.77 $\pm$ 3.63	69.99 $\pm$ 5.21	$\downarrow$ 15.4%	58.02 $\pm$ 3.24	42.85 $\pm$ 6.39	$\downarrow$ 26.1%	56.79 $\pm$ 3.75	43.32 $\pm$ 6.06	$\downarrow$ 23.7%
GraphWalker	84.19 $\pm$ 3.35	81.29 $\pm$ 3.91	$\downarrow$ 3.4%	59.46 $\pm$ 7.64	52.98 $\pm$ 6.01	$\downarrow$ 10.9%	58.24 $\pm$ 5.89	54.08 $\pm$ 5.84	$\downarrow$ 7.1%

Appendix

Table 5: Performance comparison of GraphWalker against supervised methods on MIMIC-III in-hospital mortality. “In-Dom.” refers to training and testing on MIMIC-III; “OOD” refers to training on MIMIC-IV and testing on MIMIC-III without fine-tuning. “Drop” shows the relative degradation from In-Domain to OOD. GraphWalker requires no parameter updates in either regime.

Figure 11: In-domain vs. cross-dataset OOD AUROC on MIMIC-III in-hospital mortality, complementing Figure 7 in the main text. Hatched bars show in-domain performance (trained and tested on MIMIC-III); solid bars show OOD performance (trained on MIMIC-IV, tested on MIMIC-III without fine-tuning); shaded regions show the relative drop. GraphWalker shows a smaller drop than the supervised baselines we evaluated.

Method	Direct Answer (Few-shot)			+ ReAct
	AUROC	AUPRC	F1	AUROC	AUPRC	F1
Zero-shot	70.81 $\pm$ 4.86	33.69 $\pm$ 6.88	41.42 $\pm$ 6.20	71.11 $\pm$ 5.47	32.86 $\pm$ 4.97	43.61 $\pm$ 6.33
Random	70.41 $\pm$ 4.69	31.32 $\pm$ 6.94	39.12 $\pm$ 5.61	68.46 $\pm$ 3.26	29.08 $\pm$ 5.53	39.08 $\pm$ 4.96
Semantic-emb	69.91 $\pm$ 5.23	25.78 $\pm$ 6.32	36.30 $\pm$ 6.09	70.91 $\pm$ 4.11	26.01 $\pm$ 5.37	36.92 $\pm$ 5.58
EHR-emb	81.59 $\pm$ 4.87	48.45 $\pm$ 6.96	46.07 $\pm$ 5.78	82.09 $\pm$ 7.81	47.98 $\pm$ 5.96	48.51 $\pm$ 6.45
GraphWalker	87.64 $\pm$ 5.14	62.31 $\pm$ 8.96	64.79 $\pm$ 7.74	88.72 $\pm$ 3.18	64.69 $\pm$ 2.43	67.11 $\pm$ 2.45

Appendix

Table 6: Results on MIMIC-III Mortality with GPT-5.2 as the black-box LLM, comparing two inference settings under the same few-shot demonstrations: Direct Answer (Few-shot) where the LLM directly outputs the prediction with no reasoning scaffold, and + ReAct where the LLM applies ReAct-style intermediate reasoning before generating the prediction. Bold = best within column, underline = second-best. Higher is better (

\uparrow

Split	MedQA	CMB	CMB-Clin
Train	10,178	15,465	83
Val	1,272	1,940	42
Test	1,273	1,935	83
Total	12,723	19,340	208

Appendix

Table 7: Dataset statistics for additional medical reasoning benchmarks: MedQA, CMB, and CMB-Clin.

Method	CMB	MedQA	CMB-clin
	EM	EM	BLEU-4	BLEU-1	ROUGE-L
Zero-shot	80.21 $\pm$ 0.92	56.56 $\pm$ 3.59	21.06 $\pm$ 1.37	59.85 $\pm$ 2.30	44.97 $\pm$ 1.73
Random	81.50 $\pm$ 2.78	52.96 $\pm$ 3.53	26.85 $\pm$ 1.44	66.05 $\pm$ 2.08	38.63 $\pm$ 1.77
Semantic-emb	80.52 $\pm$ 2.70	59.58 $\pm$ 3.52	26.86 $\pm$ 1.47	64.90 $\pm$ 2.01	41.26 $\pm$ 2.07
Time-series	78.05 $\pm$ 2.90	52.11 $\pm$ 3.55	29.05 $\pm$ 10.65	66.56 $\pm$ 2.24	40.12 $\pm$ 1.57
SPELL	81.98 $\pm$ 2.73	54.10 $\pm$ 3.49	28.41 $\pm$ 1.46	66.75 $\pm$ 2.10	36.82 $\pm$ 1.64

Appendix

Table 8: Performance comparison on CMB, MedQA, and CMB-clin using Qwen3-14B as the backbone LLM. CMB and MedQA are evaluated by exact match (EM), while CMB-clin is evaluated by BLEU-4, BLEU-1, and ROUGE-L. Higher values indicate better performance (

\uparrow

). We highlight the best and second best results.

Component	Content
Prompt	You will be provided with longitudinal electronic health record (EHR) data for a single patient. Each clinical feature is represented as a time-ordered list of measurements corresponding to the same hospital stay. Missing values are denoted as NaN . Patient Information: • Number of measurements: 19 • Measurement times (hours since admission): [4.68, 5.18, 6.18, 7.18, 8.18, …] Task Description: You are tasked with predicting in-hospital mortality based on patient EHR data. Instructions & Output Format: Provide only a floating-point number between 0 and 1 representing the predicted probability of mortality. Do not provide any reasoning or additional text. Output only the numerical value (e.g., 0.XX ).
Target Patient	Clinical Features Over Time: • Heart Rate (bpm): [117.0, 110.0, 111.0, 107.0, 102.0, …, 119.0, 122.0, 118.0] • Systolic BP (mmHg): [99.0, 96.0, 92.0, 109.0, 87.0, …, 127.0, 118.0, 120.0] • Diastolic BP (mmHg): [39.0, 56.0, 44.0, 54.0, 47.0, …, 87.0, 75.0, 67.0] • Mean BP (mmHg): [59.0, 69.3, 60.0, 72.3, 60.3, …, 100.3, 89.3, 84.7] • SpO 2 (%): [97.0, 94.0, 95.0, 95.0, 94.0, …, 96.0, 95.0, 95.0] • GCS (Total): [NaN, 15.0, 15.0, 15.0, 15.0, …, 15.0, 15.0, 15.0] $\vdots$
Ground Truth	0 (Survival).
Zero-shot Output	Prediction: 0.72 $\rightarrow$ Incorrect
GraphWalker Examples	Example 1 Label: 0 ; • Heart Rate: [78.0, 104.0, 81.0, 98.0, 74.0, …, 73.0, NaN, 74.0, 72.0] • Systolic BP: [119.0, 134.0, 111.0, 165.0, 149.0, …, 107.0, 96.0, 112.0, 118.0] • Diastolic BP: [62.0, 70.0, 79.0, 73.0, 66.0, …, 43.0, 56.0, 62.0] • SpO 2 : [98.0, 94.0, 95.0, 98.0, 96.0, …, 98.0, NaN, 97.0, 96.0] • GCS (Total): [NaN, 14.0, NaN, 14.0, NaN, …, 15.0, …] $\vdots$ Example 2 Label: 0 ; • Heart Rate: [87.0, 92.0, 92.0, 97.0, 90.0, …, 88.0, 77.0, 75.0, 74.0] • Systolic BP: [138.0, 155.0, 143.0, 152.0, 131.0, …, 136.0, 116.0, 133.0, 136.0] • Diastolic BP: [62.0, 78.0, 69.0, 79.0, 57.0, …, 59.0, 66.0, 61.0] • SpO 2 : [93.0, 93.0, 93.0, 95.0, 91.0, …, 93.0, 97.0, 98.0, 98.0] • GCS (Total): [NaN, 15.0, NaN, …, 15.0, …] $\vdots$ Example 3 Label: 0 ; • Heart Rate: [102.0, 105.0, 95.0, 91.0, 96.0, …, 77.0, 79.0, 80.0, 81.0] • Systolic BP: [122.0, 118.0, 115.0, 118.0, 106.0, …, 132.0, 118.0, 129.0, 130.0] • Diastolic BP: [62.0, 52.0, 65.0, 74.0, 63.0, …, 84.0, 76.0, 79.0, 85.0] • SpO 2 : [83.0, 95.0, 94.0, 95.0, 93.0, …, 93.0, 93.0, 92.0, 91.0] • GCS (Total): [NaN, NaN, 15.0, …, 15.0, …] $\vdots$
GraphWalker Output	Prediction: 0.0689 $\rightarrow$ Correct

Appendix

Table 9: Case study on MIMIC-III Mortality .

Symbol	Description
$\mathbf{X}_{i}$	EHR record of patient $i$
$\mathcal{M}_{\mathrm{exp}}$	Pretrained EHR encoder
$\mathbf{h}_{i}$	Patient embedding of patient $i$
$N$	Number of patients in the EHR base
$\mathcal{G}$	Population-level patient graph
$\mathcal{V}$	Set of patient nodes in the graph

Appendix

Table 10: Key Notations Used in GraphWalker

Case-Based Reasoning

See all 12 papers

Jun 1, 2026

ChatHealthAI: Aligning Electronic Health Record Representations with Large Language Models for Grounded Clinical Reasoning

Large language models (LLMs) exhibit strong natural-language reasoning abilities for clinical decision support, but struggle to effectively model structured longitudinal electronic health records (EHRs). In contrast, EHR foundation models can learn predictive patient representations, yet lack interpretable language-based reasoning. To bridge this gap, we propose ChatHealthAI, a multimodal reasoning framework that aligns structured EHR representations from a pretrained EHR foundation model with the semantic space of a frozen LLM through a task-aware resampler. By integrating longitudinal patient representations with refined clinical event descriptions, ChatHealthAI enables clinically grounded natural-language reasoning while maintaining accurate patient prediction. We evaluated ChatHealthAI on three clinical predictive tasks from the EHRSHOT benchmark. Results show that ChatHealthAI improves reasoning quality and interpretability while preserving competitive predictive performance. These findings highlight the potential of integrating EHR foundation models with pretrained LLMs for interpretable clinical prediction.

Bo-Hong Wang, Baicheng Peng, Ruilin Wang +3

School of Computer Science, McGill University, Montreal, QC, Canada · Mila - Quebec AI Institute, Montreal, QC, Canada

LLM Grounding LLM Alignment Clinical Outcome Prediction Clinical Reasoning Longitudinal EHR Modeling EHR Foundation Models

Ask PDF

Jun 22, 2026

EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning

Clinical agents promise to democratize access to electronic health records (EHRs), yet existing benchmarks fail to reflect the complexity of practical EHR analysis, e.g., often operating on idealized, clean EHRs via static SQL generation rather than interactive execution. In this work, we introduce EHR-Complex, a large-scale benchmark designed for interactive clinical database reasoning. Built on the large MIMIC-IV substrate (365K patients, 31 tables, 500M+ records), EHR-Complex comprises about 52K tasks spanning six clinical intents, supporting both patient-level and population-level queries, where each task requires an agent to interact with a sandboxed environment by executing SQL queries or Python code. Notably, EHR-Complex considers the real-world SQL task complexity for longitudinal multi-table aggregation and compositional reasoning, resulting in 31.93 SQL structural components per query on average. Evaluation results on EHR-Complex reveal the clinical difficulty of these EHR reasoning scenarios, with the top-performing model achieving only 62.3% exact-match accuracy. Pass^k consistency drops below 50% for nearly all evaluated models at k=4, exposing broad stochastic fragility. A fine-grained analysis of more than 3,800 failed trajectories for representative LLMs reveals three dominant failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings. EHR-Complex provides a rigorous testbed for clinical agents and highlights remaining gaps in robust reasoning for large-scale EHR analysis.

Yitong Qiao, Lei Liu, Yue Shen +4

1Zhejiang University · †Work done during an internship at Ant Group. · 2Ant Group

AI Agent Benchmarks Clinical Reasoning Electronic Health Records EHR QA

Ask PDF

Mar 27, 2026

Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

Clinicians often need to retrieve patient-specific information from electronic health records (EHRs), a task that is time-consuming and error-prone. We present a locally deployable Clinical Contextual Question Answering (CCQA) framework that answers clinical questions directly from EHRs without external data transfer. Open-source large language models (LLMs) ranging from 4B to 70B parameters were benchmarked under fully offline conditions using 1,664 expert-annotated question-answer pairs derived from records of 183 patients. The dataset consisted predominantly of Finnish clinical text. In free-text generation, Llama-3.1-70B achieved 95.3% accuracy and 97.3% consistency across semantically equivalent question variants, while the smaller Qwen3-30B-A3B-2507 model achieved comparable performance. In a multiple-choice setting, models showed similar accuracy but variable calibration. Low-precision quantization (4-bit and 8-bit) preserved predictive performance while reducing GPU memory requirements and improving deployment feasibility. Clinical evaluation identified clinically significant errors in 2.9% of outputs, and semantically equivalent questions occasionally yielded discordant responses, including instances where one formulation was correct and the other contained a clinically significant error (0.96% of cases). These findings demonstrate that locally hosted open-source LLMs can accurately retrieve patient-specific information from EHRs using natural-language queries, while highlighting the need for validation and human oversight in clinical deployment.

Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten +2

Department of Computer Science, Aalto University School of Science, Espoo, 02150, Finland. · Faculty of Medicine and Health Technology, Tampere University, Tampere, 33520, Finland. · Department of Oncology, TAYS Cancer Centrer, Tampere University Hospital, Tampere, 33520, Finland.

EHR QA Clinical QA Medical LLMs Clinical Language Model Evaluation

Ask PDF

Abstract

Case-Based Reasoning

Explore similar work

ChatHealthAI: Aligning Electronic Health Record Representations with Large Language Models for Grounded Clinical Reasoning

EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning

Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models