Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

Mikko Saukkoriipi; Nicole Hernandez; Jaakko Sahlsten; Kimmo Kaski; Otso Arponen

cs.CLMar 27, 2026

Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

Authors: Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten, Kimmo Kaski, Otso Arponen

Organizations: Department of Computer Science, Aalto University School of Science, Espoo, 02150, Finland. · Faculty of Medicine and Health Technology, Tampere University, Tampere, 33520, Finland. · Department of Oncology, TAYS Cancer Centrer, Tampere University Hospital, Tampere, 33520, Finland.

Abstract

Clinicians often need to retrieve patient-specific information from electronic health records (EHRs), a task that is time-consuming and error-prone. We present a locally deployable Clinical Contextual Question Answering (CCQA) framework that answers clinical questions directly from EHRs without external data transfer. Open-source large language models (LLMs) ranging from 4B to 70B parameters were benchmarked under fully offline conditions using 1,664 expert-annotated question-answer pairs derived from records of 183 patients. The dataset consisted predominantly of Finnish clinical text. In free-text generation, Llama-3.1-70B achieved 95.3% accuracy and 97.3% consistency across semantically equivalent question variants, while the smaller Qwen3-30B-A3B-2507 model achieved comparable performance. In a multiple-choice setting, models showed similar accuracy but variable calibration. Low-precision quantization (4-bit and 8-bit) preserved predictive performance while reducing GPU memory requirements and improving deployment feasibility. Clinical evaluation identified clinically significant errors in 2.9% of outputs, and semantically equivalent questions occasionally yielded discordant responses, including instances where one formulation was correct and the other contained a clinically significant error (0.96% of cases). These findings demonstrate that locally hosted open-source LLMs can accurately retrieve patient-specific information from EHRs using natural-language queries, while highlighting the need for validation and human oversight in clinical deployment.

Figures & tables

Figure 1: Visual abstract of the Clinical Contextual Question Answering (CCQA) pipeline. A clinician submits a natural language question about a patient record. A long-context large language model processes the complete electronic health record and generates a context-aware answer, which is returned to the clinician for use in clinical practice.

Developer	Model	Accuracy (%) $\uparrow$			VRAM (GB) $\downarrow$		Latency (s) $\downarrow$
Developer	Model	V1	V2	Cons.	Min	Max	Min	Max
Google	Gemma-3-4B-it GemmaTeam et al. (2025)	72.6	68.2	83.2	8.5	18.4	0.2	5.8
	Gemma-3-27B-it GemmaTeam et al. (2025)	92.7	92.8	96.3	52.4	71.9	0.7	16.6
	MedGemma-4B-it Sellergren et al. (2025)	74.5	73.1	89.6	8.5	18.4	0.3	5.9
	MedGemma-27B-text-it Sellergren et al. (2025)	92.1	90.0	94.0	51.6	71.2	0.7	16.7
DeepSeek	R1-Distill-Qwen-7B Guo et al. (2025)	50.4	51.2	43.4	14.7	22.6	5.7	41.4

Table 1: Clinical contextual question answering performance and deployment metrics. Accuracy is reported for two question variants (V1 and V2) and their consistency across variants. Peak VRAM and latency are reported as the minimum and maximum per-query values across the dataset. The best results per column are shown in bold. OOM indicates models exceeding the 160 GB memory limit. Models marked with † were evaluated on a reduced subset (

N=955

) due to context window constraints. All other models were evaluated on the full dataset (

N=1{,}664

Figure 2: Accuracy as a function of EHR length for the best-performing model from each model family. Results are shown across quartiles of EHR length (Q1–Q4), each containing 25% of the dataset. The x-axis represents the median EHR length (tokens) within each quartile, and the y-axis shows average accuracy. Accuracy values reported next to each model correspond to performance in the longest-length quartile (Q4).

Developer	Model	Accuracy (%) $\uparrow$			4-bit VRAM (GB) $\downarrow$
Developer	Model	bf16	4-bit	8-bit	Min	Max
Google	Gemma-3-4B-it GemmaTeam et al. (2025)	72.6	66.9	71.8	3.4	11.6
	Gemma-3-27B-it GemmaTeam et al. (2025)	92.7	92.5	92.6	16.7	36.0
	MedGemma-4B-it Sellergren et al. (2025)	74.5	74.0	74.3	3.4	11.6
	MedGemma-27B-text-it Sellergren et al. (2025)	92.1	93.0	92.2	16.5	35.8
DeepSeek	R1-Distill-Qwen-7B Guo et al. (2025)	50.4	46.3	49.8	5.6	10.2

Table 2: Quantization effects on clinical contextual question answering accuracy and memory usage. Accuracy is reported for the V1 formulation at three numerical precisions (bf16, 4-bit, and 8-bit). Peak 4-bit VRAM is reported as the minimum and maximum per-query values across the dataset. The best results per column are shown in bold. Models marked with † were evaluated on a reduced subset (

N=955

) due to context window constraints. All other models were evaluated on the full dataset (

N=1{,}664

Developer	Model	Acc. (%) $\uparrow$	ECE $\downarrow$	MCE $\downarrow$	Bias $\downarrow\left\|\cdot\right\|$
Google	Gemma-3-4B-it GemmaTeam et al. (2025)	72.5	0.238	0.397	0.114
	Gemma-3-27B-it GemmaTeam et al. (2025)	94.1	0.045	0.277	0.023
	MedGemma-4B-it Sellergren et al. (2025)	73.9	0.156	0.539	-0.007
	MedGemma-27B-text-it Sellergren et al. (2025)	93.7	0.193	0.707	-0.193
Meta	Llama-3.1-8B Grattafiori et al. (2024)	80.3	0.186	0.624	-0.186
	Llama-3.1-70B Grattafiori et al. (2024)	95.7	0.123	0.561	-0.123

Table 3: Calibration performance for clinical contextual question answering under multiple-choice selection. Accuracy is reported for the V1 question formulation. Calibration metrics include expected calibration error (ECE), maximum calibration error (MCE), and bias. The best results per column are shown in bold. Models marked with † were evaluated on a reduced subset (

N=955

) due to context window constraints. All other models were evaluated on the full dataset (

N=1{,}664

). DeepSeek R1 models were excluded because their architecture requires intermediate reasoning generation.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Developer	Model	1-shot		Context
Developer	Model	Acc.	$\boldsymbol{\Delta}$	Acc.	$\boldsymbol{\Delta}$
Google	Gemma-3-4B-it GemmaTeam et al. (2025)	69.5	$-3.1$	72.5	$-0.1$
	Gemma-3-27B-it GemmaTeam et al. (2025)	94.0	$+1.3$	94.1	$+1.4$
	MedGemma-4B-it Sellergren et al. (2025)	70.4	$-4.1$	73.9	$-0.6$
	MedGemma-27B-text-it Sellergren et al. (2025)	94.0	$+1.9$	93.7	$+1.6$
DeepSeek	R1-Distill-Qwen-7B Guo et al. (2025)	50.3	$-0.1$	N/A	N/A

Appendix

Table 4: Conditioning strategies for clinical contextual question answering. Accuracy is reported for models evaluated in BF16 precision under one-shot prompting and extended context conditioning.

\Delta

denotes absolute change in accuracy relative to baseline, expressed in percentage points. Models marked with † were not evaluated under these strategies because the additional prompt content exceeded their supported context window. All other models were evaluated on the full dataset (

N=1{,}664

Developer	Model	V1 Acc. (%)
Developer	Model	Q1	Q2	Q3	Q4
Google	Gemma-3-4B-it GemmaTeam et al. (2025)	67.9	73.1	73.0	76.2
	Gemma-3-27B-it GemmaTeam et al. (2025)	94.6	93.4	93.5	89.3
	MedGemma-4B-it Sellergren et al. (2025)	70.9	71.1	77.0	78.2
	MedGemma-27B-text-it Sellergren et al. (2025)	94.1	93.4	91.6	89.6
DeepSeek	R1-Distill-Qwen-7B Guo et al. (2025)	61.3	56.1	48.3	40.6

Appendix

Table 7: Clinical contextual question answering accuracy by electronic health record length quartile. Records were stratified into quartiles based on input length measured in tokens using the Meta Llama tokenizer as a fixed reference (Q1 shortest, Q4 longest; ranges: Q1 1,420–5,808, Q2 5,808–8,197, Q3 8,197–11,635, Q4 11,635–26,739 tokens). Token counts may vary slightly across models because of tokenizer differences. Values report accuracy for the primary question variant (V1). Results marked with † were evaluated only on records that fully fit within the supported context window (

\leq

8,192 tokens,

N=955

) and are not directly comparable across all quartiles. Bold indicates the highest accuracy among directly comparable models within each quartile.

Developer	Model	V1 accuracy	V2 accuracy	Consistency (V1–V2)
Developer	Model	% (95% CI)	% (95% CI)	% (95% CI)
Google	Gemma-3-4B-it GemmaTeam et al. (2025)	72.6 (70.4–74.7)	68.2 (65.9–70.4)	83.2 (81.3–85.0)
	Gemma-3-27B-it GemmaTeam et al. (2025)	92.7 (91.4–93.9)	92.8 (91.4–94.0)	96.3 (95.3–97.2)
	MedGemma-4B-it Sellergren et al. (2025)	74.5 (72.3–76.5)	73.1 (70.9–75.3)	89.6 (88.0–91.0)
	MedGemma-27B-text-it Sellergren et al. (2025)	92.1 (90.7–93.4)	90.0 (88.4–91.4)	94.0 (92.7–95.1)
DeepSeek	R1-Distill-Qwen-7B Guo et al. (2025)	50.4 (47.9–52.8)	51.2 (48.8–53.6)	43.4 (41.1–45.9)

Appendix

Table 8: Clinical contextual question answering accuracy with 95% confidence intervals. Accuracy is reported for Version 1 (V1) and Version 2 (V2) questions, as well as consistency across variants. Confidence intervals (CI) were computed using the exact Clopper–Pearson method. Bold indicates the best result. OOM indicates models that exceeded the available GPU memory and could not be evaluated. Models marked with † were evaluated on a reduced subset (

N=955

). All other models were evaluated on the full dataset (

N=1{,}664

Model A	Model B	$\boldsymbol{n_{10}}$	$\boldsymbol{n_{01}}$	$p$ -value
Llama-3.1-70B	Qwen3-30B-A3B-it-2507	35	31	0.843
Llama-3.1-70B	R1-Distill-Llama-70B	36	27	0.843
Llama-3.1-70B	Gemma-3-27B-it	70	27	$<0.001$
Llama-3.1-70B	MedGemma-27B-text-it	83	30	$<0.001$
Llama-3.1-70B	Llama-3.3-70B	19	12	0.843

Appendix

Table 9: Pairwise McNemar tests comparing the best-performing model with competing models. The contingency counts report the number of questions where Model A was correct and Model B incorrect (

n_{10}

), and where Model A was incorrect and Model B correct (

n_{01}

). The McNemar test was computed using the exact binomial formulation on paired predictions across

N=1{,}664

questions. Holm correction was applied for multiple comparisons.

Hugging Face Repository	Commit Hash
google/gemma-3-4b-it	093f9f388b31de276ce2de164bdc2081324b9767
google/gemma-3-27b-it	005ad3404e59d6023443cb575daa05336842228a
google/medgemma-4b-it	290cda5eeccbee130f987c4ad74a59ae6f196408
google/medgemma-27b-text-it	5b667cf2ddcf064085bc90952edb35a0edbfb79c
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B	916b56a44061fd5cd7d6a8fb632557ed4f724f60
deepseek-ai/DeepSeek-R1-Distill-Qwen-32B	711ad2ea6aa40cfca18895e8aca02ab92df1a746

Appendix

Table 10: Full reproducibility information for all instruction-tuned models evaluated in this study. Includes Hugging Face repository identifiers and exact commit hashes used for inference.

Figure 3: GPU memory usage as a function of input token length for bfloat16 precision. The dashed horizontal line indicates the 160 GB memory budget used in the main experiments.

Figure 4: GPU memory usage as a function of input token length for 8-bit quantization. The dashed horizontal line indicates the 160 GB memory budget used in the main experiments. Llama-4 and GPT-OSS models are not included because these architectures are not compatible with bitsandbytes quantization.

Figure 5: GPU memory usage as a function of input token length for 4-bit quantization. The dashed horizontal line indicates the 160 GB memory budget used in the main experiments. Llama-4 and GPT-OSS models are not included because these architectures are not compatible with bitsandbytes quantization.

EHR QA

See all 14 papers

FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records
Feb 26, 2026Michael Frew, Nishit Bheda, Bryan TrippEHR QA Clinical QA
University of British Columbia · University of Waterloo
HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering
Apr 29, 2026Md Biplob Hosen, Md Alomgeer Hussein, Md Akmol Masud +3LLM Grounding Healthcare
University of Maryland Baltimore County, Baltimore, MD, USA · 2Jahangirnagar University, Savar, Dhaka, Bangladesh
A Living Benchmark for Information Retrieval from Electronic Health Records
Sep 24, 2026Jordan L. Cahoon, Chloe O. Stanwyck, Sulaiman Somani +23Synthetic Benchmark Generation Clinical QA
Department of Biomedical Data Science, Stanford University, Stanford, CA · Department of Pathology, Stanford University, Stanford, CA · Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA +6

Feb 26, 2026J/K move · Enter open · S save

FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records

Michael Frew, Nishit Bheda, Bryan Tripp

University of British Columbia · University of Waterloo

EHR QA Clinical QA Electronic Health Records

Open paper AskPDF arXiv

Though patients are increasingly granted digital access to their electronic health records (EHRs), existing interfaces may not support precise, trustworthy answers to patient-specific questions. Large language models (LLM) show promise in clinical question answering (QA), but retrieval-based approaches are computationally inefficient, prone to hallucination, and difficult to deploy over real-life EHRs. This work introduces FHIRPath-QA, the first open dataset and benchmark for patient-specific QA that includes open-standard FHIRPath queries over real-world clinical data. A text-to-FHIRPath QA paradigm is proposed that shifts reasoning from free-text generation to FHIRPath query synthesis. For o4-mini, this reduced average token usage by 391x relative to retrieval-first prompting (629,829 vs 1,609 tokens per question) and lowered failure rates from 0.36 to 0.09 on clinician-phrased questions. Built on MIMIC-IV on FHIR Demo, the dataset pairs over 14k natural language questions in patient and clinician phrasing with validated FHIRPath queries and answers. Empirically, the evaluated LLMs achieve at most 42% accuracy, highlighting the challenge of the task, but benefit strongly from supervised fine-tuning, with query synthesis accuracy improving from 27% to 79% for 4o-mini. These results highlight that text-to-FHIRPath synthesis has the potential to serve as a practical foundation for safe, efficient, and interoperable consumer health applications, and the FHIRPath-QA dataset and benchmark serve as a starting point for future research on the topic. The full dataset and generation code can be accessed at: https://github.com/mooshifrew/fhirpath-qa.

Abstract

EHR QA

Explore similar work

FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records

HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering

A Living Benchmark for Information Retrieval from Electronic Health Records