sebis at ArchEHR-QA 2026: How Much Can You Do Locally? Evaluating Grounded EHR QA on a Single Notebook

Ibrahim Ebrar Yurt; Fabian Karl; Tejaswi Choppa; Florian Matthes

cs.CLMar 14, 2026

sebis at ArchEHR-QA 2026: How Much Can You Do Locally? Evaluating Grounded EHR QA on a Single Notebook

Authors: Ibrahim Ebrar Yurt, Fabian Karl, Tejaswi Choppa, Florian Matthes

Organizations: Technical University of Munich, School of Computation, Information and Technology, Germany

Abstract

Clinical question answering over electronic health records (EHRs) can help clinicians and patients access relevant medical information more efficiently. However, many recent approaches rely on large cloud-based models, which are difficult to deploy in clinical environments due to privacy constraints and computational requirements. In this work, we investigate how far grounded EHR question answering can be pushed when restricted to a single notebook. We participate in all four subtasks of the ArchEHR-QA 2026 shared task and evaluate several approaches designed to run on commodity hardware. All experiments are conducted locally without external APIs or cloud infrastructure. Our results show that such systems can achieve competitive performance on the shared task leaderboards. In particular, our submissions perform above average in two subtasks, and we observe that smaller models can approach the performance of much larger systems when properly configured. These findings suggest that privacy-preserving EHR QA systems running fully locally are feasible with current models and commodity hardware. The source code is available at https://github.com/ibrahimey/ArchEHR-QA-2026.

Figures & tables

Figure 1: Overall scores for different threshold values for both methods on the development set for Subtask 2. Dashed red line shows the naive baseline of classifying everything as relevant.

Figure 2: Overall scores for different threshold values on the development set for Subtask 4. Dashed red line shows the naive baseline of classifying everything as relevant.

Statistic	Development	Test	Test-2026
Cases	20	100	47
Note sentences	428	2599	1596
Average note sentences per case	21.40	25.99	33.96
Average note sentence length	15.00	14.64	14.64
Average note length per case	320.90	380.42	497.26
Average patient question length	16.20	14.64	14.72

Table 1: Descriptive statistics of the ArchEHR-QA dataset across the development and two test splits. All text length measurements are reported as word counts.

Model	#Parameter
Autoregessive Generative Models
mlx-community/gpt-oss-120b-MXFP4-Q4	117B total / 5.1B active
RepublicOfKorokke/Qwen3.5-35B-A3B-mlx-lm-mxfp4	35B total / 3B active
Qwen/Qwen2.5-14B-Instruct	14.7B
Qwen/Qwen3-4B-Instruct-2507	4B
Retrieval Models

Table 2: Evaluated models categorized by architecture type and sorted by decreasing parameter count. All models are obtained from Hugging Face (B: billions, M: millions).

Task	Model	LR	Epochs	Dropout
Subtask 2	Bio_ClinicalBERT	$2\times 10^{-5}$	1	0.1
Subtask 2	DeBERTa-base	$2\times 10^{-5}$	1	0.1
Subtask 4	Bio_ClinicalBERT	$3\times 10^{-5}$	1	0.1
Subtask 4	DeBERTa-base	$2\times 10^{-5}$	1	0.1

Table 3: Optimized hyperparameters for the BERT-style classifiers in Subtask 2 and Subtask 4. Values are selected via a grid search evaluated with 5-fold cross-validation on the development set.

Metric	SQ	DQ	TS
Dev Set
ROUGE-Lsum	23.51	24.24	28.74
BERTScore	41.15	43.79	41.94
Test Set
ROUGE-Lsum	22.92	23.09	21.19
BERTScore	36.93	37.29	33.10

Table 4: Test and development set results for Subtask 1. Only ROUGE and BERTScore are used on the development set. SQ: Single Query, DQ: Double Query, TS: Two Step.

	Dev Set			Test Set
Setup	Strict P	Strict R	Strict F1	Strict P	Strict R	Strict F1
Baseline
Everything is relevant	28.27	100.00	44.08	–	–	–
Similarity Threshold
Qwen3-Embedding-8B	48.55	55.37	51.74	37.92	80.74	51.61
MedCPT-Cross-Encoder	34.04	79.34	47.64	–	–	–

Table 5: Subtask 2 results. P and R stand for Precision and Recall, respectively. “Synth” indicates training with synthetic data, while “HYDRA” denotes multiple heads. Development set results for fine-tuned models are computed from out-of-fold predictions. Bold values mark the highest overall F1 score, and underlined values show the best F1 within each method category on the development set.

Metric	Dev Set		Test Set
	Qwen	gpt	gpt
Overall Score	27.76	32.38	31.51
BLEU	1.19	4.34	3.82
ROUGE-Lsum	15.73	21.34	22.04
SARI	49.75	58.28	56.55
BERTScore	30.95	39.66	39.30

Table 6: Evaluation results for Subtask 3 on the development and test sets. For the development set MEDCON score was not calculated and therefore was also not included in the overall score, which is the mean of all other scores.

	Dev Set			Test Set
Model	Micro-P	Micro-R	Micro-F1	Micro-P	Micro-R	Micro-F1
Baseline
Everything is relevant	8.16	75.11	14.73	–	–	–
Similarity Threshold
Qwen3-Embedding-8B	22.82	21.27	22.01	52.41	68.69	59.45
Prompting

Table 7: Subtask 4 results. Micro-Precision (P), Micro-Recall (R), and Micro-F1 are reported. “Synth” denotes training augmented with synthetic data. Development set results for fine-tuned models are derived from merged out-of-fold predictions. Bold values denote the highest overall F1 score, and underlined values indicate the best F1 score within each method category on the development set.

EHR QA

See all 14 papers

HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering
Apr 29, 2026Md Biplob Hosen, Md Alomgeer Hussein, Md Akmol Masud +3LLM Grounding Healthcare
University of Maryland Baltimore County, Baltimore, MD, USA · 2Jahangirnagar University, Savar, Dhaka, Bangladesh
Neural at ArchEHR-QA 2026: One Method Fits All: Unified Prompt Optimization for Clinical QA over EHRs
May 11, 2026Abrar Majeedi, Viswanatha Reddy Gajjala, Sai Prasanna Teja Reddy Bogireddy +1Healthcare Clinical QA
University of Wisconsin–Madison · University of Chicago · Independent Researcher
Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models
Mar 27, 2026Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten +2EHR QA Clinical QA
Department of Computer Science, Aalto University School of Science, Espoo, 02150, Finland. · Faculty of Medicine and Health Technology, Tampere University, Tampere, 33520, Finland. · Department of Oncology, TAYS Cancer Centrer, Tampere University Hospital, Tampere, 33520, Finland.

Apr 29, 2026J/K move · Enter open · S save

HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering

Md Biplob Hosen, Md Alomgeer Hussein, Md Akmol Masud, Omar Faruque+2

University of Maryland Baltimore County, Baltimore, MD, USA · 2Jahangirnagar University, Savar, Dhaka, Bangladesh

LLM Grounding Healthcare Clinical QA Evidence-Grounded Generation EHR QA Evidence Retrieval

Open paper AskPDF arXiv

Patient portals now give individuals direct access to their electronic health records (EHRs), yet access alone does not ensure patients understand or act on the complex clinical information contained in these records. The ArchEHR-QA 2026 shared task addresses this challenge by focusing on grounded question answering over EHRs, and this paper presents the system developed by the HealthNLP_Retrievers team for this task. The proposed approach uses a multi-stage cascaded pipeline powered by the Gemini 2.5 Pro large language model to interpret patient-authored questions and retrieve relevant evidence from lengthy clinical notes. Our architecture comprises four integrated modules: (1) a few-shot query reformulation unit which summarizes verbose patient queries; (2) a heuristic-based evidence scorer which ranks clinical sentences to prioritize recall; (3) a grounded response generator which synthesizes professional-caliber answers restricted strictly to identified evidence; and (4) a high-precision many-to-many alignment framework which links generated answers to supporting clinical sentences. This cascaded approach achieved competitive results. Across the individual tracks, the system ranked 1st in question interpretation, 5th in answer generation, 7th in evidence identification, and 9th in answer-evidence alignment. These results show that integrating large language models within a structured multi-stage pipeline improves grounding, precision, and the professional quality of patient-oriented health communication. To support reproducibility, our source code is publicly available in our GitHub repository

Abstract

EHR QA

Explore similar work

HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering

Neural at ArchEHR-QA 2026: One Method Fits All: Unified Prompt Optimization for Clinical QA over EHRs

Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models