FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records

Michael Frew; Nishit Bheda; Bryan Tripp

cs.CLFeb 26, 2026

FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records

Authors: Michael Frew, Nishit Bheda, Bryan Tripp

Organizations: University of British Columbia · University of Waterloo

Abstract

Though patients are increasingly granted digital access to their electronic health records (EHRs), existing interfaces may not support precise, trustworthy answers to patient-specific questions. Large language models (LLM) show promise in clinical question answering (QA), but retrieval-based approaches are computationally inefficient, prone to hallucination, and difficult to deploy over real-life EHRs. This work introduces FHIRPath-QA, the first open dataset and benchmark for patient-specific QA that includes open-standard FHIRPath queries over real-world clinical data. A text-to-FHIRPath QA paradigm is proposed that shifts reasoning from free-text generation to FHIRPath query synthesis. For o4-mini, this reduced average token usage by 391x relative to retrieval-first prompting (629,829 vs 1,609 tokens per question) and lowered failure rates from 0.36 to 0.09 on clinician-phrased questions. Built on MIMIC-IV on FHIR Demo, the dataset pairs over 14k natural language questions in patient and clinician phrasing with validated FHIRPath queries and answers. Empirically, the evaluated LLMs achieve at most 42% accuracy, highlighting the challenge of the task, but benefit strongly from supervised fine-tuning, with query synthesis accuracy improving from 27% to 79% for 4o-mini. These results highlight that text-to-FHIRPath synthesis has the potential to serve as a practical foundation for safe, efficient, and interoperable consumer health applications, and the FHIRPath-QA dataset and benchmark serve as a starting point for future research on the topic. The full dataset and generation code can be accessed at: https://github.com/mooshifrew/fhirpath-qa.

Figures & tables

Figure 1: Building FHIRPath-QA involved question selection, LLM-based paraphrasing, and query generation grounded in real patient data.

FHIR Resource	Clinical Topics
Patient	Demographics, birth date, gender
Encounter	Hospital visits, ICU stays, admission types, discharge times
Observation	Lab tests, vital signs, weight, intake events, output events, microbiology results
Condition	Diagnoses, clinical status
Medication (Med.)	Drug names
Med.Request	Prescriptions, dosages, drug delivery routes

Table 1: The FHIR resource types used in FHIRPath-QA cover a range of clinical topics required for inpatient care.

Figure 2: The query generated to answer the question Were any organisms found in my last SEROLOGY/BLOOD microbiology test this year? Colored regions indicate logical stages of the query: resource type selection (blue), specimen filtering (green), temporal constraints (orange), and selection of the most recent observation (purple). A final post-aggregation constraint checks that an organism was found (grey).

Split	Benchmark		Large
Split	Clinician	Patient	Clinician	Patient
Train	954 (45.5%)	675 (32.2%)	4721 (38.7%)	4775 (39.1%)
Val	104 (5.0%)	89 (4.2%)	535 (4.4%)	661 (5.4%)
Test	181 (8.6%)	92 (4.4%)	791 (6.5%)	717 (5.9%)
Total	1239 (59.1%)	856 (40.9%)	6047 (49.6%)	6153 (50.4%)
Combined	2095		12200

Table 2: Dataset Statistics. There are 100 distinct patient records and 61 question templates. The Benchmark contains one question, query, and executed answer per unique paraphrase. FHIRPath-QA Large contains multiple questions and queries per paraphrase, without executed answers. Samples are stratified by split and further categorized by perspective (Clinician vs. Patient). The bottom row displays the combined counts across both perspectives.

Strategy	Model	Tokens	SD
Retrieval	o4-mini	629,839	1,926,723
Retrieval	4.1-mini	577,281	1,889,508
Query-first	o4-mini	1,609	420
Query-first	4.1-mini	697	24

Table 3: The retrieval-first approach consumed a substantially higher number of tokens per question than the query-first approach.

Model	Accuracy		Failure Rate		Acc. Excl. Failures
	Clinical	Patient	Clinical	Patient	Clinical	Patient
a) Retrieval
o4-mini	0.33	0.33	0.36	0.34	0.51	0.49
4.1-mini	0.42	0.41	0.17	0.14	0.50	0.48
4o-mini	0.24	0.15	0.46	0.40	0.45	0.25
b) Query-first

Table 4: Overall accuracy slightly favors the retrieval-first approach, but this metric is influenced by its higher failure rate. When excluding failures, the retrieval-first approach consistently outperforms the query-first approach. Retrieval-first performs better on clinically phrased questions, whereas the query-first approach performs better on patient-style paraphrases.

Figure 3: SFT for the query-first approach improved executed answer accuracy on questions with paraphrased inputs over shared query structures (Test 1: Paraphrase) and on questions requiring novel query compositions over FHIR resource types seen during fine-tuning (Test 2: Query). In contrast, performance did not generalize to questions involving unseen resource types (Test 3: Resource). Increasing the number of patients in the fine-tuning dataset (10, 30, 100) led to progressively larger performance gains.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Base/FT	Accuracy		Failure Rate		Acc. Excl. Failures
		Clinical	Patient	Clinical	Patient	Clinical	Patient
a) Retrieval
4.1-mini	Base	0.42	0.41	0.17	0.14	0.50	0.48
4o-mini	Base	0.24	0.15	0.46	0.40	0.45	0.25
o4-mini	Base	0.33	0.33	0.36	0.34	0.51	0.49
b) Query-first

Appendix

Table A1: Comprehensive experimental results of retrieval and query-first strategies across clinical and patient perspectives for all models evaluated on the test split. FT-X indicates the fine-tuning dataset included X patients.

Model	Base/FT	Token Usage
		Tokens / Question	STD
a) Retrieval
4.1-mini	Base	577,281	1,889,508
4o-mini	Base	737,846	1,992,794
o4-mini	Base	629,839	1,926,723
b) Query-first

Appendix

Table A2: Token consumption per question on the test split across retrieval and query-first strategies for all models evaluated. FT-X indicates the fine-tuning dataset included X patients.

Model (Train Size)	Query Match Rate	Query Match Rate with Masked Numeric Quantities
4.1-mini (Base)	0.01	0.01
4.1-mini (10)	0.31	0.58
4.1-mini (30)	0.35	0.63
4.1-mini (100)	0.34	0.63
4.1-nano (Base)	0.00	0.01
4.1-nano (10)	0.07	0.39

Appendix

Table A3: The query match rate with masked numeric quantities consistently increased after fine-tuning, demonstrating that a high number of query inaccuracies were due to incorrect date filtering in otherwise correct queries. This metric was computed by masking all numeric substrings (i.e. dates, patient ids, dose quantities) in the generated and ground truth query before comparing strings.

Figure B1: Patient record size vs token usage (log scale) using (o4-mini). The upper bound on tokens used in the retrieval-first approach scales linearly with the patient EHR size, whereas query-first remains constant. Retrieval tokens corresponds to total tokens or attempted tokens if the context length limit was exceeded.

Figure B2: Histograms of a) the minimum normalized Levenshtein distance between paraphrase and each other paraphrase of the same question template and b) the semantic similarity of paraphrase to original question template by perspective.

EHR QA

See all 14 papers

Mar 27, 2026

Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

Clinicians often need to retrieve patient-specific information from electronic health records (EHRs), a task that is time-consuming and error-prone. We present a locally deployable Clinical Contextual Question Answering (CCQA) framework that answers clinical questions directly from EHRs without external data transfer. Open-source large language models (LLMs) ranging from 4B to 70B parameters were benchmarked under fully offline conditions using 1,664 expert-annotated question-answer pairs derived from records of 183 patients. The dataset consisted predominantly of Finnish clinical text. In free-text generation, Llama-3.1-70B achieved 95.3% accuracy and 97.3% consistency across semantically equivalent question variants, while the smaller Qwen3-30B-A3B-2507 model achieved comparable performance. In a multiple-choice setting, models showed similar accuracy but variable calibration. Low-precision quantization (4-bit and 8-bit) preserved predictive performance while reducing GPU memory requirements and improving deployment feasibility. Clinical evaluation identified clinically significant errors in 2.9% of outputs, and semantically equivalent questions occasionally yielded discordant responses, including instances where one formulation was correct and the other contained a clinically significant error (0.96% of cases). These findings demonstrate that locally hosted open-source LLMs can accurately retrieve patient-specific information from EHRs using natural-language queries, while highlighting the need for validation and human oversight in clinical deployment.

Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten +2

Department of Computer Science, Aalto University School of Science, Espoo, 02150, Finland. · Faculty of Medicine and Health Technology, Tampere University, Tampere, 33520, Finland. · Department of Oncology, TAYS Cancer Centrer, Tampere University Hospital, Tampere, 33520, Finland.

EHR QA Clinical QA Medical LLMs Clinical Language Model Evaluation

Ask PDF

Apr 29, 2026

HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering

Patient portals now give individuals direct access to their electronic health records (EHRs), yet access alone does not ensure patients understand or act on the complex clinical information contained in these records. The ArchEHR-QA 2026 shared task addresses this challenge by focusing on grounded question answering over EHRs, and this paper presents the system developed by the HealthNLP_Retrievers team for this task. The proposed approach uses a multi-stage cascaded pipeline powered by the Gemini 2.5 Pro large language model to interpret patient-authored questions and retrieve relevant evidence from lengthy clinical notes. Our architecture comprises four integrated modules: (1) a few-shot query reformulation unit which summarizes verbose patient queries; (2) a heuristic-based evidence scorer which ranks clinical sentences to prioritize recall; (3) a grounded response generator which synthesizes professional-caliber answers restricted strictly to identified evidence; and (4) a high-precision many-to-many alignment framework which links generated answers to supporting clinical sentences. This cascaded approach achieved competitive results. Across the individual tracks, the system ranked 1st in question interpretation, 5th in answer generation, 7th in evidence identification, and 9th in answer-evidence alignment. These results show that integrating large language models within a structured multi-stage pipeline improves grounding, precision, and the professional quality of patient-oriented health communication. To support reproducibility, our source code is publicly available in our GitHub repository

Md Biplob Hosen, Md Alomgeer Hussein, Md Akmol Masud +3

University of Maryland Baltimore County, Baltimore, MD, USA · 2Jahangirnagar University, Savar, Dhaka, Bangladesh

LLM Grounding Healthcare Clinical QA Evidence-Grounded Generation EHR QA Evidence Retrieval

Ask PDF

May 11, 2026

Neural at ArchEHR-QA 2026: One Method Fits All: Unified Prompt Optimization for Clinical QA over EHRs

Automated question answering (QA) over electronic health records (EHRs) demands precise evidence retrieval, faithful answer generation, and explicit grounding of answers in clinical notes. In this work, we present Neural1.5, our method for the ArchEHR-QA 2026 shared task at CL4Health@LREC 2026, which comprises four subtasks: question interpretation, evidence identification, answer generation, and evidence alignment. Our approach decouples the task into independent, modular stages and employs DSPy"s MIPROv2 optimizer to automatically discover high-performing prompts, jointly tuning instructions and few-shot demonstrations for each stage. Within every stage, self-consistency voting over multiple stochastic inference runs suppresses spurious errors and improves reliability, while stage-specific verification mechanisms (e.g., self-reflection and chain-of-verification for alignment) further refine output quality. Among all teams that participated in all four subtasks, our method ranks second overall (mean rank 4.00), placing 4th, 1st, 4th, and 7th on Subtasks 1-4, respectively. These results demonstrate that systematic, per-stage prompt optimization combined with self-consistency mechanisms is a cost-effective alternative to model fine-tuning for multifaceted clinical QA.

Abrar Majeedi, Viswanatha Reddy Gajjala, Sai Prasanna Teja Reddy Bogireddy +1

University of Wisconsin–Madison · University of Chicago · Independent Researcher

Healthcare Clinical QA Evidence-Grounded Generation Electronic Health Records EHR QA

Ask PDF

Abstract

EHR QA

Explore similar work

Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering

Neural at ArchEHR-QA 2026: One Method Fits All: Unified Prompt Optimization for Clinical QA over EHRs