FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records

Michael Frew; Nishit Bheda; Bryan Tripp

cs.CLFeb 26, 2026

FHIRPath-QA: Executable Question Answering over FHIR Electronic Health Records

Authors: Michael Frew, Nishit Bheda, Bryan Tripp

Organizations: University of British Columbia · University of Waterloo

Abstract

Though patients are increasingly granted digital access to their electronic health records (EHRs), existing interfaces may not support precise, trustworthy answers to patient-specific questions. Large language models (LLM) show promise in clinical question answering (QA), but retrieval-based approaches are computationally inefficient, prone to hallucination, and difficult to deploy over real-life EHRs. This work introduces FHIRPath-QA, the first open dataset and benchmark for patient-specific QA that includes open-standard FHIRPath queries over real-world clinical data. A text-to-FHIRPath QA paradigm is proposed that shifts reasoning from free-text generation to FHIRPath query synthesis. For o4-mini, this reduced average token usage by 391x relative to retrieval-first prompting (629,829 vs 1,609 tokens per question) and lowered failure rates from 0.36 to 0.09 on clinician-phrased questions. Built on MIMIC-IV on FHIR Demo, the dataset pairs over 14k natural language questions in patient and clinician phrasing with validated FHIRPath queries and answers. Empirically, the evaluated LLMs achieve at most 42% accuracy, highlighting the challenge of the task, but benefit strongly from supervised fine-tuning, with query synthesis accuracy improving from 27% to 79% for 4o-mini. These results highlight that text-to-FHIRPath synthesis has the potential to serve as a practical foundation for safe, efficient, and interoperable consumer health applications, and the FHIRPath-QA dataset and benchmark serve as a starting point for future research on the topic. The full dataset and generation code can be accessed at: https://github.com/mooshifrew/fhirpath-qa.

Figures & tables

Figure 1: Building FHIRPath-QA involved question selection, LLM-based paraphrasing, and query generation grounded in real patient data.

FHIR Resource	Clinical Topics
Patient	Demographics, birth date, gender
Encounter	Hospital visits, ICU stays, admission types, discharge times
Observation	Lab tests, vital signs, weight, intake events, output events, microbiology results
Condition	Diagnoses, clinical status
Medication (Med.)	Drug names
Med.Request	Prescriptions, dosages, drug delivery routes

Table 1: The FHIR resource types used in FHIRPath-QA cover a range of clinical topics required for inpatient care.

Figure 2: The query generated to answer the question Were any organisms found in my last SEROLOGY/BLOOD microbiology test this year? Colored regions indicate logical stages of the query: resource type selection (blue), specimen filtering (green), temporal constraints (orange), and selection of the most recent observation (purple). A final post-aggregation constraint checks that an organism was found (grey).

Split	Benchmark		Large
Split	Clinician	Patient	Clinician	Patient
Train	954 (45.5%)	675 (32.2%)	4721 (38.7%)	4775 (39.1%)
Val	104 (5.0%)	89 (4.2%)	535 (4.4%)	661 (5.4%)
Test	181 (8.6%)	92 (4.4%)	791 (6.5%)	717 (5.9%)
Total	1239 (59.1%)	856 (40.9%)	6047 (49.6%)	6153 (50.4%)
Combined	2095		12200

Table 2: Dataset Statistics. There are 100 distinct patient records and 61 question templates. The Benchmark contains one question, query, and executed answer per unique paraphrase. FHIRPath-QA Large contains multiple questions and queries per paraphrase, without executed answers. Samples are stratified by split and further categorized by perspective (Clinician vs. Patient). The bottom row displays the combined counts across both perspectives.

Strategy	Model	Tokens	SD
Retrieval	o4-mini	629,839	1,926,723
Retrieval	4.1-mini	577,281	1,889,508
Query-first	o4-mini	1,609	420
Query-first	4.1-mini	697	24

Table 3: The retrieval-first approach consumed a substantially higher number of tokens per question than the query-first approach.

Model	Accuracy		Failure Rate		Acc. Excl. Failures
	Clinical	Patient	Clinical	Patient	Clinical	Patient
a) Retrieval
o4-mini	0.33	0.33	0.36	0.34	0.51	0.49
4.1-mini	0.42	0.41	0.17	0.14	0.50	0.48
4o-mini	0.24	0.15	0.46	0.40	0.45	0.25
b) Query-first

Table 4: Overall accuracy slightly favors the retrieval-first approach, but this metric is influenced by its higher failure rate. When excluding failures, the retrieval-first approach consistently outperforms the query-first approach. Retrieval-first performs better on clinically phrased questions, whereas the query-first approach performs better on patient-style paraphrases.

Figure 3: SFT for the query-first approach improved executed answer accuracy on questions with paraphrased inputs over shared query structures (Test 1: Paraphrase) and on questions requiring novel query compositions over FHIR resource types seen during fine-tuning (Test 2: Query). In contrast, performance did not generalize to questions involving unseen resource types (Test 3: Resource). Increasing the number of patients in the fine-tuning dataset (10, 30, 100) led to progressively larger performance gains.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Base/FT	Accuracy		Failure Rate		Acc. Excl. Failures
		Clinical	Patient	Clinical	Patient	Clinical	Patient
a) Retrieval
4.1-mini	Base	0.42	0.41	0.17	0.14	0.50	0.48
4o-mini	Base	0.24	0.15	0.46	0.40	0.45	0.25
o4-mini	Base	0.33	0.33	0.36	0.34	0.51	0.49
b) Query-first

Appendix

Table A1: Comprehensive experimental results of retrieval and query-first strategies across clinical and patient perspectives for all models evaluated on the test split. FT-X indicates the fine-tuning dataset included X patients.

Model	Base/FT	Token Usage
		Tokens / Question	STD
a) Retrieval
4.1-mini	Base	577,281	1,889,508
4o-mini	Base	737,846	1,992,794
o4-mini	Base	629,839	1,926,723
b) Query-first

Appendix

Table A2: Token consumption per question on the test split across retrieval and query-first strategies for all models evaluated. FT-X indicates the fine-tuning dataset included X patients.

Model (Train Size)	Query Match Rate	Query Match Rate with Masked Numeric Quantities
4.1-mini (Base)	0.01	0.01
4.1-mini (10)	0.31	0.58
4.1-mini (30)	0.35	0.63
4.1-mini (100)	0.34	0.63
4.1-nano (Base)	0.00	0.01
4.1-nano (10)	0.07	0.39

Appendix

Table A3: The query match rate with masked numeric quantities consistently increased after fine-tuning, demonstrating that a high number of query inaccuracies were due to incorrect date filtering in otherwise correct queries. This metric was computed by masking all numeric substrings (i.e. dates, patient ids, dose quantities) in the generated and ground truth query before comparing strings.

Figure B1: Patient record size vs token usage (log scale) using (o4-mini). The upper bound on tokens used in the retrieval-first approach scales linearly with the patient EHR size, whereas query-first remains constant. Retrieval tokens corresponds to total tokens or attempted tokens if the context length limit was exceeded.

Figure B2: Histograms of a) the minimum normalized Levenshtein distance between paraphrase and each other paraphrase of the same question template and b) the semantic similarity of paraphrase to original question template by perspective.

EHR QA

See all 14 papers

Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models
Mar 27, 2026Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten +2EHR QA Clinical QA
Department of Computer Science, Aalto University School of Science, Espoo, 02150, Finland. · Faculty of Medicine and Health Technology, Tampere University, Tampere, 33520, Finland. · Department of Oncology, TAYS Cancer Centrer, Tampere University Hospital, Tampere, 33520, Finland.
HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering
Apr 29, 2026Md Biplob Hosen, Md Alomgeer Hussein, Md Akmol Masud +3LLM Grounding Healthcare
University of Maryland Baltimore County, Baltimore, MD, USA · 2Jahangirnagar University, Savar, Dhaka, Bangladesh
Neural at ArchEHR-QA 2026: One Method Fits All: Unified Prompt Optimization for Clinical QA over EHRs
May 11, 2026Abrar Majeedi, Viswanatha Reddy Gajjala, Sai Prasanna Teja Reddy Bogireddy +1Healthcare Clinical QA
University of Wisconsin–Madison · University of Chicago · Independent Researcher

Mar 27, 2026J/K move · Enter open · S save

Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten, Kimmo Kaski+1

Department of Computer Science, Aalto University School of Science, Espoo, 02150, Finland. · Faculty of Medicine and Health Technology, Tampere University, Tampere, 33520, Finland. · Department of Oncology, TAYS Cancer Centrer, Tampere University Hospital, Tampere, 33520, Finland.

EHR QA Clinical QA Medical LLMs Clinical Language Model Evaluation

Open paper AskPDF arXiv

Clinicians often need to retrieve patient-specific information from electronic health records (EHRs), a task that is time-consuming and error-prone. We present a locally deployable Clinical Contextual Question Answering (CCQA) framework that answers clinical questions directly from EHRs without external data transfer. Open-source large language models (LLMs) ranging from 4B to 70B parameters were benchmarked under fully offline conditions using 1,664 expert-annotated question-answer pairs derived from records of 183 patients. The dataset consisted predominantly of Finnish clinical text. In free-text generation, Llama-3.1-70B achieved 95.3% accuracy and 97.3% consistency across semantically equivalent question variants, while the smaller Qwen3-30B-A3B-2507 model achieved comparable performance. In a multiple-choice setting, models showed similar accuracy but variable calibration. Low-precision quantization (4-bit and 8-bit) preserved predictive performance while reducing GPU memory requirements and improving deployment feasibility. Clinical evaluation identified clinically significant errors in 2.9% of outputs, and semantically equivalent questions occasionally yielded discordant responses, including instances where one formulation was correct and the other contained a clinically significant error (0.96% of cases). These findings demonstrate that locally hosted open-source LLMs can accurately retrieve patient-specific information from EHRs using natural-language queries, while highlighting the need for validation and human oversight in clinical deployment.

Abstract

EHR QA

Explore similar work

Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

HealthNLP_Retrievers at ArchEHR-QA 2026: Cascaded LLM Pipeline for Grounded Clinical Question Answering

Neural at ArchEHR-QA 2026: One Method Fits All: Unified Prompt Optimization for Clinical QA over EHRs