Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering
Organizations: Istanbul Aydın University · Queen Mary University of London · The Alan Turing Institute
Abstract
Despite the impressive reported performance of Large Language Models (LLMs) on medical Question-Answering (QA), evaluation still overrelies on multiple-choice and extractive settings primarily assessing textbook knowledge, with limited work on open-ended QA over patient records. We present a comprehensive study investigating LLM long-context (LC) comprehension on longitudinal, patient-centric Electronic Health Records (EHRs) across multiple datasets, models and metrics, covering extractive, multiple-choice, and open-ended generative QA under different context formulation and Retrieval Augmented Generation (RAG) settings. We show that medically fine-tuned models offer no measurable benefit over base counterparts, smaller models exhibit stronger memorization, and longitudinal supplementary notes aid generative QA. Reasoning models gain on complex multi-note QA under LLM Correctness but not stricter reference-aligned metrics, while hybrid RAG with reranking outperforms Full-Context (FC) prompting. A question-category analysis and multi-faceted metric evaluation further reveal where RAG and FC excel and struggle, with RAG falling behind in longitudinal reasoning, thus uncovering systematic failures pointing to the need for reasoning-appropriate metrics.
Figures & tables
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Source |
|---|---|
| MedQA | Jin et al. (2021) |
| MedQuAD | Abacha and Demner-Fushman (2019) |
| MashQA | Zhu et al. (2020) |
| MedOdyssey | Fan et al. (2025a) |
| BioMRC | Pappas et al. (2020) |
| BioASQ | Nentidis et al. (2025) |
| Setting | Token Range | LLM Corr. | LLM Comp. | LLM Faith. | NLI Entail. | NLI Contra. | Bio Clin. F1 | METEOR |
|---|---|---|---|---|---|---|---|---|
| RAG 512 | 8-16K | 41.52 | 35.27 | 40.18 | 50.50 | 18.55 | 77.69 | 35.01 |
| RAG 1K | 38.84 | 33.04 | 35.71 | 36.74 | 19.72 | 76.90 | 32.65 | |
| RAG Full Note | 39.73 | 28.57 | 36.61 | 43.71 | 19.84 | 75.49 | 33.31 | |
| RAG 512 | 16-32K | 60.85 | 54.72 | 51.42 | 55.75 | 17.37 | 80.49 | 45.11 |
| RAG 1K | 53.77 | 47.17 | 50.94 | 61.68 | 13.65 | 79.44 | 41.87 | |
| RAG Full Note | 55.19 | 47.64 | 45.28 | 60.17 | 13.66 | 78.60 | 41.80 |
Clinical QA
Clinical question answering uses AI to answer medical and patient-specific questions from clinical knowledge, records, guidelines, and evidence.
See all 65 papers