cs.CLOct 21, 2025

Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering

Authors: Feras AlMannaa, Talia Tseriotou, Jenny Chim, Maria Liakata

Organizations: Istanbul Aydın University · Queen Mary University of London · The Alan Turing Institute

Abstract

Despite the impressive reported performance of Large Language Models (LLMs) on medical Question-Answering (QA), evaluation still overrelies on multiple-choice and extractive settings primarily assessing textbook knowledge, with limited work on open-ended QA over patient records. We present a comprehensive study investigating LLM long-context (LC) comprehension on longitudinal, patient-centric Electronic Health Records (EHRs) across multiple datasets, models and metrics, covering extractive, multiple-choice, and open-ended generative QA under different context formulation and Retrieval Augmented Generation (RAG) settings. We show that medically fine-tuned models offer no measurable benefit over base counterparts, smaller models exhibit stronger memorization, and longitudinal supplementary notes aid generative QA. Reasoning models gain on complex multi-note QA under LLM Correctness but not stricter reference-aligned metrics, while hybrid RAG with reranking outperforms Full-Context (FC) prompting. A question-category analysis and multi-faceted metric evaluation further reveal where RAG and FC excel and struggle, with RAG falling behind in longitudinal reasoning, thus uncovering systematic failures pointing to the need for reasoning-appropriate metrics.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models

    Mar 27, 2026Mikko Saukkoriipi, Nicole Hernandez, Jaakko Sahlsten +2EHR QAClinical QA

  2. Can LLMs Reason Over Long Horizons? An Empirical Evaluation of Context Strategies for Longitudinal Clinical Reasoning

    Sep 30, 2026Taye Akinrele, Noorbakhsh Amiri Golilarz, Subash Neupane +2Long-Context Language Model EvaluationClinical Reasoning

  3. COTCAgent: Preventive Consultation via Probabilistic Chain-of-Thought Completion

    May 14, 2026Zihan Deng, Xiaozhen Zhong, Chuanzhi XuHealthcareClinical Decision Support