cs.CLOct 1, 2026

A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined

Authors: Zhangshu Joshua Jiang, Zina Ibrahim, James T. Teo

Organizations: DRIVE-Health CDT Department of Biostatistics and Health Informatics Institute of Psychiatry, Psychology and Neuroscience King’s College London London, UK

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Exam-style accuracy does not establish whether large language models (LLMs) reason well over clinical records. We define clinical reasoning as integrating and updating evidence across time and sources to form, revise and justify a patient's problem representation and a defensible plan. This structured narrative review maps three literatures: medical education assessment instruments, clinical LLM benchmarks published from 2023 onwards, and general-domain methods for evaluating long-form generation. We examine six dimensions: problem representation, temporal synthesis, differential and management reasoning, counterfactual reasoning, calibrated uncertainty, and reasoning faithfulness. Preprints are included and flagged. No single instrument covers all six dimensions. Problem representation and differential or management reasoning are reasonably covered, although reliability varies by instrument and setting. TIMER-Eval targets temporal synthesis, and ER-Reason assesses sequential diagnostic belief updating. Dedicated uncertainty and counterfactual evaluations are emerging, but their applicability to longitudinal free-text reasoning remains limited. Factual completeness is well theorised in general-domain evaluation, with early clinical evidence of important omissions. Faithfulness remains the weakest dimension, with one identified clinical causal-ablation study on multiple-choice questions. Existing tools should be combined through binary rubric items, separate completeness and correctness scores, case-specific importance weighting with non-compensable safety caps, temporal order-consistency checks, and chance-corrected reliability reporting. Further design work is needed for calibrated uncertainty, counterfactual reasoning and faithfulness over longitudinal free-text records. This review provides a design rationale, not a validated instrument.

Figures & tables

DimensionBest-available instrument(s)What is measuredWhat remains open
Problem representationIDEA, R-IDEA, ART ( Baker et al., 2015 ; Schaye et al., 2022 ; Thammasitboon et al., 2018 )Structured write-up of interpretive summary and differential in clinical notesReliability varies by instrument and setting; transfer to LLM outputs requires validation; documentation quality does not establish causal faithfulness ( Baker et al., 2015 ; Schaye et al., 2022 )
Differential and management reasoningHealthBench, MedR-Bench, ER-Reason, Brodeur et al. ( Arora et al., 2025 ; Qiu et al., 2025 ; Mehandru et al., 2025 ; Brodeur et al., 2026 )Quality, safety and completeness of clinically relevant answers; correspondence to reference reasoningCorrespondence to reference text is not causal reasoning; text-only input, no bedside cues ( Brodeur et al., 2026 )
Temporal / longitudinal synthesisTIMER-Eval ( Cui et al., 2025 )Respect for temporal boundaries, trend identification, chronological orderDoes not test whether a later record supersedes, rather than supplements, an earlier one
Counterfactual reasoningMamaBench ( Adewuyi et al., 2026 )Bias Trap Rate on paired vignettes where one parameter changes the correct diagnosisSingle specialty; paired vignettes, not longitudinal free text
Calibrated uncertaintyDu et al. uncertainty-preservation benchmark, SCT-Bench ( Du et al., 2026 ; McCoy et al., 2025 )Preservation of five-level uncertainty labels; belief-shift patterns against expert panelsNo rubric for uncertainty that must change as records accumulate
Reasoning faithfulnessAfolabi et al. causal-ablation probe ( Afolabi et al., 2026 )Causal necessity of individual chain-of-thought steps via redactionWorkshop-scale study on multiple-choice only; no free-text, longitudinal, rubric-integrated version
Table 1: Coverage of the six clinical-reasoning dimensions by existing instruments and benchmarks.

Explore similar work

CardsList
Sep 29, 2026cs.CL

A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses

We propose a rubric for assessing expressed clinical reasoning in model responses, drawing on three bodies of work: medical education assessment frameworks (ART, SCT, Key Feature Problems and OSCE); clinical LLM benchmarks (MedR-Bench, HealthBench, TIMER-Bench, DR. BENCH, PrIME-LLM and PatientSafeBench); and general LLM reasoning evaluation research, including the Factuality-Validity-Coherence-Utility taxonomy, FaithCoT-Bench and C2-Faith. We use groundedness as a clinically oriented adaptation of the taxonomy's factuality category. The rubric brings these concepts together in a multidimensional framework for scoring free-text responses to gold-standard clinical vignettes. It includes provisional behavioural anchors, applicability rules and a separate flag for case-specific safety-critical errors. General-domain frameworks inform its design but are not treated as validated clinical instruments. The rubric does not replace case-specific reference criteria or the task-specific metrics of existing benchmarks. It has not yet been tested for inter-rater reliability, construct validity or clinical utility. Its immediate purpose is to make evaluation decisions explicit and open to scrutiny before empirical testing.

Zhangshu Joshua Jiang, Zina Ibrahim, James T. Teo

DRIVE-Health CDT Department of Biostatistics and Health Informatics Institute of Psychiatry, Psychology and Neuroscience King’s College London, London, UK · Neurological Institute, Cleveland Clinic London, London, UK · King’s College London, London, UK +1

RubricsTask-Specific RubricsLarge Language Model EvaluationLLM Reasoning StrategiesLarge Language Model ResponsesTaxonomy
AskPDF
Jun 30, 2026cs.CL

CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning

Large Language Models (LLMs) achieve strong results on many medical benchmarks, but their clinical reasoning remains difficult to evaluate reliably. A central risk is an evaluation illusion: fluent and well-structured explanations can appear clinically convincing even when the final diagnosis is incorrect. We introduce CLExEval, a human-in-the-loop framework for evaluating LLM clinical reasoning under progressive information masking. CLExEval combines 5,600 expert-physician annotations with 200 clinical reasoning traces derived from 40 rare diagnostic cases. Our analysis identifies three recurring failure patterns: (i) verbosity bias, where GPT-4o-mini's diagnostic accuracy drops from 95.0% to 32.5% under information scarcity; (ii) a hidden knowledge paradox, where a specialist model reaches 92.5% maximum diagnostic potential but fails to retrieve that knowledge reliably in verbose contexts; and (iii) a 68.6% reasoning-to-output mismatch, where correct diagnoses appear in reasoning traces but are not reflected in final answers. We further evaluate the LLM-as-a-Judge paradigm on a human-verified failure set (n = 142). GPT-4o-mini approved 47.9% of clinically incorrect outputs, while HuatuoGPT-o1 approved all validly scored failures and showed a positive self-preference bias. These results suggest that standalone automated clinical evaluations can substantially overestimate clinical reliability without expert-grounded validation.

Ajmal M., Abin Roy, Afthab Salam Kanniyan +4

1MBZUAI · 2IIT Madras · 3Calicut Medical College

LLM Reasoning StrategiesHuman-In-The-Loop
AskPDF
Apr 13, 2026cs.CL

Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces

Should we trust Large Language Models (LLMs) with high accuracy? LLMs achieve high accuracy on reasoning benchmarks, but correctness alone does not reveal the quality of the reasoning used to produce it. This highlights a fundamental limitation of outcome-based evaluation: models may arrive at correct answers through flawed reasoning, and models with substantially different reasoning capabilities can nevertheless exhibit similar benchmark accuracy, for example due to memorization or over-optimization. In this paper, we ask: given existing benchmarks, can we move beyond outcome-based evaluation to assess the quality of reasoning itself? We seek metrics that (1) differentiate models with similar accuracy and (2) are robust to variations in input prompts and generation configurations. To this end, we propose a reasoning score that evaluates reasoning traces along dimensions such as faithfulness, coherence, utility, and factuality. A remaining question is how to aggregate this score across multiple sampled traces. Naively averaging them is undesirable, particularly in long-horizon settings, where the number of possible trajectories grows rapidly, and low-confidence correct traces are more likely to be coincidental. To address this, we introduce the Filtered Reasoning Score (FRS), which computes reasoning quality using only the top-K% most confident traces. Evaluating with FRS, models that are indistinguishable under standard accuracy exhibit significant differences in reasoning quality. Moreover, models with higher FRS on one benchmark tend to perform better on other reasoning benchmarks, in both accuracy and reasoning quality. Together, these findings suggest that FRS complements accuracy by capturing a model's transferable reasoning capabilities. We open source our evaluation codebase: https://github.com/Manas2006/benchmark_reproducibility.

Manas Pathak, Xingyao Chen, Shuozhe Li +2

University of Texas at Austin

Reasoning Benchmark
AskPDF