cs.CLNov 17, 2026

Where Does Hindsight Enter Clinical Agent Benchmarks? An Audit of Simulators and Case Construction

Authors: Meilu Zhu, Xiaohan Li, Axiu Mao

Organizations: Department of Electrical and Computer Engineering, The University of Hong Kong · Department of Restorative Dental Sciences, Faculty of Dentistry, The University of Hong Kong · Department of Infectious Diseases and Public Health, City University of Hong Kong

Abstract

Benchmarks for LLM diagnostic agents are built from retrospective records: the case an agent reads is rewritten from discharge summaries, and unrecorded test results are generated by a simulator that often sees the final diagnosis. HindAudit separates three hindsight channels (the diagnosis given to the simulator, the retrospective context it conditions on, and the agent-facing case text) and measures each with judges trained only on real data. On 27,489 emergency admissions from MIMIC-IV, five registered tests found no leakage into simulated laboratory results: providing the diagnosis to five simulators had a negligible effect, and prompted gatekeepers diluted the diagnostic signal, although exploratory results suggest they can raise agent scores. The largest effect lay in the case text. Against a presentation-time reconstruction from triage records, benchmark-style summaries raised the zero-shot accuracy of four LLMs from 25.4% to 39.7% and discharge histories to 45.3%, and raised six interactive agents' scores while barely changing their ranking. The exploratory gap was reproduced in a registered replication on held-out patients. Masking diagnosis names left it unchanged. Because MIMIC lacks a contemporaneous bedside history, the gap is an upper bound on hindsight. We report all registered outcomes and recommend construction practices.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Hx2Dx: From Anamnesis to Diagnosis - A Multimodal Arena for Conversational Clinical Reasoning in LLMs

    Nov 10, 2026Federico Ravenda, Mirko Cesarini, Daniele Montagnani +3interactive medical diagnosisClinical Language Model Evaluation

  2. DiagTrace: An Interactive Benchmark for Clinical Decision Making

    Nov 16, 2026Yufan Wang, Rui Yang, Yi Lin +2interactive medical diagnosisClinical Decision-Making

  3. KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy

    Sep 29, 2026Xueting Fang, Zehui Li, Yang Yang +5Clinical Decision-MakingMulti-Turn Dialogue Evaluation