Where Does Hindsight Enter Clinical Agent Benchmarks? An Audit of Simulators and Case Construction
Organizations: Department of Electrical and Computer Engineering, The University of Hong Kong · Department of Restorative Dental Sciences, Faculty of Dentistry, The University of Hong Kong · Department of Infectious Diseases and Public Health, City University of Hong Kong
Abstract
Benchmarks for LLM diagnostic agents are built from retrospective records: the case an agent reads is rewritten from discharge summaries, and unrecorded test results are generated by a simulator that often sees the final diagnosis. HindAudit separates three hindsight channels (the diagnosis given to the simulator, the retrospective context it conditions on, and the agent-facing case text) and measures each with judges trained only on real data. On 27,489 emergency admissions from MIMIC-IV, five registered tests found no leakage into simulated laboratory results: providing the diagnosis to five simulators had a negligible effect, and prompted gatekeepers diluted the diagnostic signal, although exploratory results suggest they can raise agent scores. The largest effect lay in the case text. Against a presentation-time reconstruction from triage records, benchmark-style summaries raised the zero-shot accuracy of four LLMs from 25.4% to 39.7% and discharge histories to 45.3%, and raised six interactive agents' scores while barely changing their ranking. The exploratory gap was reproduced in a registered replication on held-out patients. Masking diagnosis names left it unchanged. Because MIMIC lacks a contemporaneous bedside history, the gap is an upper bound on hindsight. We report all registered outcomes and recommend construction practices.
Figures & tables
| Input | top-1 | top-5 | |
| Retrospective history | 0.559 | 0.891 | – |
| with diagnosis words deleted | 0.558 | 0.878 | – |
| Chief complaint | 0.368 | 0.702 | – |
| Sex + chief complaint ( ) | 0.366 | 0.703 | – |
| LLM pre-hospital summary (filtered) | 0.491 | 0.857 | 0.192 |
| Chief complaint (same cases) | 0.357 | 0.706 | 0.190 |
| V1 | V2 | V3 | V4 | V1 V3 | V2 V3 | |
| Qwen2.5-7B | 37.5 | 42.8 | 23.1 | 23.2 | +14.4 | +19.8 |
| Qwen2.5-14B | 39.3 | 45.1 | 24.4 | 24.5 | +14.9 | +20.7 |
| Qwen2.5-32B | 40.0 | 46.0 | 27.5 | 25.8 | +12.4 | +18.4 |
| Phi-4 | 41.9 | 47.2 | 26.7 | 27.3 | +15.2 | +20.5 |
| Llama-3.1-8B † | 34.4 | 39.8 | 17.7 | 18.9 | +16.7 | +22.1 |
| Mean (four) | 39.7 | 45.3 | 25.4 | 25.2 | +14.2 | +19.8 |
| Test | Simulator / setting | Primary metric | [95% CI] | Met? |
| D1 | FT-3B, sequential | clf top-1, real | [ , ] | no |
| A | GK-14B, sequential | clf top-1, | [ , ] | no |
| A2 | GK-14B, independent | num , real and | [ , ] [ , ] | no |
| Family A ′ (one-sided, Holm, ) | ||||
| A ′ 1 | summary vs. chief complaint | text clf , summary complaint | [ , ], | no |
| A ′ 2 | agent, GK-14B | top-1, | [ , ], | no |
| Simulator | Type | Gen. | Retention, linear | Retention, boosted | Typ. / | ||
| FT-3B (ours) ∗ | trained | seq. | +0.001 | – | 0.84 [0.75, 0.94] | 0.81 [0.73, 0.89] | – |
| DiagGym 7B | trained | seq. | +0.007 | – | 0.77 [0.67, 0.88] | 0.82 [0.74, 0.92] | – |
| GK-14B ∗ | prompted | seq. | 0.001 | – | 0.25 [0.14, 0.37] | 0.46 [0.37, 0.55] | – |
| GK-14B ∗ | prompted | ind. | +0.002 | +0.019 [+0.011, +0.028] | 0.16 [0.06, 0.27] | 0.43 [0.34, 0.53] | 0.45/0.36 |
| GK-32B + units ∗ | prompted | ind. | – | +0.028 [+0.015, +0.042] | 0.32 [0.21, 0.46] | 0.58 [0.50, 0.67] | 0.61/0.39 |
| GK-72B + units | prompted | ind. | +0.003 | +0.023 [+0.012, +0.035] | 0.27 [0.17, 0.40] | 0.62 [0.53, 0.72] | 0.61/0.53 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Test | Decision rule as registered | Outcome |
| D1 | Stop if, after removing explicit leakage, the CI of real (clf top-1) includes 0 or the gain is pp; both judges must agree. Implemented two-sided at first; corrected to the registered one-sided form before the decision. | stop (both judges) |
| A | Pass if pp with CI lower bound and the real point estimate is (clf top-1). Otherwise no third simulator is tested. | fail |
| A2 | Pass if the CIs of real and both lie above 0 (num judge, ). Declared the last test of this hypothesis. | fail |
| A ′ 1–3 | Three one-sided tests, patient-level bootstrap ( ), Holm at . H1: summary chief complaint, text-classifier , filtered test cases. H3: real, num , 32B gatekeeper with unit hints. H2: agent top-1, simulator. | none rejected (smallest ) |
| E1 | On the validation split, four-model mean top-1 gaps V1 V3 and V2 V3; one-sided paired bootstrap ( ); replicated if the null is rejected for both under Holm at and both gaps are 5 pp. | replicated |
| Component | Setting |
| FT-3B simulator | Qwen2.5-3B-Instruct + LoRA ( , , all linear layers); diagnosis dropout 0.5; lr , cosine; 2 epochs; gradient accumulation 16; maximum length 6,144 tokens; loss on results only. |
| Simulated results | 5 samples per item, temperature 1.0, at most 512 new tokens; sequential (given real earlier results) or independent (profile only). |
| Numeric judge (num) | Multinomial logistic regression ( ) on the 300 most frequent laboratory items: standardized value, abnormal flag, missing indicator; lenient number parsing. Values-only variant drops the flags. |
| Boosted judge | Histogram gradient boosting, at most 200 iterations, early stopping, same features as num. |
| Text judges | TF-IDF (1–2-grams, at most 30,000 features, min. document frequency 3, sublinear tf) + logistic regression ( ). |
| LLM diagnosis | Temperature 0, at most 200 new tokens; one follow-up turn if no valid code is returned (afterwards at most 0.7% of answers per model and view lacked a valid code); input truncated to 12,000 characters. Llama-3.1-8B-Instruct (bf16) was added as a third model family on the test split only. |
| Classifier | top-1 | mean | mean max-prob | entropy | ECE | NLL | |
| Summary, raw | 1.000 | 0.491 | 0.192 | 0.252 | 3.089 | 0.239 | 2.122 |
| Summary, scaled | 0.561 | 0.491 | 0.352 | 0.510 | 1.874 | 0.057 | 1.776 |
| Chief complaint, raw | 1.000 | 0.357 | 0.190 | 0.294 | 2.734 | 0.063 | 2.406 |
| Chief complaint, scaled | 0.856 | 0.357 | 0.221 | 0.355 | 2.430 | 0.038 | 2.375 |
| Category (CCSR) | Largest-weight terms |
| Acute pulmonary embolism | pulmonary, pleuritic, dyspnea, clots , chest, pulmonary embolism, pe , embolism |
| Gastrointestinal hemorrhage | melena , stools, black , bleeding, stool, dark , blood, gi |
| Intestinal obstruction and ileus | flatus , bowel, abdominal, abdominal pain, small bowel, obstruction, bowel movement |
| Chronic kidney disease | esrd , kidney, blood pressure, renal, stage, kidney disease, dialysis |
| Cardiac dysrhythmias | palpitations , atrial, fibrillation, tachycardia, atrial fibrillation , rate |
| Heart failure | dyspnea, heart failure, edema , orthopnea , heart, failure, of breath |
| Agent | V1 | V2 | V3 | V1 V3 | V2 V3 | requests per case (V1/V2/V3) |
| Qwen2.5-7B | 33.0 | 38.7 | 24.6 | +8.4 | +14.1 | 6.1 / 6.1 / 6.0 |
| Llama-3.1-8B | 29.4 | 36.1 | 22.5 | +6.9 | +13.6 | 6.1 / 5.7 / 6.0 |
| Qwen2.5-14B-AWQ | 38.5 | 44.3 | 29.9 | +8.6 | +14.4 | 5.8 / 4.8 / 5.4 |
| Phi-4 | 40.5 | 45.0 | 35.6 | +4.9 | +9.4 | 6.7 / 6.3 / 6.7 |
| Qwen2.5-32B-AWQ | 41.2 | 46.5 | 33.0 | +8.2 | +13.5 | 6.0 / 5.7 / 6.0 |
| Qwen2.5-72B-AWQ | 44.6 | 48.7 | 38.7 | +5.9 | +10.0 | 6.2 / 6.3 / 6.2 |
| GK-14B | 0.454 | 0.438 | 0.359 |
| GK-32B + units | 0.612 | – | 0.394 |
| GK-72B + units | 0.609 | 0.582 | 0.534 |
| Phi-4 + units | 0.618 | 0.598 | 0.481 |
| Recommendation | Evidence in this paper | |
| 1 | Build agent-facing cases from information documented at presentation (e.g. triage records), or report a presentation-time baseline next to every score. | Benchmark-style text made zero-shot diagnosis 11–20 pp easier than a presentation-time reconstruction, on test and held-out patients (§ 5.2 ). |
| 2 | Do not rely on diagnosis-name filters; test filters against implicit cues. | Deleting diagnosis terms, alone or with four other word classes, left the gap intact (Table 2 , Figure 3 ). |
| 3 | When auditing with text-based judges, mask by deletion rather than by a placeholder token. | A placeholder raised summary top-1 from 0.491 to 0.525 (Appendix J ). |
| 4 | Report the signal retention of simulated results against real results, using a judge trained on real data only. | Prompted gatekeepers retained 16–62% of the real signal and trained world models 70–84% (§ 5.4 ). |
| 5 | Condition simulators on presentation-time profiles where possible, and report the gap to retrospective profiles. | Retrospective profiles increased predictability by 0.014–0.059 in , whereas the diagnosis line had a negligible effect (§ 5.4 ). |
| 6 | Before attributing score differences to simulators, verify that the agent uses the simulated results. | Overall, agent accuracy differed by at most about two points with and without laboratory results; in a laboratory-defined subset, a stronger agent scored higher with the hindsight gatekeeper than with real results (§ 5.3 ). |
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,304 papers