From Hindsight to Foresight: Dissecting Clinical Evidence Acquisition in Large Language Models

Ziyang Ling

cs.CLOct 13, 2026

From Hindsight to Foresight: Dissecting Clinical Evidence Acquisition in Large Language Models

Authors: Ziyang Ling

Organizations: School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, China

Abstract

Large language models (LLMs) often diagnose full sight, static clinical cases well yet struggle in foresight, dynamic diagnosis. Prior work has improved this limitation by quantifying the information models acquire and by developing stronger follow-up strategies to improve diagnostic accuracy. In contrast, we introduce a distinctive hindsight to systematically dissect why LLMs fail to acquire diagnostically critical evidence, tracing the failure from evidence identification, utilization and downstream diagnostic updating through not only diagnostic accuracy, but also evidence coverage, trajectory dynamics, and elicited probability and information entropy. Starting experiments from the breakdown at the level of capabilities, to the diagnosis of the pre-event state before the evidence arrives, and to the post-event state transition after the evidence arrives, our research has revealed the profound limitations and internal causes existing in the dynamic diagnostic process. At the same time, we have also conducted sufficient verification and generalization experiments to prove that these limitations are widespread. Together, these findings shift the central challenge from whether LLMs possess relevant clinical knowledge to whether they can identify, acquire, and effectively use the right evidence under clinical diagnosis and conduct a more detailed mechanism analysis to identify the exact location of the capability gap.

Figures & tables

Figure 1: Diagnostic accuracy of ten general-purpose LLMs under three evidence-access settings (798 model–case pairs). Fullsight receives the complete case upfront; Foresight prospectively acquires evidence over at least six turns; and Hindsight retrospectively selects and orders diagnostically critical evidence from the completed case without reference diagnosis, before re-diagnosis.

Figure 2: Experiment design overview. Starting from the observed ability gap (A) to three steps of clinical evidence acquisition of LLMs, including (B) before evidence, (C) acquired evidence analysis and (D) after evidence examination.

Round	Panel A: Primary cohort ( $N=181$ )					Panel B: Continuation cohort ( $N=140$ )
$k$	Foresight	Hindsight	Gap [95% CI]	OR	Exact $p$	Foresight	Hindsight	Gap [95% CI]	OR	Exact $p$
1	18.8	55.8	37.0 [29.3, 45.3]	12.17	$1.00\times 10^{-15}$	15.7	57.1	41.4 [32.9, 50.0]	20.33	$4.74\times 10^{-15}$
2	29.3	66.3	37.0 [28.7, 45.3]	9.38	$9.10\times 10^{-15}$	25.7	66.4	40.7 [31.4, 50.0]	12.40	$1.42\times 10^{-13}$
3	33.1	73.5	40.3 [32.6, 48.6]	15.60	$6.40\times 10^{-18}$	29.3	70.0	40.7 [31.4, 50.0]	15.25	$3.92\times 10^{-14}$
4	35.9	75.1	39.2 [31.5, 47.0]	18.75	$5.24\times 10^{-18}$	30.7	75.0	44.3 [35.7, 52.9]	32.00	$6.00\times 10^{-17}$
5	34.8	74.0	39.2 [30.9, 47.5]	9.88	$8.38\times 10^{-16}$	28.6	73.6	45.0 [35.7, 54.3]	16.75	$8.74\times 10^{-16}$

Table 1: Fine-grained recognition–acquisition gaps across evidence depth, comparing retrospective Hindsight with prospective Foresight in two separately analyzed cohorts.

Figure 3: Acquired evidence analysis. Left: Diagnostic accuracy across request rounds under hindsight, free-form Foresight, and result-blind menu foresight; Right: Outcomes of free-form and menu requests, partitioned into new evidence, duplicate evidence, and no match.

Figure 4: Recovery after unsuccessful evidence requests. Next-request recovery rates are stratified by failure timing, failed-request category, and whether the next request stays in or switches category.

Panel A: same number of evidence items Cohort /

\bm{k}

Mean

\bm{m_{i}(k)}

Foresight Matched Hindsight Gap 95% CI Primary / 3 2.09 33.1% 63.0% 29.8 [22.1, 37.6] Primary / 6 3.52 42.0% 70.7% 28.7 [20.4, 37.0] Continuation / 6 3.70 37.1% 72.9% 35.7 [27.1, 44.3] Continuation / 12 5.96 47.9% 69.3% 21.4 [12.1, 30.7] Panel C: direct set and order interventions (

N=100

) Contrast Ref. Alt. Effect 95% CI Measure Same-set reranking 34.3% 43.7% 9.3 [5.5, 13.2] AUSC Set replacement 43.7% 70.3% 26.7 [19.0, 34.5] AUSC Hindsight-order 44.3% 70.3% 26.0 [20.2, 32.0] AUSC

Panel B: evidence identity and stable sufficiency (

N=181

)

\bm{k}

CER Hindsight Top-1 reached Foresight stable Hindsight stable Stable gap 1 5.5% 5.5% 12.7% 40.3% 27.6 2 12.2% 13.3% 18.8% 45.3% 26.5 3 16.0% 18.2% 20.4% 50.3% 29.8 4 20.9% 23.8% 23.2% 50.8% 27.6 5 23.8% 30.4% 25.4% 55.2% 29.8 6 26.9% 35.4% 27.6% 58.6% 30.9 7 28.6% 38.7% 34.3% 60.8% 26.5 8 30.1% 42.5% 36.5% 62.4% 26.0 9 31.6% 44.8% 39.2% 66.3% 27.1 10 33.6% 45.9% 44.2% 68.0% 23.8 11 34.6% 49.2% 49.2% 71.3% 22.1 12 35.7% 53.6% 52.5% 77.3% 24.9

Table 2: Evidence-value decomposition. Panel A matches the realized evidence count case by case; Panel B reports evidence identity and diagnostic stability through

k=12

on the primary cohort; Panel C reports paired set and order interventions. Stable denotes a correct diagnosis at every remaining depth through

k=12

; when fewer than

k

non-initial facts are available, CER is computed over the available Hindsight prefix. Gap and effect columns are percentage points.

Figure 5: Distributional change and diagnostic value. Rows compare Foresight and Hindsight; columns show case-level JSD versus $\Delta p_{\mathrm{ref}}$ , aligned reference-probability trajectories, and normalized-entropy trajectories.

Figure 6: Diagnostic updating after evidence arrival from initial to round 12.

Figure 7: Diagnostic rescue by evidence rank and insertion depth. Cells report rescue rates and eligible counts after inserting evidence. The left panel shows the critical-minus-random difference in percentage points, and the right three are individuals.

Figure 8: Robustness and external validation. The recognition–acquisition gap recurs across selector perturbations, model backbones, source cohorts, and longer horizons.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Panel	Cases	Purpose
Source universe / Fullsight-correct / strict primary	320 / 190 / 181	Conditional six-step paired analysis
Result-blind menu	120	Frozen primary subset
Evidence-set/order intervention	100	Fixed six-prefix panel
Selector perturbation	100	Repeats, prompt, and order; alternative-model selector: 80
Four-model paired replication	59	Same case IDs across backbones
Shared four-model menu	50	Free/menu/Hindsight at three requests

Appendix

Table 3: Frozen study panels. Subsets overlap; their counts must not be added. A panel’s own eligibility and denominator govern its result.

Model	$N$	Full	Fore	Hind
GPT-5.5	89	96.4	75.3	88.8
Gemini 3.1 Pro	86	98.8	84.9	91.9
DeepSeek V3.2	73	92.6	63.0	82.2
Qwen3.8-Max	82	90.9	73.2	86.6
Claude Sonnet 4.6	80	90.8	70.0	82.5
GLM 5.2	84	88.7	81.0	81.0

Appendix

Table 4: All ten completed models in the early capability screen. Values are percent correct among each model’s eligible cases. The early retrospective selector was diagnosis-conditioned and is not pooled with the strict diagnosis-unconditioned experiment.

$k$	Foresight	Hindsight	Gap	95% CI low	95% CI high
1	18.8%	55.8%	37.0%	29.3%	45.3%
2	29.3%	66.3%	37.0%	28.7%	45.3%
3	33.1%	73.5%	40.3%	32.6%	48.6%
4	35.9%	75.1%	39.2%	31.5%	47.0%
5	34.8%	74.0%	39.2%	30.9%	47.5%
6	42.0%	75.7%	33.7%	26.0%	41.4%

Appendix

Table 5: Primary paired accuracy at every original request budget (

N=181

). The interval is for the within-case Hindsight-minus-Foresight difference, not for two independent accuracies.

Turn	New	Duplicate	No match	New/181	Mean unique by turn
1	164	0	17	90.6%	0.91
2	124	25	32	68.5%	1.59
3	91	50	40	50.3%	2.09
4	95	55	31	52.5%	2.62
5	88	64	29	48.6%	3.10
6	75	81	25	41.4%	3.52

Appendix

Table 6: Action realization by request position in the 181-case primary cohort. Every position has 181 attempts. Cumulative unique facts count newly returned IDs, so duplicate and no-match turns do not increase it.

	Accuracy (%)			Free-form		Menu
$k$	Free	Menu	Hind	New	Duplicate	New	Duplicate
3	29.3	32.1	70.0	296	55	230	130
6	37.1	38.6	73.6	518	191	306	396
9	45.7	37.9	77.9	689	391	351	696
12	47.9	37.9	77.1	834	626	378	1006

Appendix

Table 8: Extended menu comparison on the frozen 140-case subset. The right columns count realized new and duplicate outcomes cumulatively over

140k

request slots per prospective condition; each condition’s diagnostic accuracy is evaluated at the indicated budget.

Failed turn	After duplicate	After no-match
1	—	6/17 (35.3%)
2	12/25 (48.0%)	9/32 (28.1%)
3	22/50 (44.0%)	18/40 (45.0%)
4	24/55 (43.6%)	11/31 (35.5%)
5	25/64 (39.1%)	7/29 (24.1%)
All	83/194 (42.8%)	51/149 (34.2%)

Appendix

Table 9: Immediate recovery by position of the failed request. Entries are recovered/eligible events (percentage). No duplicate is possible at turn 1 because no fact has yet been acquired.

Failure	Next category	Recovered	Eligible	Rate [95% CI]
duplicate	stay	7	41	17.1 [6.1, 30.8]
duplicate	switch	76	153	49.7 [41.2, 58.7]
no match	stay	17	26	65.4 [47.6, 82.1]
no match	switch	34	123	27.6 [19.2, 38.3]

Appendix

Table 10: Failure-specific category switching and next-turn recovery. Rates and 95% case-cluster bootstrap intervals are taken from the frozen B.3 analysis.

Failed category	After duplicate	After no-match
History	29/82 (35.4%)	18/40 (45.0%)
Physical exam	23/45 (51.1%)	16/43 (37.2%)
Laboratory	15/31 (48.4%)	13/34 (38.2%)
Imaging	7/11 (63.6%)	1/11 (9.1%)
Medication	9/25 (36.0%)	1/14 (7.1%)
Other	—	2/7 (28.6%)

Appendix

Table 11: Recovery stratified by the failed request’s declared category. Small imaging and medication no-match strata are descriptive.

Cohort / requests	Mean $m_{i}(k)$	Foresight	Matched Hindsight	Gap	95% CI (pp)
Primary / 3	2.09	33.1	63.0	29.8	[22.1, 37.6]
Primary / 6	3.52	42.0	70.7	28.7	[20.4, 37.0]
Continuation / 6	3.70	37.1	72.9	35.7	[27.1, 44.3]
Continuation / 12	5.96	47.9	69.3	21.4	[12.1, 30.7]

Appendix

Table 12: Actual fact-count matching. Accuracy and paired difference are percentages and percentage points, respectively. The continuation cohort is reported separately from the primary 181 cases.

$k$	$m$	$N$	Foresight (%)	Matched Hindsight (%)	Gap [95% CI] (pp)
3	0	11	0.0	0.0	0.0 [0.0, 0.0]
3	1	26	19.2	42.3	23.1 [3.8, 42.3]
3	2	79	36.7	74.7	38.0 [25.3, 50.6]
3	3	65	40.0	67.7	27.7 [15.4, 40.0]
6	0	9	0.0	0.0	0.0 [0.0, 0.0]
6	1	5	20.0	40.0	20.0 [0.0, 60.0]

Appendix

Table 13: Primary-cohort count-matched strata.

m

is the number of unique facts actually obtained. The last column is the paired Hindsight-minus-Foresight accuracy difference with a case bootstrap 95% interval, in percentage points.

$k$	CER	Top-1 reached	F stable (6)	Hindsight stable (6)
1	5.5	5.5	14.4	43.1
2	12.2	13.3	23.2	50.3
3	16.0	18.2	27.1	56.4
4	20.9	23.8	30.9	60.2
5	23.8	30.4	34.8	66.9
6	26.9	35.4	42.0	75.7

Appendix

Table 14: Critical evidence and stable correctness in the primary 181 cases, requests 1–6. Stable (6) requires correctness at every later prefix through request 6. CER and Top-1 reach use exact canonical fact IDs. All entries are percentages.

Sequence or contrast	Estimate (%)	95% CI
FF: Foresight set, original order	34.3	[26.8, 42.0]
FR: Foresight set, reranked	43.7	[34.8, 52.7]
HR: Hindsight set, reversed	44.3	[38.0, 50.8]
HH: Hindsight set, Hindsight order	70.3	[63.3, 77.0]
FR $-$ FF: same-set order gain (pp)	9.3	[5.5, 13.2]
HH $-$ FR: set contrast (pp)	26.7	[19.0, 34.5]

Appendix

Table 15: Fixed 100-case set/order intervention. AUSC is the mean fraction of the first six prefix diagnoses that are correct; intervals are paired case-level bootstrap 95% intervals. The last three rows report paired percentage-point contrasts, rather than independent arm comparisons.

Figure 9: Four aligned supplementary panels for Experiment C: residual diagnostic gap within actual fact-count strata; critical-fact reach and six-step stable sufficiency; four set/order arms; and source-stratified effects. The horizontal four-panel arrangement keeps equally sized panels at the same visual scale.

Sequence	Interval	$n$	Mean JSD [95% CI]	Mean $\Delta p_{\rm ref}$ [95% CI]
Foresight	$0\to 3$	170	0.531 [0.472, 0.586]	0.084 [0.043, 0.127]
Foresight	$3\to 6$	146	0.509 [0.449, 0.572]	0.099 [0.040, 0.157]
Hindsight	$0\to 3$	181	0.633 [0.578, 0.686]	0.298 [0.239, 0.359]
Hindsight	$3\to 6$	181	0.477 [0.420, 0.536]	0.053 [-0.015, 0.122]

Appendix

Table 16: Six-step C2 scatter inputs from the 181-case rereadout, restricted to intervals with at least one new fact. Means and percentile 95% case-bootstrap intervals are shown for JSD (bits) and reference-probability change. Denominators are case–intervals, not a fixed 50-case heatmap sample.

Figure 10: Detailed current-protocol state-pair readouts for the 181 cases. The upper two panels show reference-diagnosis probability and the lower two show diagnostic entropy, each separated by Foresight and Hindsight. Faint lines represent individual elicited state pairs; diamonds and intervals summarize endpoint means. The main figure reports the more legible interval changes from these same states.

Figure 11: Four panels for the all-eligible D.2 sensitivity analysis: critical-minus-control rescue in percentage points, critical-fact rescue, pre-frozen same-case control rescue, and unaugmented repeat rescue. Each cell prints its eligible count; denominators vary across ranks and insertion depths and cannot be read as a fixed-cohort timing series. The control follows the coarse modality/inventory rule.

Figure 12: Four aligned secondary panels: readout reliability, calibration, elicited-state transition classes, and local value-policy comparison. These analyses are sensitivity evidence.

$k$	Strategy	ICR	Accuracy	Facts
3	Official baseline	16.0	16.0	2.98
3	Menu	23.2	19.0	4.26
3	REFINE	16.5	14.0	2.93
3	Hindsight reference	18.9	36.0	3.00
6	Official baseline	25.2	22.0	5.01
6	Menu	27.1	25.5	5.20

Appendix

Table 29: EviMed prefix endpoints. The rows at 3 and 6 turns are descriptive; the official evaluation budget is 16 turns. ICR and accuracy entries are percentages.

Manual clinical-review outcome	Estimate	95% CI
Top-1 diagnostically relevant	100.0%	[100.0%, 100.0%]
Top-3 contains critical evidence	100.0%	[100.0%, 100.0%]
Top-3 order reasonable (strict)	97.5%	[95.5%, 99.5%]
Critical evidence missing from Top-3	1.0%	[0.0%, 2.5%]
Any Top-6 item answer-revealing	1.0%	[0.0%, 2.5%]
Reviewer–Hindsight Top-1 agreement	89.0%	[83.0%, 94.5%]

Appendix

Table 30: Outcomes of the manual clinical audit. Confidence intervals resample cases.

Manual review judgment	Exact agreement	Kappa / ICC
Top-1 relevance	100.0%	n.a.
Top-3 critical evidence	100.0%	n.a.
Top-3 order	90.0%	0.000
Missing-evidence flag	96.0%	0.000
Answer-revealing flag	96.0%	0.000
Preferred Top-1 evidence	88.0%	–

Appendix

Table 31: Agreement between two manual review records on 50 doubly assessed cases. Chance-adjusted and continuous-rating measures are reported alongside exact agreement where applicable.

Medical LLMs

Medical large language models are AI systems adapted to clinical and biomedical data for health reasoning, information extraction, and communication.

See all 397 papers

Can LLMs Value the Right Evidence? Evidence-Value Misalignment in Dynamic Medical Diagnosis
Sep 28, 2026Kehua Feng, Yunsheng Lu, Yitong Qiao +5LLM Evaluation Evidence-Grounded Reasoning
Zhejiang University · Ant Healthcare, Ant Group
Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support
May 21, 2026Chen Zhan, Xihe Qiu, Xiaoyu Tan +8Healthcare Clinical Decision Support
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, 201620, China. · Department, Tencent Youtu Lab, Shanghai, 200233, China. · Department of Oncology, Jinshan Hospital, Fudan University, Shanghai, 201508, China. +4
Large language models exhibit unreliable updating of clinical judgment as patient evidence evolves
Oct 2, 2026Min Zeng, Rui ZhangBelief Updating in LLMs LLM Reliability
Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, 55455, MN, USA.

Sep 28, 2026J/K move · Enter open · S save

Can LLMs Value the Right Evidence? Evidence-Value Misalignment in Dynamic Medical Diagnosis

Kehua Feng, Yunsheng Lu, Yitong Qiao, Tiantian He+4

Zhejiang University · Ant Healthcare, Ant Group

LLM Evaluation Evidence-Grounded Reasoning Medical Diagnosis Clinical Decision-Making

Open paper AskPDF arXiv

A correct diagnosis reached from insufficient or misleading evidence can pose a clinical hazard, yet outcome-based accuracy may reward such lucky guesses. We call this mismatch between diagnostic decisions and the value of available evidence Evidence-Value Misalignment (EVM). To disentangle evidential grounding independently from diagnostic accuracy, we introduce MedEVM, a dynamic benchmarking environment comprising 1,050 cases across 24 disease systems. Observations arrive turn by turn, requiring models to continuously calibrate its decision by deciding whether to wait for more evidence or submit a diagnosis. Across 9 LLMs, four interesting patterns are observed. (1) Miscalibrated evidence tracking. Making a diagnosis often fails to calibrate evidence sufficiency, even in more capable models, and even worsens in reasoning mode. (2) Misaligned diagnosis submission. Confidence in the correct diagnosis often fails to ensure timely submission despite sufficient evidence. (3) Evidence order matters. Reordering the same evidence changes diagnoses even when model confidence remains similar. (4) Misleading evidence remains influential. Added misleading evidence redirects diagnoses even after prior evidence becomes sufficient. We further verify that EVM predicts errors and that preventing premature submission improves accuracy. These findings motivate Evidence-Verified Diagnosis Harness (EVD-Harness). It decouples diagnosis generation from submission through an offline Contrastive Diagnostic Wiki and three online control stages, namely observation management, proposal and witness verification, and diagnosis submission control. Across five LLMs, EVD-Harness improves accuracy by 12.0--51.1 percentage points while mitigating EVM-related failures. Our results demonstrate that verifying evidential support before submission can make diagnostic decisions more reliable.

Abstract

Medical LLMs

Explore similar work

Can LLMs Value the Right Evidence? Evidence-Value Misalignment in Dynamic Medical Diagnosis

Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support

Large language models exhibit unreliable updating of clinical judgment as patient evidence evolves