Answer continuation often returns a diagnosis already stated in unfinished clinical reasoning. We study this behavior in MamaReason, adapted from dense Gemma 4 31B, on MamaBench's 217 paired maternal and child-health presentations. An explicit-commitment extractor identifies a diagnosis in 75 of 94 failed RL first passes and 75 of 93 failed SFT first passes. The continuation matches that diagnosis in 67/75 RL cases and 58/75 SFT cases, using diagnosis strings alone. Reading these commitments without additional generation recovers 30 of the 40 RL pairs restored by continuation under the archived correctness labels. Recurrence controls distinguish routine mentions from repeated answer selection. An offline second-commitment trigger measures the tradeoff between earlier readout and paired accuracy. The resulting decomposition separates diagnostic content already present in a trace, completion of an accepted answer, and paired diagnostic correctness. For clinical language-model evaluation, these are distinct outcomes that become visible when the same reasoning trace is examined through multiple readouts.
Figures & tables
Figure 1: Diagnosis preservation and paired recovery. The direct-answer reference is generated separately; the other bars share the same bounded-thinking first pass. Continuation adds a conditional answer-generation step. Commitment uses the completed diagnostic adjudication. All robust scores use 217 pairs.
Protocol
Maximum output
Emission procedure
Direct answer
1,024 tokens
Thinking disabled through the chat template.
Bounded thinking
3,000 + 512 tokens
Conditional thought closure and answer-rescue continuation.
Table 1: Principal inference protocols and maximum output lengths.
Checkpoint
Readout
Available cases
Available pairs
Correct cases
Correct pairs
Robust (%)
SFT
Direct answer
434
217
307
113
52.1
SFT
Thinking first pass
341
133
279
95
43.8
SFT
Thinking end-to-end
434
217
332
132
60.8
RL
Direct answer
434
217
311
116
53.5
RL
Thinking first pass
340
135
285
98
45.2
RL
Thinking end-to-end
434
217
339
138
63.6
Table 2: Answer availability and correctness over 434 cases and 217 pairs. First-pass and end-to-end readouts share the bounded-thinking first pass; the latter adds conditional continuation. Direct answering is generated separately. An available pair has two accepted predictions.
Figure 2: Length and availability in fixed first-pass trajectories. Left: empirical token-length distributions. Right: parser availability at saved prefixes, with denominators of 434 cases and 217 pairs. RL case availability rises from 332 to 340 between 2,000 and 3,000 tokens.
Group and diagnosis target
n
Mentions
Assertions
Available; accepted diagnosis
340
338
15
Failed; extracted commitment
75
71
35
Failed; continuation diagnosis
94
89
33
Table 3: RL recurrence in the final 500 thought tokens. Mentions and assertions count traces with at least two matches. Assertions use frozen selection rules. The extracted-commitment group contains the 75 eligible failures; the continuation-target group retains all 94.
Checkpoint
Correct pairs / 217
Robust (%)
Difference
95% interval
Gains / losses
p
RL
129
59.4
+6.0
[+0.5,+11.5]
27 / 14
0.0596
SFT
123
56.7
+4.6
[−1.4,+10.6]
26 / 16
0.1641
Table 4: Completed fallback evaluation after adjudicating all extracted diagnoses. Every score uses 217 pairs. Differences and paired-bootstrap 95% intervals are in percentage points, relative to the corresponding direct-answer condition. p is the exact McNemar result.
Checkpoint
Triggered cases
Diagnosis matches
Omitted text (%)
Correct pairs / 217
Robust (%)
RL
91
84 / 91
24.1
122
56.2
SFT
72
67 / 72
18.5
114
52.5
Table 5: Offline second-commitment readout on all 434 cases per checkpoint. Agreement compares triggered diagnoses with saved final predictions. Omitted text is measured over reconstructed first-pass tokens; scores use 217 pairs and completed diagnostic labels.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Records
Supervised source QA
28,082
Unique normalized questions
22,798
Sampling target
12,000
Saved reasoning demonstrations
11,591
Cleaned demonstrations
10,655
SFT train / validation
10,122 / 533
Appendix
Table 6: Data construction. The final 1,000-case subset is the dataset used for RL; the larger pool is not the training size.
Failed-output pattern
RL
SFT
All first-pass failures
94
93
JSON parse failures
93
91
Blank/template predictions
1
2
Exactly 3,000 reconstructed tokens
90
90
Within one token of cap
92
90
Below cap with exact suffix cycle
2
3
Appendix
Table 7: First-pass failure signatures. Cap-consistent means within one reconstructed token of 3,000. Shorter exact-cycle failures form a separate group.
Table 8: Added diagnostic evaluations. L denotes a last-commitment candidate; E denotes an early-trigger candidate. Verdicts use the same automatic diagnostic criterion as the main evaluation.
Rubric item
Clinician 1 Yes
Clinician 2 Yes
Both Yes
Agree / 35
κ
Agreement with automatic verdict
26
33
26
28
0.298
Overthinking
15
12
7
22
0.222
Underthinking
13
3
1
21
−0.017
Missed critical information
3
1
0
31
−0.045
Misweighted finding
4
1
0
30
−0.048
Hallucinated finding
2
0
0
33
0.000
Appendix
Table 9: All completed binary review axes. “Yes” follows the original rubric; for thought–answer consistency, Yes affirms consistency. Every count has denominator 35. Agreement compares the clinicians’ ratings, not agreement between a clinician and the automatic evaluator. κ is computed from the corresponding two-by-two table.
Score / summary
Clinician 1
Clinician 2
1
0
0
2
6
0
3
8
6
4
16
20
5
5
9
Mean
3.57
4.09
Appendix
Table 10: Clinical acceptability ratings, preserving the ordinal five-point scale. Every clinician rates all 35 cases.
Figure 4: Clinician flags on the same 35 selected cases. Inconsistency is the inverse of the rubric’s thought–answer consistency item; the other bars count affirmative flags.
Protocol
Token interval
Unanswered at start
New answers
Yield (%)
Clarified bounded thinking
1,000–2,000
133
31
23.3
Clarified bounded thinking
2,000–3,000
102
8
7.8
Historical direct thinking
1,000–2,000
309
79
25.6
Historical direct thinking
2,000–3,000
230
20
8.7
Historical direct thinking
3,000–4,096
210
17
8.1
Appendix
Table 11: Within-run answer-availability yields. Unanswered outputs lack an accepted prediction at the lower prefix, including earlier-stopped outputs. New answers become available at the upper prefix. No accepted-prefix losses occur at these boundaries.
Checkpoint
Protocol
PP
PF
FP
FF
Base ↑
CF ↑
Robust ↑
BTR ↓
SFT
DA
113
48
33
23
74.2
67.3
52.1
29.8
SFT
BT+R
132
41
27
17
79.7
73.3
60.8
23.7
RL
DA
116
48
31
22
75.6
67.7
53.5
29.3
RL
BT+R
138
42
21
16
82.9
73.3
63.6
23.3
Appendix
Table 12: MamaReason on MamaBench: all conditions cover 217 pairs / 434 cases. DA denotes direct answering; BT+R denotes bounded thinking with conditional rescue. Values are percentages; pair counts precede the metrics. Arrows indicate preferred direction. SFT rows retain archived labels except the documented CASE_085 correction; scoring-snapshot details are given in Appendix H .
Figure 5: Diagnostic accuracy and pair outcomes across protocols. Left: base and robust accuracy. Right: the full PP , PF , FP , and FF distributions. BT+R denotes bounded thinking with conditional rescue.
Figure 6: Paired transitions for the RL checkpoint. Rows are direct-answer outcomes and columns are bounded-thinking outcomes for the same 217 pairs. Joint correctness is gained in 30 pairs and lost in eight. Cell counts describe a protocol change, including a larger budget and conditional continuation.
Published system
Base
Robust
BTR
1. GLM-5
54.8
27.2
50.4
2. Kimi K2.5
76.0
45.6
40.0
3. GPT-4o + RAG
77.4
46.5
39.9
4. Nemotron-120B
72.4
47.0
35.0
5. GPT-4o + EA-RAG
77.0
49.3
35.9
6. GPT-5.4
77.4
57.6
25.6
Appendix
Table 13: Published MamaBench reference scores, transcribed from its reported model/configuration comparison [ 11 ] . All use 217 pairs. These are external reported scores, not controlled reruns. EA-RAG denotes Evidence-Anchored RAG.
Figure 7: Published MamaBench systems (gray, indexed as in Table 13 ) and MamaReason RL bounded thinking (teal, M). The upper-left direction favors higher robust accuracy and lower BTR. Systems use different models and inference configurations.
Figure 8: RL correctness by rescue status. Case denominators are 94 and 340; pair denominators are 82 and 135. Rescued pairs contain at least one rescued member. Both members must be correct for pair success.
Model
Rescued
Correct
n
Median r4
SFT
No
No
62
0.046
SFT
No
Yes
279
0.046
SFT
Yes
No
40
0.474
SFT
Yes
Yes
53
0.429
RL
No
No
55
0.055
RL
No
Yes
285
0.047
Appendix
Table 14: Whole-thought repeated four-gram fraction by rescue status and final correctness. Continuation text is excluded.
Large language models (LLMs) achieve high scores on medical benchmarks, but does this performance translate into reliable diagnosis as a conversation unfolds? In diagnostic conversations, users can provide clinical findings across multiple turns, ask for a diagnosis without supplying answer options, and introduce opinions from others. Models must therefore form diagnoses from the available evidence and decide how to respond to subsequent advice. We introduce ROUNDS, a framework that evaluates these abilities after a fixed sequence of clinical evidence delivered over multiple turns. ROUNDS compares diagnosis with and without answer options, presents the same incorrect diagnosis under clinician and lay framing, and varies support for competing diagnoses in written five-clinician panels. Across seven models and 1,035 cases from five datasets, average accuracy was 76.9% with answer options and 57.0% without them. Clinician-framed misleading advice produced lower final accuracy than lay-framed advice in every model, with differences of 4.6 to 38.2 percentage points. Initially correct diagnoses more often became incorrect answers than abstentions, and many adopted the suggested disease. Conversely, accuracy increased as more panel members supported the correct diagnosis, including when the majority opinion remained unchanged. These findings reveal a gap between medical benchmark performance and reliable diagnosis in conversation: models respond to external opinions, but remain vulnerable to abandoning correct diagnoses without new clinical evidence.
Jinrui Fang, Runhan Chen, Yining Wu +6
School of Information, The University of Texas at Austin · Peter O’Donnell Jr. School of Public Health, The University of Texas Southwestern Medical Center · Department of Computer Science, The University of Texas at Austin +2
Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn, yet how they behave under multi-turn evidence accumulation closer to real clinical reasoning remains unexplored. We introduce MINT (Medical Incremental N-Turn Benchmark), a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases with clinically labeled evidence shards, controlled turn granularity, and information-preserving decomposition. Through systematic evaluation of 11 LLMs on MINT, we uncover three persistent behavioral patterns that significantly impact diagnostic decisions: (1) intent to answer, models rush to answer before sufficient evidence has been observed, with over 55% of answers committed within the first two turns; (2) self-correction, incorrect-to-correct answer revisions occur at up to 10.6 times the rate of correct-to-incorrect flips, revealing a latent capacity for self-correction that premature commitment forecloses; and (3) strong lures, clinically salient information such as laboratory results trigger premature answering even when models are explicitly instructed to wait. We translate these findings into clinically actionable guidance: deferring the diagnostic question to later turns reduces premature answering and improves accuracy at the first point of commitment by up to 62.6%, while reserving salient clinical evidence for later turns prevents a catastrophic accuracy drop of up to 23.3% caused by premature commitment. Our work provides both a controlled evaluation framework and concrete recommendations for improving the reliability of LLMs in multi-turn medical diagnosis.
Jinrui Fang, Runhan Chen, Xu Yang +9
The University of Texas at Austin · New York University · The University of Texas Southwestern Medical Center +2
Large language models achieve strong scores on medical benchmarks, yet these benchmarks evaluate each question in isolation, providing no measure of whether a system can distinguish clinically similar presentations requiring different interventions. We introduce MamaBench, the first counterfactual benchmark for maternal and paediatric AI: 434 expert-authored clinical narratives in 217 pairs across 371 pathologies, evaluated via the Bias Trap Rate (BTR), the conditional probability that a model fails the counterfactual given success on the base case. We propose Evidence-Anchored RAG (EA-RAG), a three-stage retrieval method that replaces aggregate similarity with an evidence coverage objective through clinical parameter extraction, coverage auditing, and contrastive sub-queries. Across eight configurations of four frontier LLMs, base accuracy overstates robust accuracy by 16-28 percentage points in every model. EA-RAG achieves 20.3% BTR and 65.0% robust accuracy on Claude Sonnet 4.6, a 5.5 percentage point BTR reduction without degrading base accuracy. The residual 20% BTR confirms that counterfactual robustness in clinical AI remains an open challenge. Keywords: counterfactual evaluation, clinical AI, maternal healthcare, retrieval-augmented generation, diagnostic robustness