Patient-clinician communication is an asymmetric-information problem: patients often do not disclose fears, misconceptions, or practical barriers unless clinicians elicit them skillfully. Effective medical dialogue therefore requires reasoning under partial observability: clinicians must elicit latent concerns, confirm them through interaction, and respond in ways that guide patients toward appropriate care. However, existing medical dialogue benchmarks largely sidestep this challenge by exposing hidden patient state, collapsing elicitation into extraction, or evaluating responses without modeling what remains hidden. We present MedConceal, a benchmark with an interactive patient simulator for evaluating hidden-concern reasoning in medical dialogue, comprising 300 curated cases and 600 clinician-LLM interactions. Built from clinician-answered online health discussions, each case pairing clinician-visible context with simulator-internal hidden concerns derived from prior literature and structured using an expert-developed taxonomy. The simulator withholds these concerns from the dialogue agent, tracks whether they have been revealed and addressed via theory-grounded turn-level communication signals, and is clinician-reviewed for clinical plausibility. This enables process-aware evaluation of both task success and the interaction process that leads to it. We study two abilities: confirmation, surfacing hidden concerns through multi-turn dialogue, and intervention, addressing the primary concern and guiding the patient toward a target plan. Results show that no single system dominates: frontier models lead on different confirmation metrics, while human clinicians (N=159) remain strongest on intervention success. Together, these results identify hidden-concern reasoning under partial observability as a key unresolved challenge for medical dialogue systems.
Figures & tables
Figure 1: Framework overview. Cases are collected from real-world clinician-responded r/AskDocs conversations, filtered to longer common-condition exchanges, and converted into structured patient charts, intervention targets, and hidden-concern annotations with clinician-guided verification. The patient simulator exposes only clinician-visible information while keeping hidden concerns internal. In Task 1 (confirmation), the clinician must surface and extract fine-grained hidden concerns through natural dialogue rather than meta-category probing. In Task 2 (intervention), the clinician must address the primary concern and guide the patient toward the target plan. The same framework is used to compare AI clinician agents and human clinicians under matched observable information.
System
Setting
Reveal Rate ↑
Coarse P ↑
Coarse R ↑
Coarse F1 ↑
Fine P ↑
Fine R ↑
Fine F1 ↑
MBNR ↓
Human clinicians
Human summary
0.558
0.791
0.451
0.574
0.721
0.411
0.523
0.227
Claude summary
0.558
0.581
0.744
0.652
0.504
0.646
0.567
0.257
Qwen-3.5-9B
8-turn
0.431
0.861
0.129
0.225
0.740
0.111
0.193
0.057
Adaptive
0.934
0.452
0.284
0.349
0.102
0.064
0.079
0.006
Claude Sonnet 4.5
8-turn
0.522
0.608
0.631
0.619
0.494
0.513
0.503
0.270
Adaptive
0.542
0.573
0.676
0.620
0.470
0.555
0.509
0.297
Table 1: Task 1 (confirmation) performance metrics for humans and AI clinicians. Metrics follow Section 3 : Reveal Rate, Coarse Precision/Recall/F1, Fine Precision/Recall/F1, and MBNR (matched-but-no-reveal rate). The two human rows report the original human submitted findings and a hybrid reference that uses Claude Sonnet 4.5 summarization over the same human dialogues for the submitted-finding metrics. For each AI model, the first row uses a fixed 8-turn setting and the second row uses the adaptive confirmation policy with a hard cap of 20 turns. Blue : best under matched-length comparison when it differs from the overall best; green : best overall.
System
Setting
Empathy ↑
Collaboration ↑
Problem-Solving ↑
Values Det. ↑
Early Open ↑
Readability ↑
Word/Turn
Human clinicians
—
1.263
1.073
0.833
2.663
0.477
63.393
23.706
Qwen-3.5-9B
8-turn
1.487
1.040
0.447
1.887
0.279
59.141
25.054
Adaptive
2.000
1.760
1.347
3.247
0.458
53.875
73.161
Claude Sonnet 4.5
8-turn
1.900
1.473
0.890
2.353
0.447
57.465
68.227
Adaptive
1.930
1.670
1.130
2.693
0.441
57.261
69.759
GPT-5.2
8-turn
1.880
1.543
0.990
2.543
0.405
56.845
51.471
Table 2: Task 1 (confirmation) communication-style metrics for humans and AI clinicians. Metrics follow Section 3 : Empathy, Collaboration, Problem-Solving, Values Det. (mean number of patient values detected), Early Open, Readability, and Word/Turn. AI row ordering and color coding follow Table 1 .
System
Setting
Success ↑
Reveal Rate ↑
Turn-to-Address ↓
Reveal-to-Address ↓
Meta-Probe ↓
Human clinicians
—
0.427
0.487
7.125
3.336
0.018
Qwen-3.5-9B
8-turn
0.030
0.063
6.333
3.000
0.417
20-turn
0.073
0.073
8.591
3.045
0.006
Claude Sonnet 4.5
8-turn
0.293
0.313
5.239
3.080
0.005
20-turn
0.333
0.337
5.960
3.040
0.004
GPT-5.2
8-turn
0.077
0.087
5.043
3.000
0.016
Table 3: Task 2 (intervention) performance metrics for humans and AI clinicians. Metrics follow Section 3 : Success, Reveal Rate, Turn-to-Address, Reveal-to-Address, and Meta-Probe. For every AI model, rows correspond to matched 8-turn and full 20-turn settings. Color coding follows Table 1 .
System
Setting
Empathy ↑
Rationale ↑
Problem-Solving ↑
Actionability ↑
Early Open ↑
Readability ↑
Word/Turn
Human clinicians
—
1.441
1.722
0.800
1.159
0.423
58.831
30.315
Qwen-3.5-9B
8-turn
1.537
1.550
0.657
1.030
0.279
53.901
29.484
20-turn
1.973
1.947
1.133
1.837
0.391
53.070
76.421
Claude Sonnet 4.5
8-turn
1.900
1.883
0.890
1.207
0.455
57.465
68.227
20-turn
1.997
1.980
1.113
1.790
0.475
62.121
70.100
GPT-5.2
8-turn
1.953
1.983
1.220
1.657
0.281
56.845
51.471
Table 4: Task 2 (intervention) communication-style metrics for humans and AI clinicians. Metrics follow Section 3 : Empathy, Rationale, Problem-Solving, Actionability, Early Open, Readability, and Word/Turn. Rows correspond to matched 8-turn and full 20-turn runs for every AI model. Color coding follows Table 1 .
Figure 2: Cumulative performance over turns for the AI clinician baselines. Panel (a) shows cumulative confirmation Reveal Rate, and panel (b) shows cumulative intervention Success. Dashed horizontal lines mark the final observed human reference level, and dotted vertical lines mark the mean observed human conversation length (8.2 turns in both tasks). Average realized dialogue lengths for all AI conditions are reported in Appendix Table 7 .
Outcome cell
n
Concern probe (%)
Reassure+plan (%)
Turns
H / D
H / D
H / D
Both succeed
63
16.8 / 10.7
32.1 / 61.4
7.2 / 11.2
Human only
65
19.2 / 0.8
33.2 / 89.4
7.9 / 19.3
Doctor-R1 only
65
6.0 / 12.6
42.9 / 60.2
8.8 / 11.2
Both fail
107
5.7 / 1.3
55.3 / 89.8
8.7 / 18.5
Table 5: Outcome-conditioned intervention behavior for humans (H) and Doctor-R1 at 20 turns (D). Intent percentages are shares of clinician turns labeled as concern probes or as reassurance/plan instruction.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Best-fit specialty
Cases
Unmapped medical specialty
51
Gastroenterologist
40
Family Physician / General Practitioner
29
Dermatologist
28
Cardiologist
22
Psychiatrist
20
Appendix
Table 6: Best-fit specialty distribution for the 300-case benchmark subset. “Unmapped” categories denote clinically appropriate specialties not explicitly represented in the study’s clinician-role taxonomy.
Figure 3: Login screen for the clinician study platform.
Figure 4: Consent-form screen shown before study entry.
Figure 5: Background-survey screen completed before the study tasks.
Figure 6: Confirmation-task chart view, which presents the clinician-visible case summary before the dialogue starts.
Figure 7: Confirmation-task dialogue page used for the chat-based interview and structured concern submission.
Figure 8: Intervention-task chart view, which presents the clinician-visible case summary together with the initial preference and target plan.
Figure 9: Intervention-task dialogue page used until acceptance or timeout.
Model / Condition
Confirmation
Intervention
Human clinicians
8.2
8.2
Qwen-3.5-9B
19.4
18.9
Claude Sonnet 4.5
11.7
15.4
GPT-5.2
11.7
18.6
Doctor-R1
18.1
15.5
Llama3-OpenBioLLM-8B
15.5
19.5
Appendix
Table 7: Average realized dialogue length by model and task. Human values are the observed study averages; AI values are the realized averages of the corresponding benchmark runs.
Metric
Best AI
Ref
AI
Δ
95% CI
Holm p
Fine R
Claude (Adaptive)
0.649
0.555
+0.094
[+0.060, +0.129]
0.0002***
Fine F1
Claude (Adaptive)
0.570
0.513
+0.058
[+0.028, +0.087]
0.0004***
Appendix
Table 8: Case-level comparison for the Claude summary reference row in Table 1 versus the best AI system under any confirmation setting. Only metrics with a human-reference advantage are shown.
Metric
Best AI
Ref
AI
Δ
95% CI
Holm p
Coarse P
GPT-5.2 (8-turn)
0.854
0.705
+0.149
[+0.108, +0.190]
0.0002***
Fine P
GPT-5.2 (8-turn)
0.788
0.623
+0.165
[+0.116, +0.212]
0.0002***
Appendix
Table 9: Case-level comparison for the Human summary reference row in Table 1 versus the best AI system in the matched 8-turn confirmation comparison. Only metrics with a human advantage are shown.
Metric
Best AI
Ref
AI
Δ
95% CI
Holm p
Fine P
GPT-5.2 (Adaptive)
0.788
0.663
+0.125
[+0.080, +0.169]
0.0002***
Appendix
Table 10: Case-level comparison for the Human summary reference row in Table 1 versus the best AI system under any confirmation setting. Only metrics with a human advantage are shown.
Metric
Best AI
Ref
AI
Δ
95% CI
Holm p
Coarse R
Claude (8-turn)
0.746
0.630
+0.115
[+0.087, +0.142]
0.0002***
Coarse F1
Claude (8-turn)
0.657
0.620
+0.037
[+0.013, +0.060]
0.0042**
Fine R
Claude (8-turn)
0.649
0.515
+0.134
[+0.101, +0.167]
0.0002***
Fine F1
Claude (8-turn)
0.570
0.509
+0.061
[+0.032, +0.090]
0.0002***
Appendix
Table 11: Case-level comparison for the Claude summary reference row in Table 1 versus the best AI system in the matched 8-turn confirmation comparison. Only metrics with a human-reference advantage are shown.
Metric
Best AI
Ref
AI
Δ
95% CI
Holm p
Reveal Rate
Claude (8-turn)
0.487
0.313
+0.173
[+0.100, +0.247]
0.0001***
Success
Claude (8-turn)
0.427
0.293
+0.133
[+0.060, +0.203]
0.0004***
Appendix
Table 12: Case-level comparison for the Human clinicians reference row in Table 3 versus the best AI system in the matched 8-turn intervention comparison. Only metrics with a human advantage are shown.
Family
Main-paper metrics
Literature grounding
Interpretation
Information structure
Word/Turn, Readability, Early Open
Health-information readability ( Daraz et al., 2018 ; Wallace and Lennon, 2004 )
Describes verbosity, readability, and early open-ended questioning without using hidden-state labels.
Clinical counseling and adherence-promotion skills ( Schlundt et al., 1994 )
Scores how well the clinician validates concerns, explains need, collaborates, addresses barriers, and gives actionable next steps.
Receiver value alignment
Values Det.
Value-framed health communication ( Heine and Wolters, 2021 ; Winters et al., 2024 )
Measures whether the clinician recognizes patient values explicitly stated in the conversation.
Appendix
Table 13: Provenance of the communication-style metric families used in the main paper.
Judgment
n
Cohen’s κ
Raw agreement
Reveal
80
0.725
0.863
Address
80
0.775
0.887
Overall
160
0.750
0.875
Appendix
Table 14: Blind agreement for per-turn reveal and address judgments.
Task / metric
Configuration
Doctor-R1
Qwen-3.5-9B
Ordering
Confirmation Coarse F1
Anchor
0.670
0.225
Doctor-R1 > Qwen
Reveal threshold 0.20→0.25
0.638
0.536
Doctor-R1 > Qwen
Reveal EMA 0.65→0.55
0.634
0.572
Doctor-R1 > Qwen
Concern-family deltas zeroed
0.616
0.570
Doctor-R1 > Qwen
Intervention Success
Anchor
0.150
0.030
Doctor-R1 > Qwen
Reveal threshold 0.20→0.25
0.023
0.017
Doctor-R1 > Qwen
Appendix
Table 15: End-to-end sensitivity to latent-policy choices. Confirmation ordering is preserved in every tested configuration. Intervention ordering is preserved in four of five configurations; zeroing concern-family deltas produces a small reversal at a near-zero floor.
Figure 10: Matched 8-turn reveal rate after recomputing the state trace over fixed transcripts across reveal thresholds. Lower thresholds represent more readily disclosing patients and higher thresholds more guarded patients. All systems decline as disclosure tightens, with the human trace most robust in the guarded regime. Because doctor and patient utterances are fixed in this diagnostic sweep, the absolute values are not directly comparable to the gold-matched headline Reveal Rate.
Figure 11: Performance by hidden-concern type. “Best baseline per type” selects the strongest evaluated model separately for each type and setting; it is not one fixed model. Extended confirmation uses the adaptive policy, and extended intervention uses 20 turns.
Severity ( n )
Human conf. Reveal
Human interv. Success
Doctor-R1 Success
Low (6)
0.625
0.667
0.333
Moderate (219)
0.541
0.411
0.416
High (75)
0.601
0.453
0.467
Appendix
Table 16: Outcomes by primary-concern severity. Doctor-R1 uses the 20-turn intervention setting. The low-severity stratum is too small for interpretation.
Contrast
Interv. Success
Interv. Reveal
Conf. Reveal
Conf. Coarse F1
Conf. Fine F1
Native / non-native English
0.482 / 0.405
0.529 / 0.470
0.688 / 0.504*
0.563 / 0.579
0.528 / 0.522
Early-career / senior
0.417 / 0.451
0.482 / 0.500
0.559 / 0.556
0.570 / 0.586
0.524 / 0.522
Appendix
Table 17: Human-reference subgroup outcomes. *The native/non-native confirmation Reveal contrast is significant ( p<0.001 ); the other nine contrasts are non-significant ( p>0.2 for scalar metrics or bootstrap 95% intervals include zero for F1).
Figure 12: Cumulative human-reference outcomes and the share of clinicians still active by turn limit. From turn 12 to 20, Reveal Rate increases from 0.552 to 0.558 and Success from 0.393 to 0.427, while the active share falls below one fifth after turn 12.
Symbol
Name
Framework
Interpretation
rtDG
Data gathering
RIAS
How strongly the turn elicits medically relevant information (questions, clarifications).
rtER
Emotional responsiveness
RIAS
Acknowledgment/validation of patient affect and concerns.
Degree to which the turn invites disclosure of worries/concerns.
rtSP
Space provision
VR-CoDES
Whether the turn leaves room to elaborate (non-cutting, open invitation).
rtNS
Necessity support
NCF
Strength of rationale for why the recommended plan is needed/beneficial.
Appendix
Table 18: Turn-level rubric feature vector zt used as model inputs. Framework-linked dimensions are adapted from RIAS, VR-CoDES, and the Necessity–Concerns Framework; task-rubric dimensions are project-defined.
Large language models (LLMs) are increasingly used for medical consultation and health information support, where safety depends not only on medical knowledge but also on robust responses to unclear, inconsistent, or misleading patient input. However, most existing medical LLM evaluations assume idealized and well-posed patient questions, limiting their realism. We study challenging patient behaviors that commonly arise in real medical consultations and complicate safe clinical reasoning. We define four clinically grounded categories of such behaviors: information contradiction, factual inaccuracy, self-diagnosis, and care resistance. For each behavior, we specify concrete failure criteria that capture unsafe responses. Building on four existing medical dialogue datasets, we introduce CPB-Bench (Challenging Patient Behaviors Benchmark), a bilingual (English and Chinese) benchmark of multi-turn dialogues annotated for these behaviors. We find that although models perform well overall, they exhibit consistent behavior-specific failures, especially when handling contradictory or medically implausible patient information. We further evaluate four intervention strategies and find inconsistent improvements, with some interventions introducing unnecessary corrections.
Interactive medical dialogue benchmarks have shown that LLM diagnostic accuracy degrades significantly when interacting with non-cooperative patients, yet existing approaches either apply adversarial behaviors without graded severity or case-specific grounding, or reduce patient non-cooperation to a single ungraded axis, and none analyze cross-dimension interactions. We introduce MedDialBench, a benchmark enabling controlled, dose-response characterization of how individual patient behavior dimensions affect LLM diagnostic robustness. It decomposes patient behavior into five dimensions -- Logic Consistency, Health Cognition, Expression Style, Disclosure, and Attitude -- each with graded severity levels and case-specific behavioral scripts. This controlled factorial design enables graded sensitivity analysis, dose-response profiling, and cross-dimension interaction detection. Evaluating five frontier LLMs across 7,225 dialogues (85 cases x 17 configurations x 5 models), we find a fundamental asymmetry: information pollution (fabricating symptoms) produces 1.7-3.4x larger accuracy drops than information deficit (withholding information), and fabricating is the only configuration achieving statistical significance across all five models (McNemar p < 0.05). Among six dimension combinations, fabricating is the sole driver of super-additive interaction: all three fabricating-involving pairs produce O/E ratios of 0.70-0.81 (35-44% of eligible cases fail under the combination despite succeeding under each dimension alone), while all non-fabricating pairs show purely additive effects (O/E ~ 1.0). Inquiry strategy moderates deficit but not pollution: exhaustive questioning recovers withheld information, but cannot compensate for fabricated inputs. Models exhibit distinct vulnerability profiles, with worst-case drops ranging from 38.8 to 54.1 percentage points.
Medical question-answering benchmarks predominantly evaluate single-turn exchanges, failing to capture the iterative, clarification-seeking nature of real patient consultations. We introduce ThReadMed-QA, a benchmark of 2,437 fully-answered patient-physician conversation threads extracted from r/AskDocs, comprising 8,204 question-answer pairs across up to 9 turns. Unlike prior work relying on simulated dialogues, adversarial prompts, or exam-style questions, ThReadMed-QA captures authentic patient follow-up questions and verified physician responses, reflecting how patients naturally seek medical information online. We evaluate five state-of-the-art LLMs -- GPT-5, GPT-4o, Claude Haiku, Gemini 2.5 Flash, and Llama 3.3 70B -- on a stratified test split of 238 conversations (948 QA pairs) using a calibrated LLM-as-a-judge rubric grounded in physician ground truth. Even the strongest model, GPT-5, achieves only 41.2% fully-correct responses. All five models degrade significantly from turn 0 to turn 2 (p < 0.001), with wrong-answer rates roughly tripling by the third turn. We identify a fundamental tension between single-turn capability and multi-turn reliability: models with the strongest initial performance (GPT-5: 75.2; Claude Haiku: 72.3 out of 100) exhibit the steepest declines by turn 2 (dropping 16.2 and 25.0 points respectively), while weaker models plateau or marginally improve. We introduce two metrics to quantify multi-turn failure modes: Conversational Consistency Score (CCS) and Error Propagation Rate (EPR). CCS reveals that nearly one in three Claude Haiku conversations swings between a fully correct and a completely wrong response within the same thread. EPR shows that a single wrong turn raises the probability of a subsequent wrong turn by 1.9-6.1x across all models.
Monica Munnangi, Saiph Savage
Khoury College of Computer Sciences Northeastern University Boston