MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors

Xiaotian Luo; Xun Jiang; Jiangcheng Wu

cs.CLApr 8, 2026

MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors

Authors: Xiaotian Luo, Xun Jiang, Jiangcheng Wu

Organizations: Shanda Group

Abstract

Interactive medical dialogue benchmarks have shown that LLM diagnostic accuracy degrades significantly when interacting with non-cooperative patients, yet existing approaches either apply adversarial behaviors without graded severity or case-specific grounding, or reduce patient non-cooperation to a single ungraded axis, and none analyze cross-dimension interactions. We introduce MedDialBench, a benchmark enabling controlled, dose-response characterization of how individual patient behavior dimensions affect LLM diagnostic robustness. It decomposes patient behavior into five dimensions -- Logic Consistency, Health Cognition, Expression Style, Disclosure, and Attitude -- each with graded severity levels and case-specific behavioral scripts. This controlled factorial design enables graded sensitivity analysis, dose-response profiling, and cross-dimension interaction detection. Evaluating five frontier LLMs across 7,225 dialogues (85 cases x 17 configurations x 5 models), we find a fundamental asymmetry: information pollution (fabricating symptoms) produces 1.7-3.4x larger accuracy drops than information deficit (withholding information), and fabricating is the only configuration achieving statistical significance across all five models (McNemar p < 0.05). Among six dimension combinations, fabricating is the sole driver of super-additive interaction: all three fabricating-involving pairs produce O/E ratios of 0.70-0.81 (35-44% of eligible cases fail under the combination despite succeeding under each dimension alone), while all non-fabricating pairs show purely additive effects (O/E ~ 1.0). Inquiry strategy moderates deficit but not pollution: exhaustive questioning recovers withheld information, but cannot compensate for fabricated inputs. Models exhibit distinct vulnerability profiles, with worst-case drops ranging from 38.8 to 54.1 percentage points.

Figures & tables

Work	Dim. Isolation	Graded Severity	Interaction
AgentClinic	Partial †	No	No
MAQuE	Partial ‡	No	No
MedPI	No	No	No
MedDialogRubrics	N/A	N/A	N/A
MediQ	No	No	No
Q4Dx	Yes	Partial §	No

Table 1: Comparison along three analytical capabilities. † Binary on/off per bias. ‡ Fixed sequential addition. § Single-axis (exposure rate). ∥ Utterance-level annotation; dialogues may contain multiple co-occurring behaviors.

Figure 1: Left: Behavioral framework with five dimensions spanning three degradation pathways, each with graded severity levels. Right: Dialogue examples illustrating adversarial behaviors from different pathways.

Model	Acc%	Turns	Coverage	Cov/Turn
Gemini 3.1 Pro †	90.6	10.9	.732	.072
GPT-5.4	82.4	17.5	.793	.051
Claude Opus 4.6	77.6	9.4	.744	.083
DeepSeek V3.2	74.1	18.2	.832	.047
Qwen 3.5 Plus	69.4	12.3	.723	.065

Table 2: Baseline performance (85 cases). Cov/Turn = coverage per turn (inquiry efficiency). † Thinking enabled (HIGH).

Expression (Friction)
Config	GPT-5.4	Claude	DeepSeek	Gemini	Qwen
Baseline	82.4	77.6	74.1	90.6	69.4
vague	81.2	84.7	69.4	89.4	72.9
incoherent	81.2	82.4	72.9	92.9	76.5
Cognition (Pollution)
partial_understanding	76.5	75.3	61.2	85.9	65.9

Table 3: Diagnostic accuracy (%) by single-dimension configuration. Bold : fabricating, the only universally significant configuration.

Figure 2: Accuracy (top) and information coverage (bottom) heatmaps across 5 models, 5 dimensions, and 3 severity levels (B=Baseline, M=Moderate, E=Extreme). Pollution dimensions show disproportionate accuracy drops relative to coverage loss; deficit dimensions show the reverse pattern.

Figure 3: Dose-response curves for the two pollution dimensions. Cognition (left) shows monotonic degradation with largely preserved model ranking. Logic (right) displays a threshold effect at the extreme level, with rank convergence—the best-to-worst gap narrows from 21.2 pp to 14.1 pp.

Combo	Type	Mean Acc	Mean $\Delta$	O/E
C1: fab.+with.	Fab $\times$ Deficit	40.2	$-$ 38.6	0.81
C2: fab.+incoh.	Fab $\times$ Friction	40.9	$-$ 37.9	0.70
C3: fab.+den.	Fab $\times$ Pollution	37.4	$-$ 41.4	0.78
C4: with.+dom.	Deficit $\times$ Deficit	67.3	$-$ 11.5	1.09
C5: with.+incoh.	Deficit $\times$ Friction	69.2	$-$ 9.6	0.97
C6: with.+den.	Pollution $\times$ Deficit	60.0	$-$ 18.8	0.99

Table 4: Multi-dimension combinations: mean accuracy across 5 models, drop from baseline, and O/E ratio (

<

1.0 = super-additive). Fab = fabricating.

Figure 4: Per-model O/E ratios for six multi-dimension combinations. Fabricating-involving combinations (C1–C3) consistently fall below 1.0 (super-additive), while all non-fabricating combinations (C4–C6) cluster at or above 1.0—including C6, which involves another pollution dimension (denial).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Script Adherence	Content Filter
Claude Opus 4.5	9/9	Pass
Qwen3-Max	6/9	Pass
Gemini 3 Pro	5/9	Pass
GPT-5.1	2/9	Pass

Appendix

Table 5: Patient agent candidate evaluation. Adherence = number of case-specific behavioral checkpoints correctly executed (out of 9). Results are consistent across both pilot cases (Case 004: DLBCL; Case 011: hemorrhoids); Case 004 shown as representative.

Model	Correct Acc.	Misled Acc.	Reps
Qwen3-Max	100%	100%	5
Gemini 3 Pro	100%	80%	1
GPT-5.1	100%	20%	3

Appendix

Table 6: Judge candidate evaluation on 15 human-annotated dialogues. Correct Acc. = agreement with human labels on diagnostic accuracy. Misled Acc. = agreement on whether fabricated/withheld information causally contributed to diagnostic error.

Config Type	N	Activation	Isolation	Full Pass
Baseline	30	N/A	100%	100%
Single-dimension	44	95.5%	100%	95.5%
Dual-dimension	26	92.3%	100%	92.3%
All non-baseline	70	94.3%	100%	94.3%

Appendix

Table 7: Behavioral adherence results. Activation = strict full-pass rate (all activated dimensions exhibited). Mean activation score (with partial credit) was 97.1%.

Config	Correct?	Models correct	Models wrong	Wrong diagnosis
Baseline	5/5	All	—	—
Fabricating	5/5	All	—	—
Withholding	5/5	All	—	—
C1 (both)	1/5	Qwen	4	TIA

Appendix

Table 8: Case 079 diagnostic outcomes across configurations.

		Expr.		Cog.		Logic		Disc.		Att.		Combos
Model	BL	vag	inc	par	den	occ	fab	rel	wit	imp	dom	C1	C2	C3	C4	C5	C6
GPT-5.4	82.4	81.2	81.2	76.5	71.8 ∗	78.8	63.5 ∗	80.0	74.1	74.1	75.3	47.1 ∗	51.8 ∗	40.0 ∗	72.9	70.6	68.2
Claude	77.6	84.7	82.4	75.3	69.4	77.6	55.3 ∗	80.0	68.2	76.5	69.4	29.4 ∗	41.2 ∗	36.5 ∗	63.5 ∗	70.6	60.0 ∗
DeepSeek	74.1	69.4	72.9	61.2 ∗	55.3 ∗	65.9	54.1 ∗	68.2	62.4 ∗	63.5 ∗	61.2 ∗	38.8 ∗	28.2 ∗	38.8 ∗	62.4 ∗	62.4	44.7 ∗
Gemini	90.6	89.4	92.9	85.9	83.5	85.9	60.0 ∗	87.1	76.5 ∗	83.5	82.4	52.9 ∗	52.9 ∗	36.5 ∗	74.1 ∗	81.2	76.5 ∗
Qwen	69.4	72.9	76.5	65.9	62.4	70.6	49.4 ∗	64.7	63.5	63.5	64.7	32.9 ∗	30.6 ∗	35.3 ∗	63.5	61.2	50.6 ∗

Appendix

Table 9: Diagnostic accuracy (%) across all 17 configurations and 5 models. BL = baseline. Single-dim abbreviations: vag = vague, inc = incoherent, par = partial understanding, den = complete denial, occ = occasional contradiction, fab = fabricating, rel = reluctant, wit = withholding, imp = impatient, dom = dominant. ∗ McNemar

p<0.05

vs. baseline.

Adversarial Robustness of Language Models

Adversarial robustness of language models studies how to keep their outputs and decisions reliable under manipulative prompts, input changes, and malicious fine-tuning.

See all 191 papers

Mar 31, 2026

Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultations

Large language models (LLMs) are increasingly used for medical consultation and health information support, where safety depends not only on medical knowledge but also on robust responses to unclear, inconsistent, or misleading patient input. However, most existing medical LLM evaluations assume idealized and well-posed patient questions, limiting their realism. We study challenging patient behaviors that commonly arise in real medical consultations and complicate safe clinical reasoning. We define four clinically grounded categories of such behaviors: information contradiction, factual inaccuracy, self-diagnosis, and care resistance. For each behavior, we specify concrete failure criteria that capture unsafe responses. Building on four existing medical dialogue datasets, we introduce CPB-Bench (Challenging Patient Behaviors Benchmark), a bilingual (English and Chinese) benchmark of multi-turn dialogues annotated for these behaviors. We find that although models perform well overall, they exhibit consistent behavior-specific failures, especially when handling contradictory or medically implausible patient information. We further evaluate four intervention strategies and find inconsistent improvements, with some interventions introducing unnecessary corrections.

Yahan Li, Xinyi Jie, Wanjia Ruan +5

University of Southern California

Healthcare Clinical Reasoning LLM Safety Evaluation Clinical Language Model Evaluation Language Model Robustness Medical LLMs

Ask PDF

Jul 23, 2026

Auditing Evidence Use in Medical LLM Diagnosis

Medical LLMs are often evaluated by whether they select the correct diagnosis, but diagnostic accuracy alone does not show whether the model used the case evidence appropriately. We present a behavioral audit of evidence use in medical diagnosis. For each case, we decompose patient information into evidence units, score candidate diagnoses under controlled evidence subsets, and mine low-order interactions in diagnostic margins. Because medical evidence is diagnosis-relative, the audit separates interaction discovery from failure assignment: large or negative interactions can reflect plausible differential diagnosis, while suspicious interactions require robustness checks and clinical review. We evaluate five open-weight LLMs on DDXPlus, CupCase, and MedCase. Across datasets, faithful support and differential conflict or cancellation account for most interaction strength, showing that many evidence interactions are clinically plausible rather than failures. In a DDXPlus-focused blinded five-reviewer 130-item enriched review sample, invalid or shortcut-like cases concentrate in negated or absent findings and clinically local evidence. These results show that accuracy can hide candidate evidence-use failures and motivate role-aware audits for medical LLM evaluation.

Junchi Liao, Jiawen Deng, Fuji Ren

University of Electronic Science and Technology of China Chengdu, China

LLM Auditing Medical Diagnosis Clinical Reasoning Differential Diagnosis

Ask PDF

Sep 29, 2026

KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy

Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients present information in different ways, and clinicians must obtain relevant history and determine which examinations are needed before reaching a diagnosis. Diagnostic accuracy alone therefore cannot establish whether an agent gathered essential information or conducted an appropriate clinical assessment. Furthermore, existing benchmarks lack professional clinicians' verification. To address this gap, we introduce KlinikeBench, a benchmark of 333 clinician-authored tasks, each providing an isolated sandbox environment with a virtual patient, clinical tools, and task-specific success criteria. More than 35 clinicians contributed to case authoring and benchmark evaluation. In an empirical study, clinicians gave simulated dialogues higher mean quality ratings than reference conversations, which is adapted from real conversation. In each task, an LM has a fixed budget of turns to communicate with the patient, ask about relevant history, request examinations, follow action constraints, and record a final diagnosis. We score these steps separately as well as together. Across 31 models and seven model families, the best-performing models (e.g., GPT-6-astra and Claude Opus 5) succeed on less than 30% of tasks, even though their diagnosis accuracy reaches 90.7%. Some models benefit from talking with the patient; others diagnose well from a complete chart but perform much worse in conversation. Overall, KlinikeBench provides a testbed for evaluating the full clinical encounter and reveals a substantial gap between diagnostic accuracy and performance in interactive clinical assessment. All the code and data is available on https://zehui127.github.io/klinikebench/

Xueting Fang, Zehui Li, Yang Yang +5

Zhejiang University · Imperial College London · Nanchang University +2

Clinical Decision-Making Multi-Turn Dialogue Evaluation Clinical Reasoning Clinical Language Model Evaluation

Ask PDF

Abstract

Adversarial Robustness of Language Models

Explore similar work

Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultations

Auditing Evidence Use in Medical LLM Diagnosis

KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy