A Structured Debate-Mixture-of-Agents Framework for Complex Clinical Diagnostic Decision Support
Authors: Chang Xia, Leilei Ouyang, Huimin Wang, Yong Zhao, Kang Li
Organizations: College of Computer Science, Sichuan University, Chengdu 610207, China · West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu 610041, China · Med-X Center for Informatics, Sichuan University, Chengdu 610041, China
Large language models (LLMs) show potential for medical tasks, but their single-turn question-answer format does not reflect how clinical diagnosis is performed in practice. As a result, they remain limited in complex diagnostic settings. We developed Debate-Mixture-of-Agents (DMoA), a novel multi-agent framework that structures role-based interaction to support iterative diagnostic reasoning. Base models and DMoA were evaluated on 297 rare disease cases and 1,719 challenging cases. Across both datasets, DMoA improved most likely diagnosis accuracy by 10.21 percentage points and safety rate by 11.36 percentage points over GPT-4o baseline. Ablation experiments showed that the gains were not simply due to the use of more models or longer outputs, but also reflected the contribution of the structured workflow. Further analyses examined how framework design, base model choice, and token budget affected performance. DMoA performed better with a 4*2 structure, stronger base models, and a larger token budget. These findings demonstrate the potential of DMoA for clinical tasks and suggest further investigation of multi-agent frameworks.
Figures & tables
Fig. 1: Research process. Researchers first curated and established the database. Subsequently, the question-answering and diagnostic tasks were conducted using single models and DMoA. The resulting outputs were then subjected to task-specific evaluation.
Fig. 2: The Debate-Mixture-of-Agents framework. DMoA uses a fixed role-constrained topology rather than an open-ended conversation. Each Debater performs generation, rebuttal, and revision, while the Aggregator integrates heterogeneous revised hypotheses into the final diagnosis.
Category
Question
Criteria
Correctness or Appropriateness
Is the diagnosis clinically correct and appropriate?
1: Completely incorrect/inappropriate with great clinical significance 2: Mostly incorrect/inappropriate with noticeable clinical significance 3: Partly correct with some clinically significant errors 4: Minor inaccuracies with little clinical significance 5: Fully correct and appropriate
Risk of critical misses
Are any critical diagnoses or red flags missed?
1: Substantial critical omissions with great clinical significance 2: Significant critical omissions with noticeable clinical significance 3: Moderate critical omissions with some clinical significance 4: Minor omissions with little clinical significance 5: No critical omission, or omissions of no clinical significance
Risk of false alarms or overdiagnosis
Does the output promote unnecessary testing?
1: Strongly promotes unnecessary escalation with great clinical significance 2: Often promotes unnecessary escalation with noticeable clinical significance 3: Sometimes promotes unnecessary escalation with some clinical significance 4: Minimally promotes unnecessary escalation with little clinical significance 5: No unnecessary escalation; recommendations are proportionate and justified
Demographic applicability or potential bias
Is it applicable across demographics?
1: Not applicable/biased with great clinical significance 2: Largely not applicable/biased with noticeable clinical significance 3: Somewhat not applicable/biased with some clinical significance 4: Slightly not applicable/biased with little clinical significance 5: Fully applicable with no bias
Calibration or Uncertainty handling
Is confidence appropriately calibrated?
1: Severely miscalibrated with great clinical significance 2: Poorly calibrated with noticeable clinical significance 3: Moderately calibrated with some clinical significance 4: Well calibrated with little clinical significance 5: Well calibrated; uncertainty and safety-netting are appropriate
TABLE I: Rubric for safety evaluation.
Evaluator
Format
Accuracy
Safety
Agree. (95% CI)
Kappa
Agree. (95% CI)
Kappa
GPT-4o
Most likely
93.29% ± 1.55%
0.766
94.29% ± 1.44%
0.829
GPT-4o
Three possible
86.88% ± 2.61%
0.607
93.89% ± 1.48%
0.842
Gemini-2.5 Flash
Most likely
96.20% ± 1.19%
0.853
90.40% ± 1.83%
0.795
Gemini-2.5 Flash
Three possible
85.20% ± 2.20%
0.632
89.90% ± 1.87%
0.769
TABLE II: Agreement between GPT-4o mini and independent LLM evaluators.
Base model
Prompt/Structure
Acc. Top-1
Safety Top-1
Acc. Top-3
Safety Top-3
Compare group
GPT-4o mini
General
11.78%
38.72%
24.58%
41.75%
GPT-3.5
General
22.22%
59.80%
34.68%
63.97%
GPT-4o
General
28.62%
58.92%
46.46%
63.30%
DMoA framework
GPT-4o mini
4×2
22.22%
47.47%
34.01%
52.19%
TABLE III: Performance of single-model, DMoA, and ablation settings in rare disease sub dataset.
Base model
Prompt/Structure
Acc. Top-1
Safety Top-1
Acc. Top-3
Safety Top-3
Compare group
GPT-4o mini
General
13.80%
52.71%
22.71%
61.50%
GPT-3.5
General
13.28%
60.94%
23.40%
71.93%
GPT-4o
General
14.50%
63.46%
24.00%
72.86%
DMoA framework
GPT-4o mini
4×2
20.35%
70.12%
37.31%
72.63%
TABLE IV: Performance of single-model, DMoA, and ablation settings in challenging cases sub dataset.
Fig. 3: Performance of DMoA compared with single model. (a) Accuracy in the whole dataset; (b) Accuracy in the rare disease sub dataset; (c) Accuracy in the challenging case sub dataset. The bars represent percentages.
Fig. 4: Performance of DMoA compared with single model. (a) , (b) and (c) Safety rate of GPT-4o mini, GPT-3.5 and GPT-4o based single model and DMoA; (d) , (e) and (f) The detailed differences of the corresponding model in the four or five safety dimensions. The bars represent percentages. In (d) , (e) , (f) , positive values in matrices indicate higher safety scores for DMoA.
Fig. 5: Performance of DMoA and three ablation configurations. (a) Performance in rare disease sub dataset; (b) Performance in challenging cases sub dataset. The bars represent percentages.
Fig. 6: Effects of configuration factors and lightweight settings on DMoA. (a) Lightweight-model setting; (b) effect of token budget; (c) effect of information partitioning; (d) effect of DMoA structure. The bars and lines represent percentages.
Model
Structure
Found. Science
Practice Learning
Professional Ethics
Patient Safety
Total
Single model
GPT-4o mini
NA
76.23%
74.48%
76.87%
73.97%
75.90%
GPT-3.5
NA
78.80%
81.82%
80.44%
78.80%
79.90%
GPT-4o
NA
87.53%
87.88%
89.38%
84.93%
88.00%
DMoA
GPT-4o mini
4×2
73.57%
73.84%
75.08%
73.52%
74.10%
TABLE V: Accuracy of single-model and DMoA in clinical Q&A.