MedMASLab: A Unified Orchestration Framework for Benchmarking Multimodal Medical Multi-Agent Systems
Authors: Yunhang Qian, Xiaobin Hu, Jiaquan Yu, Siyang Xin, Xiaokun Chen, Jiangning Zhang, Peng-Tao Jiang, Jiawei Liu, +1 more
Organizations: National University of Singapore · University of Science and Technology of China · Fudan University · Stanford University · Zhejiang University · vivo
While Multi-Agent Systems (MAS) show potential for complex clinical decision support, the field remains hindered by architectural fragmentation and the lack of standardized multimodal integration. Current medical MAS research suffers from non-uniform data ingestion pipelines, inconsistent visual-reasoning evaluation, and a lack of cross-specialty benchmarking. To address these challenges, we present MedMASLab, a unified framework and benchmarking platform for multimodal medical multi-agent systems. MedMASLab introduces: (1) A standardized multimodal agent communication protocol that enables seamless integration of 11 heterogeneous MAS architectures across 24 medical modalities. (2) An automated clinical reasoning evaluator, a zero-shot semantic evaluation paradigm that overcomes the limitations of lexical string-matching by leveraging large vision-language models to verify diagnostic logic and visual grounding. (3) The most extensive benchmark to date, spanning 11 organ systems and 473 diseases, standardizing data from 11 clinical benchmarks. Our systematic evaluation reveals a critical domain-specific performance gap: while MAS improves reasoning depth, current architectures exhibit significant fragility when transitioning between specialized medical sub-domains. We provide a rigorous ablation of interaction mechanisms and cost-performance trade-offs, establishing a new technical baseline for future autonomous clinical systems. The source code and data is publicly available at: https://github.com/NUS-Project/MedMASLab/
Figures & tables
Figure 1: MedMASLab, the first unified orchestration framework designed for medical visual-language multi-agent systems.
No.
Method
Interaction
Role
Tool
Adaptivity
Decision
Retrieval
Single-Agent Baselines
\scriptsize1⃝
VLM (Vanilla)
Single
Fixed
No
No
Direct Output
No
\scriptsize2⃝
CoT [ 36 ]
Single
Fixed
No
No
Direct Output
No
Multi-Agent Systems for General Tasks
\scriptsize3⃝
Self-Consistency [ 34 ]
Independent
Fixed
No
No
Voting
No
\scriptsize4⃝
Debate [ 9 ]
Debate
Fixed
No
Iterative
Consensus
No
Table 1: Taxonomy of MAS methods and their design dimensions: 1. Interaction: Inter-agent communication pattern. 2. Role: Fixed vs. dynamic agent identities. 3. Tool: Yes/No. 4. Adaptivity: Feedback-driven self-tuning. 5. Decision: Consensus rule. 6. Retrieval: Access to external knowledge/search.
Figure 2: Framework of MedMASLab.
Figure 3: Performance and ranking variations of different MAS methods across five evaluation protocols on DxBench, PubMedQA, and MedXpertQA.
Method
PubMedQA
MedQA
MedBullets
MMLU
VQA-RAD
SLAKE-En
MedVidQA
MedCMR
MedXpertQA-MM
DxBench
M3CoTBench
Avg-V
Qwen-2.5VL-7B-Instruct
Single [ 2 ]
68
52.8
35.7
75.2
50.4
58.3
71.6
68.1
20.8
62.9
30.8
54.1
Debate [ 9 ]
68.4
52.9
37.1
76.6
54.1
64.4
76.4
64.5
21.6
64.2
34
55.9
MDAgents [ 21 ]
68
52.3
38.4
73.9
56.6
63.8
79.1
68.9
22.6
64.7
36.8
56.8
MDTeamGPT [ 6 ]
79.4
56.1
39
77.6
50.3
58.3
71.6
62.7
23.4
64.9
34.6
56.2
Discussion [ 28 ]
56
52.3
35.2
74
57.3
65.3
75
65.9
23.3
61.5
31.8
54.3
Table 2: We compare the performance of general-task and medicine-specific methods across five aspects in the medical domain ( Medical Literature Reasoning , Medical Visual Understanding and Reasoning , Medical Question Answering , Diagnosis Decision , and Evaluation of Medical Reasoning Chains ). Avg-V denotes the average accuracy ( ↑ ). Best and second-best numbers are highlighted.
Figure 4: Trade-off between performance and token cost of Qwen2.5VL-7B-Instruct based Multi-Agent Methods across Medical Benchmarks.
Figure 5: Comparison of Method Performance and Token Cost using Qwen2.5VL-7B, LLaVA-7B, and GPT-4o-mini as Backbones on MedQA and MedVidQA.
Figure 6: Trade-off under MAS Compute Scaling on MedQA and MedVidQA.
Figure 7: Performance and Token Cost of Debate and MDTeamGPT under Different Medical Expert-Playing Modes.
Figure 8: Trade-off under MAS Model Size Scaling on MedQA and MedXpertQA-MM.
Base Model
Dataset
Right Answer
Wrong Answer
Format Error
Others
Qwen2.5VL-3B
MedVidQA
40.62%
53.38%
6.00%
0.00%
SLAKE-En
55.54%
43.36%
0.90%
0.20%
PubMedQA
3.00%
13.00%
84.00%
0.00%
MedQA
10.53%
14.14%
75.33%
0.00%
Qwen2.5VL-7B
MedVidQA
71.89%
28.11%
0.00%
0.00%
SLAKE-En
58.72%
41.28%
0.00%
0.00%
Table 3: Distribution of outcomes for the Reconcile framework using Qwen2.5VL series models as the base model. The table compares the proportions of Right Answer, Wrong Answer, Format Error, and Others across multiple medical datasets for 3B and 7B model variants.
Figure 9: Error analysis of MDTeamGPT on MedQA using LLaVA-1.6-7b as the model backend.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Modality
Choice
Testing Size
Domain
MedQA
T
MCQ
1273
US Medical Licensing Examination
PubMedQA
T
MCQ
500
Medical Academic Question Answering
MedBullets
T
MCQ
308
Online Platform Resources for Medical Study
MMLU
T
MCQ
1089
Graduate Record Examination and US Medical Licensing Examination
SLAKE-En
T, I
MCQ+Open-ended
1061
VQA Pairs Including CT, MRI and X-ray
VQA-RAD
T, I
MCQ+Open-ended
451
VQA Pairs in Radiology
Appendix
Table 4: Summary of the Medical Datasets. T: Text, I: Image, V: Video.
Figure 10: Simplified framework diagrams of DyLAN, Debate, Reconcile, Discussion, AutoGen, MDAgents, MedAgents, MDTeamGPT, ColaCare, and Meta-Prompting.
Figure 11: Trade-off between performance and token cost of Qwen2.5VL-7B-Instruct based Multi-Agent Methods across remaining Medical Benchmarks. We compute the average performance and average token cost across 11 datasets.
Figure 12: Performance and Token Cost of MedAgents under Different Medical Expert-Playing Modes.We use Qwen2.5VL-7B as the base model for MedAgents to investigate the performance and token cost of MAS frameworks on the PubMedQA, MedQA, MedVidQA, DxBench, and M3CoTBench datasets under three modes: no medical expert-playing, fixed medical expert-playing, and dynamic medical expert-playing.
Figure 13: Backbone Model Size Scaling of Multi Agent Frameworks on PubMedQA, DxBench and MedVidQA with Performance and Token Cost Comparisons. Beyond our previous findings, we discover that increasing model scale does not necessarily lead to performance improvements for both multi-agent frameworks and single-agent systems. For instance, on the PubMedQA dataset at the 72B model scale, the performance of all MAS methods and single-agent models (except for the MedAgents framework) is lower than their 32B counterparts. Notably, MAS demonstrates an amplification effect on performance variations observed in single-agent models . For example ,improvements in single-agent performance are magnified in MAS, while performance degradation is similarly amplified.
Figure 14: The API Setup interface, allowing decoupled configuration of Base and Judge models via OpenAI-compatible endpoints.
Figure 15: The Guide module, offering comprehensive documentation on supported MAS methods, dataset schemas, and tutorials.
Figure 16: The Quick Test interface for single-sample multimodal inference and real-time performance profiling.
Figure 17: The Batch Evaluation module, orchestrating large-scale dataset testing with dynamic algorithmic parameter tuning.
Figure 18: The Custom MAS Builder, providing a drag-and-drop canvas for topological design and automated Python code generation.