Learning When to Collaborate: Selective Multi-Agent Medical Reasoning via Uncertainty-Aware Routing
Authors: Jiacheng Hou, Xueliang Cui, Dan Lu, Ruxin Wang
Organizations: SIGS, Tsinghua University · Southern University of Science and Technology · Shenzhen University of Advanced Technology, China · West China Hospital, Sichuan University, China. · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences
Multi-agent systems have emerged as a promising paradigm for multimodal medical reasoning, but existing methods rely on two assumptions that hinder practical deployment: (i) always-active dense collaboration among multiple agents, and (ii) dependence on large cloud-based models that raise privacy and efficiency concerns. In this paper, we challenge both assumptions and propose SMART-Med, a fully local multi-agent method that treats collaboration as a learnable decision rather than a default behavior. Our key insight is that not every medical query requires costly multi-agent deliberation: many can be reliably solved by a single well-trained agent, while only uncertain cases benefit from expert collaboration. Based on this insight, SMART-Med introduces an uncertainty-driven routing mechanism that adaptively decides when to answer directly and when to recruit a small subset of expert agents, effectively converting dense multi-agent reasoning into selective, query-adaptive coordination. To make this routing reliable, we curate a new multimodal medical dataset MedVR-3K with high-quality reasoning traces and a difficulty hierarchy, and then fine-tune local agents using GRPO with two complementary rewards: a certainty-aware accuracy reward and a vision-language reward model for correctness and reasoning quality respectively. Experiments on four medical visual question answering benchmarks show that SMART-Med achieves competitive performance compared to existing MAS approaches while substantially reducing inference cost (reducing inference time by more than 3.8x and token usage by 72%). Moreover, after fine-tuning on MedVR-3K, SMART-Med yields an average improvement of 6.7% across Slake, PathVQA, VQA-RAD, and PMC-VQA.
Figures & tables
Figure 1: Comparison between existing MAS and our proposed SMART-Med . (a)-(e) respectively depict the collaboration topology, paradigm, base model and training objective of different approaches.
Figure 2: Overview of SMART-Med . The framework unifies reasoning-centric training and uncertainty-aware routing in a closed loop: MedVR-3K and reward-guided optimization improve both reasoning capability and uncertainty calibration, while the learned router adaptively decides whether to answer directly or trigger selective expert collaboration.
Figure 3: Statistics of our constructed MedVR-3K . (a)-(d) respectively present the average quality score, distribution of difficulty level, response length and quality assessment length.
Category
Methods
Multiple-Choice Questions (MC)
Free-form Questions (FF)
Slake
PathVQA
VQA-RAD
PMC-VQA
Slake
PathVQA
VQA-RAD
Single agent (Specialized model)
BiomedGPT
53.5 ± 0.3
57.0 ± 0.7
52.0 ± 0.2
41.5 ± 0.7
54.7 ± 0.8
39.5 ± 2.2
54.8 ± 3.4
Med-Flamingo
61.5 ± 1.5
58.5 ± 3.0
54.5 ± 1.9
38.0 ± 2.0
57.9 ± 1.1
34.7 ± 2.8
51.5 ± 4.2
LLaVA-Med
59.0 ± 0.1
52.5 ± 0.1
60.5 ± 0.4
36.0 ± 1.3
59.6 ± 0.4
48.5 ± 0.5
65.8 ± 1.7
HuatuoGPT-Vision
69.5 ± 1.8
59.0 ± 1.4
64.0 ± 1.6
41.0 ± 2.2
59.6 ± 1.8
45.9 ± 1.6
63.8 ± 0.6
Single agent (Local models)
LLaVA-v1.6-7B
65.0 ± 0.5
52.0 ± 1.2
54.0 ± 0.7
32.0 ± 2.1
52.4 ± 2.8
31.3 ± 0.7
50.8 ± 2.9
Table 1: Accuracy comparison with state-of-the-art baseline models on Slake, PathVQA, VQA-RAD and PMC-VQA datasets under multi-choice and free-form questions.
Score
MDAgent
MedAgent
SMOA
MOA
ColaCare
SMART-Med
SMART-Med *
LC
2.89
3.31
3.48
3.42
3.28
3.24
3.55
FC
2.80
3.11
3.37
3.12
3.20
2.96
3.38
IA
2.56
3.19
3.29
3.40
3.18
3.14
3.57
Avg.
2.75
3.20
3.38
3.31
3.22
3.11
3.50
Table 2: Reasoning quality comparison. LC, FC, and IA respectively represent the Logic Consistency, Factual Correctness and Image Alignment.
Figure 4: Efficiency comparison of SMART-Med and other MAS methods. We compare the average runtime (second), average I/O token usage and the average model calls (times) per question.
Model
MC
FF
Slake
PathVQA
VQA-RAD
PMC-VQA
Slake
PathVQA
VQA-RAD
Single agent
76.1 ± 1.4
64.6 ± 2.1
70.8 ± 1.8
35.3 ± 2.8
75.4 ± 0.8
44.3 ± 2.5
62.6 ± 4.2
Multi-agent
+ Dense collaboration
78.5 ± 0.3
66.7 ± 2.4
71.3 ± 2.2
50.1 ± 0.5
75.7 ± 2.8
51.2 ± 1.7
64.5 ± 1.7
+ Uncertainty routing
76.7 ± 1.0
68.6 ± 1.3
72.5 ± 0.5
48.5 ± 2.1
78.2 ± 1.3
48.7 ± 0.2
64.1 ± 3.2
Table 3: Ablation study results on MC and FF questions.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Slake
PathVQA
VQA-RAD
PMC-VQA
MedVR-3K
Size
4,717
26,516
1,778
4,000
3,477
Answer
✓
✓
✓
✓
✓
Reasoning
✗
✗
✗
✗
✓
Difficulty
✗
✗
✗
✗
✓
Modality
Radiology images
Pathology slides
Radiology images
Radiology images
Pathology + Radiology
Question Type
YES/NO/OPEN
YES/NO/OPEN
YES/NO/OPEN
A/B/C/D
YES/NO/A/B/C/D
Appendix
Table 4: Dataset statistics. We compare the modality, size, question type of 4 datasets as well as our constructed MedVR-3K that are used in our work.
Figure 5: The medical images used in the Listing 1 to Listing 6 . Notably, the image (b) is used for two queries, a multi-choice question (L2) and a free-form question (L6).
Large language models (LLMs) have revolutionized medical reasoning tasks, yet single-agent systems often falter on complex, interdisciplinary problems requiring robust handling of uncertainty and conflicting evidence. Multi-agent systems (MAS) leveraging LLMs enable collaborative intelligence, but prevailing centralized architectures suffer from scalability bottlenecks, single points of failure, and role confusion in resource-constrained environments. Decentralized MAS (D-MAS) promise enhanced autonomy and resilience via peer-to-peer interactions, but their application to high-stakes healthcare domains remains underexplored. We introduce MediHive, a novel decentralized multi-agent framework for medical question answering that integrates a shared memory pool with iterative fusion mechanisms. MediHive deploys LLM-based agents that autonomously self-assign specialized roles, conduct initial analyses, detect divergences through conditional evidence-based debates, and locally fuse peer insights over multiple rounds to achieve consensus. Empirically, MediHive outperforms single-LLM and centralized baselines on MedQA and PubMedQA datasets, attaining accuracies of 84.3% and 78.4%, respectively. Our work advances scalable, fault-tolerant D-MAS for medical AI, addressing key limitations of centralized designs while demonstrating superior performance in reasoning-intensive tasks.
Xiaoyang Wang, Christopher C. Yang
College of Computing and Informatics, Drexel University, Philadelphia, PA 19104, USA
While medical multimodal large language models (Med-MLLMs) advance medical visual question answering (VQA), existing clinical workflow-inspired multi-agent frameworks suffer from interaction patterns and excessive computational overhead caused by redundant communication topologies. In this paper, we propose MedPrune, an efficient medical multimodal multi-agent collaboration framework that dynamically prunes both nodes and edges from the communication topology to enhance reasoning ability and token efficiency. Specifically, we first formulate the diagnostic process as a heterogeneous communication graph, where nodes represent specialist agents from various departments and edges capture intra- and inter-departmental interactions. Building on this graph, we introduce two sparsification mechanisms to enable adaptive collaborative evolution: (1) Heterogeneous Node Sparsification, which eliminates task-irrelevant specialist agents irrelevant to the current multimodal question via reinforcement learning-driven topological optimization, and (2) Heterogeneous Edge Sparsification, which selectively retains only the most diagnostically salient intra- and inter-departmental connections by jointly optimizing task performance and topological complexity. Extensive medical VQA experiments under full-set and few-shot training settings prove MedPrune surpasses multi-agent baselines and boosts token efficiency with strong adversarial robustness.
Jiuheng Wan, Runze Li, Chen Chen +5
School of Computer Science and Information Engineering, Hefei University of Technology · Nanjing University · Guangdong University of Finance and Economics +2
Retrieval-Augmented Generation (RAG) is widely employed to mitigate risks such as hallucinations and knowledge obsolescence in medical question answering, yet its predominantly single-round, static retrieval paradigm misaligns with the multi-stage process of clinical reasoning. This compressed workflow induces two structural deficiencies: question-to-query translation often lacks clinically grounded semantic interpretation, and retrieval lacks iterative sufficiency feedback, making it difficult to form reliable evidence chains. We argue that both issues stem from a deeper cause: overloading a single reasoning chain with heterogeneous tasks of interpretation, exploration, and adjudication. The remedy is to reconstruct the workflow via task decoupling and dynamic multi-round exploration. To this end, we propose SEMA-RAG, a Self-Evolving Multi-Agent RAG framework for medical question answering, which assigns these roles to three specialist agents: the Interpreter Agent for clinical schema interpretation, the Explorer Agent for sufficiency-driven self-evolving retrieval, and the Arbiter Agent for evidence adjudication and answer selection. Across five benchmarks and five LLM backbones, SEMA-RAG improves the strongest baseline by +6.46 accuracy points on average, measured per backbone.
Yongfeng Huang, Ruiying Chen, James Cheng
CSE, The Chinese University of Hong Kong · Wuhan University of Technology