MedRouter: Demystifying Knowledge Differences Across Medical LLMs for Routing-Based Reasoning

Lang Cao; Binghang Lu; Yuhao Shen; Yue Guo

cs.CLSep 27, 2026

MedRouter: Demystifying Knowledge Differences Across Medical LLMs for Routing-Based Reasoning

Authors: Lang Cao, Binghang Lu, Yuhao Shen, Yue Guo

Organizations: University of Illinois Urbana-Champaign · Purdue University · The Chinese University of Hong Kong, Shenzhen

Abstract

Medical question answering spans diverse specialties and modalities, and individual medical large language models (LLMs) exhibit distinct strengths across tasks and domains. This heterogeneity suggests that combining specialists may enable broader coverage of medical questions than relying on any single model. However, existing LLM routing methods primarily seek to balance answer quality and inference cost, leaving open how to exploit differences in specialist competence to improve medical reasoning. In this paper, we introduce MedRouter, an agentic system that uses an embedding-based multi-label router to select and query specialist LLMs, then passes their responses to a generator to produce the final answer. We further propose SCALE (Specialist Competence-Aware Learning), a two-stage training framework that first trains the Router with specialist correctness supervision and then optimizes its selections through reinforcement learning. The second stage uses a Performance Gain Reward (PGR) that measures how specialist information affects the generator's answer correctness relative to answering without that information. Experiments on eight text-based and multimodal medical QA benchmarks show that MedRouter outperforms the strongest routing baseline by 8% in average accuracy. Our analysis of specialist outputs further reveals distinct strengths and complementary question-level coverage, motivating learned routing to combine these capabilities for more comprehensive medical reasoning.

Figures & tables

Figure 1: Question-level complementarity among medical specialist LLMs. (a) Specialist accuracy across datasets. (b) Distribution of questions by the number of applicable specialists that answer correctly. (c) Pairwise Jaccard similarity between specialists’ correct-answer sets on text-based QA. (d) Exclusive solve share: questions answered correctly by one specialist but by no other applicable specialist. (e) Pairwise rescue rates on text-based QA: the fraction of questions answered incorrectly by specialist $i$ that specialist $j$ answers correctly. (f) Cumulative coverage as specialists are greedily added, where a question is covered if at least one selected specialist answers correctly. (g) Per-dataset routing headroom, defined as per-question oracle accuracy minus the best fixed-specialist accuracy. (h) Macro-averaged accuracy of a single global specialist, the best specialist per dataset, and a per-question oracle. The oracle counts a question as correct whenever at least one applicable specialist answers correctly.

Figure 2: Overview of the MedRouter framework and SCALE training. Given a medical question, an embedding-based Router predicts call scores and selects specialists from a frozen LLM pool. The frozen LLM generator receives the selected specialists’ information and produces the final answer. During training, only the Router is updated: it is first trained with specialist correctness supervision using a binary cross-entropy loss, and then optimized with reinforcement learning using a Performance Gain Reward that compares answer correctness with and without specialist information.

ML Routing Methods
	Text-Based Medical QA				Multimodal Medical QA
	MedQA	MedMCQA	MedCaseRea.	MedX.-Text	SLAKE	PathVQA	VQA-RAD	MedX.-MM	Average
Best-in-Pool	65.2	59.3	15.1	15.4	67.0	41.7	57.9	27.4	43.6
Ensemble	61.6	59.0	15.7	14.5	66.2	39.5	50.8	27.0	41.8
KNN Router	61.0	56.5	12.3	16.0	70.0	41.3	56.7	26.1	42.5
MLP Router	57.7	52.2	10.7	13.7	70.7	40.9	56.1	24.9	40.9

Table 1: Accuracy (%) on text-based and multimodal medical QA benchmarks. Within each LLM generator block, the best result for each dataset and the average is shown in bold . The relative improvement of MedRouter over the second-highest average in the same block is shown in green .

	MedQA	MedMCQA	MedCaseRea.	MedX.-Text	SLAKE	PathVQA	VQA-RAD	MedX.-MM	Average
	Text-Based Medical QA				Multimodal Medical QA
MedRouter w/ SCALE	71.8	59.9	19.6	17.3	72.0	43.4	55.1	27.9	45.9
w/ Stage I FT only	69.3	57.2	17.6	16.5	69.3	41.1	51.4	26.0	43.6 (-5.0%)
w/ Stage II RL only	70.1	59.1	17.7	16.7	71.2	41.3	54.9	27.2	44.8 (-2.4%)
w/ Stage II RL using correctness reward	70.6	60.0	18.4	16.6	69.7	42.6	52.5	27.0	44.7 (-2.6%)

Table 2: Ablation study of SCALE with Qwen3.5-2B as the generator. The best result in each column is shown in bold . The relative decrease in average accuracy from the full SCALE is shown in red .

Figure 3: Routing outcomes and specialist selection patterns across eight benchmarks. (a) Decomposition of outcomes for Prompt-LLM, Encoder Router, and MedRouter . The dashed line marks the fraction of questions for which at least one specialist knows the answer. (b) Distribution of the number of specialists selected per question. (c) Cumulative fraction of questions assigned to the $k$ most frequent specialist combinations. (d) Number of unique routes on each benchmark and overall. (e) Entropy of the route distribution on each benchmark and overall.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

General-Purpose LLMs
	Text-Based Medical QA					Multimodal Medical QA
	MedQA	MedMCQA	MedCaseRea.	MedX.-Text	Average	SLAKE	PathVQA	VQA-RAD	MedX.-MM	Average
Best-in-Pool	65.2	59.3	15.1	15.4	38.7	67.0	41.7	57.9	27.4	48.5
Qwen3.5-2B	52.0	48.8	7.4	14.6	30.7	59.2	37.3	51.6	26.1	43.6
Gemma-4-E2B	63.0	49.3	10.7	14.7	34.4	50.6	34.7	48.1	24.1	39.4
Medical Specialist LLMs

Appendix

Table 3: Accuracy (%) of individual models on text-based and multimodal medical QA benchmarks. Best-in-Pool selects the highest-scoring medical specialist separately for each dataset. Bold values indicate the best result among individual medical specialists in each column.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 351 papers

Let Large Language Models Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering
Jun 13, 2026Zaifu Zhan, Shuang Zhou, Rui ZhangMulti-Agent LLM Systems LLM Answer Verification
Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 420 Delaware St SE, 55455, Minneapolis, MN, United States
Multilingual Medical Reasoning for Question Answering with Large Language Models
Dec 5, 2025Pietro Ferrazzi, Aitor Soroa, Rodrigo AgerriKnowledge Augmentation for Language Models Healthcare
Fondazione Bruno Kessler, Trento, Italy · HiTZ Center - Ixa, University of the Basque Country EHU, Spain
SEMA-RAG: A Self-Evolving Multi-Agent Retrieval-Augmented Generation Framework for Medical Reasoning
May 16, 2026Yongfeng Huang, Ruiying Chen, James ChengRetrieval-Augmented Generation Healthcare
CSE, The Chinese University of Hong Kong · Wuhan University of Technology

Jun 13, 2026J/K move · Enter open · S save

Let Large Language Models Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering

Zaifu Zhan, Shuang Zhou, Rui Zhang

Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 420 Delaware St SE, 55455, Minneapolis, MN, United States

Multi-Agent LLM Systems LLM Answer Verification Medical QA Multi-Agent Reasoning Clinical Language Model Evaluation

Open paper AskPDF arXiv

Objective: To enhance the accuracy, interpretability, and robustness of large language models (LLMs) in medical question answering (MedQA). Method: We designed a multi-agent peer-reviewed reasoning method in which multiple LLM agents independently generate chain-of-thought reasoning with candidate answers, then act as peer reviewers to evaluate each other's reasoning for factual correctness and logical soundness. The highest-rated reasoning chain is selected to produce the final answer. Experiments were conducted with five state-of-the-art LLMs (Llama-3.1-8B, Qwen2.5-7B, Phi-4, DeepSeek-LLM-7B, GPT-oss-20B) on three benchmark datasets: HeadQA, MedQA-USMLE, and PubMedQA. Performance was compared against single-model chain-of-thought reasoning and chain-of-thought-based majority voting. Results: Peer-reviewed reasoning consistently outperformed both baselines. The best model combination achieved an average accuracy of 0.820 across datasets, exceeding the strongest single model (0.777) and majority voting ensembles (up to 0.789). The method also scaled effectively with more participating models, while peer assessments reliably distinguished high- from low-quality reasoning chains. Conclusion: The proposed multi-agent peer-reviewed reasoning method enables LLMs to act as both solvers and evaluators, yielding superior performance in MedQA. By emphasizing reasoning quality rather than answer agreement alone, this approach improves accuracy, interpretability, and robustness, offering a promising direction for trustworthy biomedical AI systems.

Abstract

LLM Routing

Explore similar work

Let Large Language Models Judge Each Other: Multi-Agent Peer-Reviewed Reasoning for Medical Question Answering

Multilingual Medical Reasoning for Question Answering with Large Language Models

SEMA-RAG: A Self-Evolving Multi-Agent Retrieval-Augmented Generation Framework for Medical Reasoning