Role-Specialized Mixture-of-Agents with Open-Weight LLMs for Clinical Prediction

Jun Hou; Yi Fang; Xuan Wang

cs.AIAug 23, 2026

Role-Specialized Mixture-of-Agents with Open-Weight LLMs for Clinical Prediction

Authors: Jun Hou, Yi Fang, Xuan Wang

Organizations: Virginia Tech

Abstract

Large Language Models (LLMs) are increasingly applied to clinical prediction tasks such as in-hospital mortality and readmission from electronic health records (EHRs). Privacy and compliance constraints motivate systems that can be deployed locally, which has increased interest in open-weight multi-agent designs. However, most medical multi-agent systems are evaluated as a single block, leaving unclear which agent role contributes to prediction and whether retrieval drives observed gains. We study a role-specialized Mixture-of-Agents (MoA) that combines medical knowledge retrieval with contrastive similar-patient reasoning. By varying the role design while holding the retrieval setup fixed, we localize the main effect to the final integrator. Pairing large open-weight analysts with a small open-weight integrator matches closed-model prompting on F1 for mortality prediction while flagging substantially more true high-risk patients. Mechanism analysis shows the role assignment directly yields a high-recall operating point without threshold tuning. The effect is task-dependent, with smaller gains for readmission because the available records correlate weakly with this longer-horizon outcome. These results position role design as a key factor in privacy-constrained, training-free clinical LLM prediction.

Figures & tables

Figure 1: Pipeline overview. (A) Multi-source databases. A knowledge database is built from MedCorp and UMLS, and a patient database is built from the EHRs. (B) Role-specialized retrieval and integration. The label-blind risk analyst and protective analyst read the target record and augment their reasoning through retrieval from the knowledge database and the patient database; the integrator runs its own retrieval and integrates the two readings into a final prediction.

	MIMIC-III Mort.				MIMIC-IV Mort.
A/R/I	MacF1	Sens	Spec	AUROC	MacF1	Sens	Spec	AUROC
oss/oss/oss	40.0	25.9	58.7	.406	38.5	13.0	80.4	.543
oss/qwen/oss	38.3	18.5	67.4	.399	35.0	7.4	84.8	.533
qwen/oss/oss	36.2	24.1	52.2	.340	36.3	11.1	78.3	.517
oss/oss/qwen	47.3	75.9	23.9	.470	57.4	64.8	50.0	.603
oss/qwen/qwen	41.3	24.1	65.2	.444	40.6	18.5	73.9	.575

Table 1: Role-isolation factorial, both mortality cells. A/R/I

=

analyst/retrieval/integrator; oss

=

GPT-OSS-120B, qwen

=

Qwen2.5-7B; oss CoT

=

chain-of-thought. F1/Sens/Spec in %; AUROC computed within each cell. Bold (large analysts, small integrator) best on both.

Pre-trained (task-supervised, trained from scratch)
	MIMIC-IV Mort.			MIMIC-III Mort.			MIMIC-IV Read.			MIMIC-III Read.
	(pos 19.2%)			(pos 5.4%)			(pos 46.5%)			(pos 54.8%)
Method	Acc	F1	Sens	Acc	F1	Sens	Acc	F1	Sens	Acc	F1	Sens
All-negative	80.8	44.7	0.0	94.6	48.6	0.0	53.5	34.9	0.0	45.2	31.1	0.0
GRU	88.7	76.4	42.9	92.7	50.7	3.7	62.4	62.2	68.3	62.2	61.5	68.9
AdaCare	88.7	78.2	50.3	90.6	54.1	9.1	62.9	62.9	58.4	61.6	60.5	70.8

Table 2: Main results, all four cells (Acc/F1/Sens, %), by training regime. Our training-free rows use RAG

+

KARE exemplars (single vs. multi

=

topology). Bold marks the best training-free row. In the open rows oss

=

GPT-OSS-120B and qwen

=

Qwen2.5-7B; multi rows are named by their (analyst/retrieval/integrator) assignment.

				MIMIC-IV Mort.				MIMIC-III Mort.				MIMIC-IV Read.				MIMIC-III Read.
Model	Topo	RAG	Sim	Acc	F1	Sens	Spec	Acc	F1	Sens	Spec	Acc	F1	Sens	Spec	Acc	F1	Sens	Spec
GPT-oss	single	–	–	77.9	58.3	24.2	90.7	70.6	48.4	46.3	72.0	49.0	45.2	24.6	70.1	43.3	35.0	7.0	87.3
	single	✓	✓	71.6	52.8	22.1	83.4	61.2	42.0	33.3	62.8	47.7	47.7	48.8	46.7	43.8	43.1	49.6	36.7
	single	✓	–	48.5	37.7	17.9	55.8	59.7	42.0	40.7	60.8	49.1	49.0	56.5	42.6	51.9	46.6	76.2	22.4
	multi	–	✓	78.3	51.3	10.0	94.6	80.0	50.5	25.9	83.1	50.1	48.9	37.4	61.3	47.5	46.9	33.9	64.0
	multi	✓	✓	78.7	54.4	14.7	94.0	74.4	46.9	22.2	77.4	47.5	47.2	43.5	50.9	48.8	48.8	42.1	56.9

Table 3: Component ablation, all four cells (%). Topo single vs. multi; RAG MedRAG; Sim KARE exemplars; the single

+

RAG no-exemplar row is the added control. The rows reported in the main results table are the single

+

RAG

+

Sim and multi

+

RAG

+

Sim configurations.

Figure 2: Tjur’s $D$ (left two panels) and AUROC (right two) on MIMIC-IV mortality, for CoT , single $+$ RAG $+$ Sim , and the multi $+$ RAG $+$ Sim pipeline. Each curve is the $95\%$ bootstrap CI for that condition and the dot is its point estimate, with the dashed line marking the null ( $D{=}0$ or AUROC ${=}0.5$ ).

Figure 3: Mean predicted risk versus number of recorded conditions, among patients who did not have the event. Bars are standard error.

Cell	Prev. %	Feature AUROC	CoT-LLM AUROC	Similars $\Delta$ AUROC
mortality m4	19.2	0.77	0.763	$+0.225$
mortality m3	5.4	0.52	0.635	$+0.009$
readmission m4	46.5	0.59	0.431	$-0.029$
readmission m3	54.8	0.48	0.422	$-0.090$

Table 4: Signal-location probes, mortality vs. readmission. Feature AUROC , cross-validated bag-of-codes classifier. CoT-LLM AUROC , the unaugmented gpt-oss agent (qwen

.625/.666/.498/.542

). Similars

\Delta

AUROC , gpt-oss.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	m3		m4		rd3		rd4
Method	ROC	PRC	ROC	PRC	ROC	PRC	ROC	PRC
oss CoT	63.5	8.3	76.3	35.7	42.2	49.6	43.1	42.5
oss single $+$ RAG $+$ Sim	48.4	5.1	63.8	24.9	40.6	49.2	45.7	43.9
oss multi $+$ Sim	57.8	7.3	74.1	32.7	48.5	54.1	49.7	46.8
oss multi $+$ RAG $+$ Sim	53.3	5.7	73.0	32.1	48.5	53.6	45.4	44.2
oss multi $+$ RAG $+$ Sim (qwen)	50.1	5.6	70.6	31.4	48.7	54.5	52.4	47.6

Appendix

Table 5: AUROC / AUPRC (%) for methods that expose probabilities (m3/m4

=

MIMIC-III/IV mortality, rd3/rd4

=

readmission).

Figure 4: Discrimination slope (Tjur’s $D$ , top) and AUROC (bottom), CoT $\to$ single $+$ RAG $+$ Sim $\to$ multi $+$ RAG $+$ Sim, both models, all four cells; ridges $=$ $95\%$ bootstrap CI (normal approx.), filled $=$ significant.

	gpt-oss integrator		Qwen integrator
cell ( $n_{\text{oss}}$ / $n_{\text{qwen}}$ )	an.	int.	an.	int.
TP ( $28$ / $116$ )	54.5	13.6	32.7	4.3
FP ( $48$ / $258$ )	46.0	10.6	24.1	3.5
FN ( $162$ / $74$ )	22.0	5.0	17.6	2.2
TN ( $749$ / $539$ )	11.6	3.3	8.7	1.7

Appendix

Table 6: Severity terms per case by confusion-matrix cell, MIMIC-IV mortality (shared

n{=}987

). an.

=

shared gpt-oss analyst text; int.

=

integrator text.

	MIMIC-III Mort.				MIMIC-IV Mort.				MIMIC-III Read.				MIMIC-IV Read.
A/R/I	F1	Sn	Sp	AUC	F1	Sn	Sp	AUC	F1	Sn	Sp	AUC	F1	Sn	Sp	AUC
oss/oss/oss	40.0	25.9	58.7	.41	38.5	13.0	80.4	.54	46.6	34.5	62.2	.49	46.6	41.3	51.9	.45
oss/qwen/oss	38.3	18.5	67.4	.40	35.0	7.4	84.8	.53	43.2	12.7	97.8	.61	37.4	2.2	100.0	.47
qwen/oss/oss	36.2	24.1	52.2	.34	36.3	11.1	78.3	.52	45.7	20.0	86.7	.56	34.6	0.0	98.1	.49
oss/oss/qwen	47.3	75.9	23.9	.47	57.4	64.8	50.0	.60	34.2	94.5	0.0	.53	34.5	89.1	5.6	.47
oss/qwen/qwen	41.3	24.1	65.2	.44	40.6	18.5	73.9	.58	53.1	25.5	95.6	.60	40.5	8.7	88.9	.45

Appendix

Table 7: Role-isolation factorial, all four cells,

n{=}100

each (F1/Sens/Spec %; AUROC within-cell). A/R/I

=

analyst/retrieval/integrator; oss

=

large, qwen

=

small. Bold (large analysts, small integrator) wins F1/Sens/AUROC on mortality.

Integrator	Analysts	Retrieval	Acc	Recall	F1
GPT-4o	GPT-4o	GPT-4o	45.0	3.7	6.8
Qwen	GPT-4o	GPT-4o	53.0	57.4	56.9

Appendix

Table 8: Family-invariance check (

n{=}100

MIMIC-III mortality; Acc/Recall/pos-class F1 %). Swapping the integrator GPT-4o

\to

Qwen reproduces the gpt-oss-family recall lift from replacing only the integrator with the small model.

Mixture-of-Agents

See all 9 papers

AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Tasks
May 11, 2026Baraa Al Jorf, Farah E. ShamoutMulti-Agent LLM Systems LLM Agent Evaluation
New York University Abu Dhabi, UAE
One Panel Does Not Fit All: Case-Adaptive Multi-Agent Deliberation for Clinical Prediction
Mar 31, 2026Yuxing Lu, Yushuhong Lin, Jason ZhangMulti-Agent Collaboration Clinical Prediction
Georgia Institute of Technology · Peking University
MARC v1: An Open-Source Multi-Agent Framework for Clinical AI Reasoning and Coordination
Aug 13, 2026Saisha Shetty, Satvik Tripathi, Austin Lin +6Multi-Agent LLM Systems LLM Agent Orchestration
1. College of Engineering, University of California, Davis · 2. Perelman School of Medicine, University of Pennsylvania · 3. School of Engineering and Applied Science, University of Pennsylvania +2

May 11, 2026J/K move · Enter open · S save

AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Tasks

Baraa Al Jorf, Farah E. Shamout

New York University Abu Dhabi, UAE

Multi-Agent LLM Systems LLM Agent Evaluation AI Agent Benchmarks Multimodal Agents Clinical Risk Prediction

Open paper AskPDF arXiv

Building effective clinical decision support systems requires the synthesis of complex heterogeneous multimodal data. Such modalities include temporal electronic health records data, medical images, radiology reports, and clinical notes. Large language model (LLM)-based agents have shown impressive performance in various healthcare tasks, especially those involving textual modalities. Considering the fragmentation of healthcare data across hospital systems, collaborative agent frameworks present a promising direction to mitigate data sharing challenges. However, the effectiveness of LLM agents for multimodal clinical risk prediction remains largely unexamined. In this work, we conduct a systematic evaluation of LLM-based agents for clinical prediction tasks using large-scale real-world data. We assess performance in unimodal and multimodal settings and quantify performance gaps between single agent and multi-agent systems. Our findings highlight that single agent frameworks outperform naive multi-agent systems, are better at handling multimodal data, and are better calibrated. This underscores a critical need for improving multi-agent collaboration to better handle heterogeneous inputs. By open-sourcing our code and evaluation framework, this work offers a new benchmark to support future developments relating to agentic systems in healthcare.

Abstract

Mixture-of-Agents

Explore similar work

AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Tasks

One Panel Does Not Fit All: Case-Adaptive Multi-Agent Deliberation for Clinical Prediction

MARC v1: An Open-Source Multi-Agent Framework for Clinical AI Reasoning and Coordination