MMedAgent-Plus: Scaling Multimodal Medical Agents across Modalities, Tasks and Workflows

Ruoxin Wang; Yuanting Pan; Binxu Li; Ruiyang Ji; Yaqian Chen; Jiayuan Ding; Qifan Wang; Lifu Huang

Figure 1: Overview of MMedAgent-Plus . (a) Medical tool library across tasks, functions, and modalities. (b) Comparison with four vision-language baselines. (c) MMedAgent-Plus work-

Figure 3: Tool orchestration performance across dialogue settings. a, Aggregate performance across single-tool multi-round, multi-tool multi-round, and multi-tool single-round dialogues, evaluated on 500 held-out dialogues per setting (Section 2 ). Whole-case accuracy (Eq. 4 ) requires the complete ordered sequence of tool calls within a dialogue to match the reference sequence, whereas tool-call accuracy (Eq. 3 ) measures correctness at the level of individual required tool invocations. Tool-level precision, recall, and F1 are summarized using micro-averaging, which pools tool decisions across all dialogues, and macro-averaging, which assigns equal weight to each tool type. b-d, Per-tool precision, recall, and F1 for b, single-tool multi-round, c, multi-tool multi-round, and d, multi-tool single-round dialogues. Each row represents a specialist tool, with connected markers indicating its precision, recall, and F1. Tools are ordered by F1 within each panel. Precision captures the extent of unnecessary tool selection, whereas recall captures missed required tool invocations. Full metric definitions are provided in Section 4.5.3 .

Figure 4: Tool- and task-level routing performance across dialogue settings. Row-normalized recall and incorrect routing assignments aggregated across three dialogue settings (1,500 held-out dialogues). a, Per-tool recall, with the dashed line indicating 0.90. b, Nonzero incorrect tool assignments, showing how missed or misrouted tool calls are distributed across alternative routing outcomes. No tool indicates that

\mathcal{M}

responds without invoking a specialist. c, Per-task recall. d, Nonzero incorrect task assignments. For Registration, the proportion assigned to No tool equals

1-

recall, corresponding to the missed UniGradICON invocations in b .

Model	SM (Single-Tool MR)				MM (Multi-Tool MR)				MS (Multi-Tool SR)
	Avg	Faith.	Compl.	Conci.	Avg	Faith.	Compl.	Conci.	Avg	Faith.	Compl.	Conci.
LLaVA-Med [ 13 ]	2.68	3.14	2.15	2.76	1.27	1.03	1.01	1.77	2.45	2.03	2.15	3.18
GPT-5.4-mini [ 57 ]	3.80	3.46	3.74	4.21	4.04	3.72	3.88	4.54	2.53	2.11	2.40	3.08
MedGemma [ 58 ]	3.95	3.76	3.59	4.49	3.87	3.59	3.52	4.50	3.12	2.98	2.75	3.63
Qwen3-VL [ 59 ]	4.42	4.16	4.43	4.68	4.29	3.86	4.34	4.65	3.63	3.30	3.49	4.10
MMedAgent-Plus (ours)	4.84	4.85	4.90	4.78	4.79	4.68	4.87	4.81	4.92	5.00	5.00	4.77

Table 1: Final response quality across dialogue settings. Responses are evaluated for Faithfulness (Faith.) / Completeness (Compl.) / Conciseness (Conci.) on a 1-5 scale. Avg denotes the mean of the three scores. Dialogue settings includes single-tool multi-round (SM), multi-tool multi-round (MM), and multi-tool single-round (MS).

Figure 5: End-to-end performance of MMedAgent-Plus across seven representative tasks. Each task is evaluated on 50 held-out cases using one specialist tool (Table 4 ), giving 350 cases in total. a, LLM-as-a-judge scores for five text-producing tasks, showing the mean of faithfulness, completeness, and conciseness (1-5 scale) for each task. The legend reports the overall mean across the five tasks. b, Specialist invocation accuracy, reported as the fraction of cases in which

\mathcal{M}

selects the required specialist, together with 95% Wilson confidence intervals and the underlying counts. c, Task-level scores from a decomposed into faithfulness, completeness, and conciseness. Points indicate means, and error bars represent 95% bootstrap confidence intervals. Shaded bands highlight the highest mean within each task. Direct-answer baselines are evaluated against the original multimodal inputs because they do not receive specialist outputs. d, Task-specific output quality compared with standalone specialists and direct-answer baselines. The Specialist output row reports metrics computed directly from specialist outputs, whereas model rows report metrics computed from final responses. Abbreviations: R-L, ROUGE-L; Acc., accuracy; RG, RadGraph F1; Ent., entity-level F1; B-4, BLEU-4; CX, CheXpert F1;

\mu

F1, micro-F1; mF1, macro-F1; HD95, 95th-percentile Hausdorff distance. For segmentation, Dice is averaged over all 50 cases (missing masks score 0), whereas HD95 is averaged only over cases with generated masks (lower is better, 0 indicates a perfect match). Segmentation metrics are not applicable to text-only responses and are indicated by dashes.

System	Modality / clinical scope	Supported tasks
WSI-Agents [ 34 ]	Pathology (WSI)	Multi-task WSI analysis
PathAgent [ 35 ]	Pathology (WSI)	WSI analysis
RadAgents [ 38 ]	Chest X-ray	Reporting, interpretation
EviAgent [ 39 ]	Chest X-ray	Report generation
MARCH [ 40 ]	Chest CT	Report generation
MRGAgents [ 41 ]	Chest X-ray	Report generation

Table 2: Task and modality coverage of medical tool-orchestrating agents. Comparison of the reported modality or clinical scope and supported tasks of representative systems. MMedAgent is the base system extended by MMedAgent-Plus .

Category	Tool	#Samples
Single-tool Multi-round	PMC-LLaMA [ 47 ]	9,000
	RaTE-NER [ 48 ]	4,500
	ChatCAD-R [ 52 ]	2,528
	LLaVA-Med [ 13 ]	16,687
	BiomedCLIP [ 12 ]	7,923
	Grounding DINO [ 62 ]	4,207

Table 3: Composition of the instruction-tuning corpus across three dialogue settings. Single-tool multi-round samples are grouped by specialist tool, whereas multi-tool settings are reported by their total sample counts. Tool-chain configurations are detailed in Supplementary Table S5 - S6 .

Task	Tool	Modality	Metric(s)
QA / Rationale Generation	PMC-LLaMA [ 47 ]	Text	ROUGE-L, Accuracy
Named Entity Recognition	RaTE-NER [ 48 ]	Text	RadGraph F1, Entity-level F1
Classification (Pathology)	CONCH [ 2 ]	Pathology	Micro F1, Macro F1
Classification	BiomedCLIP [ 12 ]	X-Ray, MRI, CT, US	Micro F1, Macro F1
Segmentation (Fundus)	IterNet [ 54 ]	Retina	Dice, HD95
Segmentation (General)	MedSAM [ 3 ]	X-Ray, MRI, CT, US	Dice, HD95

Table 4: Task-specific evaluation metrics used in end-to-end assessment. Tasks span both text and image modalities; metrics are selected to reflect clinically relevant aspects of output quality. QA, question answering; NER, named entity recognition; MRI, magnetic resonance imaging; CT, computed tomography; US, ultrasound; HD95, 95th-percentile Hausdorff distance.

Tool	Task	Checkpoint	License
Text understanding and generation
PMC-LLaMA [ 47 ]	QA & rationale generation	axiong/PMC_LLaMA_13B	OpenRAIL
RaTE-NER [ 48 ]	Named entity recognition	Angelakeke/RaTE-NER-Deberta ; token classification with span aggregation	MIT
ChatCAD-R † [ 52 ]	Retrieval-augmented generation	Released ChatCAD checkpoint: https://drive.google.com/drive/folders/1Wcfspge2K0db-Fz2K9vjjFGMKX32WCvc	Apache-2.0
Visual understanding
LLaVA-Med † [ 13 ]	Visual question answering	microsoft/llava-med-v1.5-mistral-7b . The original LLaVA-Med v1.0.0 checkpoint is no longer accessible from the original release repository.	Microsoft Research License

Table S1 : Tool cards for the 18 specialist tools in

\mathcal{T}

. For each tool, we report its supported task, deployment checkpoint, and declared license, as documented in the original code repository or model card. ‘Not specified’ indicates that no explicit license was identified in the corresponding release and does not imply permission for unrestricted use. Tools marked with a dagger (

\dagger

) are inherited from MMedAgent [ 28 ] and retain their original configurations.

Tool	Text	X-ray	CT	MRI	US	Fundus	OCT	Pathology
PMC-LLaMA [ 47 ]	✓
RaTE-NER [ 48 ]	✓
ChatCAD-R [ 52 ]	✓
LLaVA-Med [ 13 ]		✓	✓	✓				✓
BiomedCLIP [ 12 ]		✓	✓	✓				✓
CONCH [ 2 ]								✓

Table S2 : Tool-modality compatibility matrix. Checkmarks indicate supported tool-modality combinations included in the instruction-tuning corpus. Text-based tools operate on textual inputs, while image-based tools may also accept textual prompts (Section S1.3 ).

Task	Tool	Modality	Data Source	#Samples
QA / Rationale Generation	PMC-LLaMA	Text	PMC-LLaMA , PubMedQA , MedQA-USMLE	9,000
Named Entity Recognition	RaTE-NER	Text	RaTE-NER , SumPubMed	4,500
Retrieval-Augmented Generation	ChatCAD-R	Text	Merck Manual	2,528
Visual Question Answering	LLaVA-Med	MRI, CT, X-ray, Histology, Gross	PMC article 60K-IM	16,687
Classification	CONCH	Pathology	CRC100k	3,600
Classification	BiomedCLIP	MRI, CT, X-ray, Histology, Gross	PMC article 60K-IM	7,923

Table S3 : Source datasets for instruction-tuning data construction. The table lists the source datasets, associated specialist tools, input modalities, and number of instruction samples used to construct

\mathcal{S}_{1}

. Dataset links are provided in footnotes.

Task	Tool	Request	Response
QA & rationale generation	PMC-LLaMA [ 47 ]	48	40
Named entity recognition	RaTE-NER [ 48 ]	40	40
Report generation (radiology)	LLaVA-Rad [ 14 ]	81	40
Report generation (retina OCT)	SpecialistVLMs [ 49 ]	90	41
Segmentation (ultrasound)	UltraSAM [ 56 ]	49	41
Segmentation (fundus)	IterNet [ 54 ]	65	—

Table S4 : Size of the request and response template inventories. For each task we report the number of distinct user-request phrasings and assistant-response templates from which samples are drawn. Inventories are frozen before corpus construction and are released with the generation scripts.

Tool chain	Workflow
BiomedCLIP $\rightarrow$ PMC-LLaMA $\rightarrow$ LLaVA-Rad	Identify the study, interpret it, then report
LLaVA-Rad $\rightarrow$ PMC-LLaMA	Generate a report, then answer a question about it
MedSAM $\rightarrow$ LLaVA-Rad	Segment a structure, then report on it
UltraSAM $\rightarrow$ LLaVA-Rad	Segment an ultrasound study, then report on it
IterNet $\rightarrow$ LLaVA-Rad	Extract retinal vessels, then report on them

Table S5 : Tool chains in the multi-tool single-round subset

\mathcal{S}_{3}

. Each sequence is selected and ordered from a single user request before any specialist tool is executed.

Tool chain	Workflow
Successive requests on the same study (two invocations; 45 distinct chains)
BiomedCLIP $\rightarrow$ PMC-LLaMA	Classify the study, then answer a question about it
BiomedCLIP $\rightarrow$ RaTE-NER	Classify the study, then extract typed entities
BiomedCLIP $\rightarrow$ ChatCAD-R	Classify the study, then retrieve supporting knowledge
MedSAM $\rightarrow$ PMC-LLaMA	Segment a structure, then answer a question about it
LLaVA-Med $\rightarrow$ PMC-LLaMA	Answer a visual question, then reason over the answer

Table S6 : Representative tool chains in the multi-tool multi-round subset

\mathcal{S}_{2}

. The first block shows dependent workflows in which a downstream text-based tool processes the output of an upstream specialist. The remaining blocks show independent tool sequences in which each turn introduces a separate request. These sequences illustrate the diversity of tool combinations rather than dependencies between successive invocations.

Figure S1 : PMC-LLaMA (QA / rationale generation).

Figure S2 : RaTE-NER (named entity recognition).

Figure S3 : LLaVA-Rad (radiology report generation).

Figure S4 : CONCH (pathology classification).

Figure S5 : BiomedCLIP (multimodal classification).

Figure S6 : IterNet (fundus vessel segmentation).

Figure S7 : MedSAM (general medical image segmentation).

Figure S8 : UniGradICON (image registration; multiple input images).

Setting	Value
Model
Initialization	Qwen2.5-VL-7B-Instruct
Vision encoder	Frozen
Vision-language projector	Trained
Language decoder	Trained (full parameters)
Data

Table S9 : Training configuration of MMedAgent-Plus .

Figure S9 : LLM-as-a-judge scoring rubric. Each criterion is scored independently. Colour encodes quality from 1 (red) to 5 (blue).

Figure S10 : Biomedical question answering. MMedAgent-Plus invokes PMC-LLaMA and correctly concludes that optic nerve head torsion is more prominent in normal-tension glaucoma, whereas MedGemma-1.5-4B-IT and LLaVA-Med contradict the cited evidence.

Figure S11 : Biomedical question answering. For a question on nitric-oxide treatment of stored erythrocytes, MMedAgent-Plus returns a concise PMC-LLaMA-grounded answer, while the direct-answer baselines produce verbose responses that fail our reference matching despite reaching the correct yes/no conclusion.

Figure S12 : Biomedical question answering. MMedAgent-Plus correctly reports that GSTM1 and GSTT1 polymorphisms are not associated with gestational diabetes mellitus; MedGemma-1.5-4B-IT and LLaVA-Med omit the explicit negative association required by the reference answer.

Figure S13 : Radiology report generation. Despite the user mislabeling a chest radiograph as an MRI, MMedAgent-Plus invokes LLaVA-Rad and produces an accurate structured report; direct-answer baselines misidentify the modality or decline to interpret the image.

Figure S14 : Radiology report generation. On a lateral chest radiograph with low lung volumes, MMedAgent-Plus synthesizes a faithful LLaVA-Rad report, whereas Qwen3-VL-8B hallucinates a pulmonary mass and LLaVA-Med incorrectly calls the study normal.

Figure S15 : Radiology report generation. For a pacemaker chest radiograph with chronic interstitial markings, MMedAgent-Plus preserves LLaVA-Rad findings on device leads and lung parenchyma; baselines misread the study as hiatal hernia, pneumothorax, or pleural effusion.

Figure S16 : Named entity recognition. MMedAgent-Plus reproduces RaTE-NER typed spans for coronary-artery anatomy and abnormality phrases; direct-answer baselines return unstructured term lists without the required entity-type labels.

Figure S17 : Named entity recognition. MMedAgent-Plus identifies grade I L5 - S1 spondylolisthesis as a single Disease entity per RaTE-NER, while baselines split or paraphrase the span without matching the specialist schema.

Figure S18 : Named entity recognition. On the negated finding no intracranial hemorrhage , MMedAgent-Plus labels intracranial hemorrhage as Non-Disease following RaTE-NER; baselines either skip entity extraction or produce free-text interpretations.

Figure S19 : Image classification. MMedAgent-Plus invokes BiomedCLIP and correctly identifies immunohistochemistry staining in a histopathology panel; direct-answer baselines default to H&E or mixed-stain descriptions.

Figure S20 : Image classification. MMedAgent-Plus classifies the scan as lung CT via BiomedCLIP; baselines give generic “CT scan” answers that omit the organ-specific label required by the reference.

Figure S21 : Image classification. MMedAgent-Plus recognizes a COVID line chart through BiomedCLIP, whereas all direct-answer baselines misclassify the figure as a chest radiograph montage.

Figure S22 : Pathology classification. MMedAgent-Plus reports CONCH stromal-tissue (STR) predictions with clinical context; baselines provide generic histology descriptions without the tissue-class label.

Figure S23 : Pathology classification. MMedAgent-Plus identifies tumor tissue (TUM) via CONCH; direct-answer baselines describe tissue architecture but fail to assign the pathology class.

Figure S24 : Pathology classification. MMedAgent-Plus correctly reports CONCH background-tissue (BACK) with no pathological findings; baselines give non-committal or incorrect interpretations of the patch.

MMedAgent-Plus: Scaling Multimodal Medical Agents across Modalities, Tasks and Workflows

Abstract

Figures & tables

Medical AI Agents

MedCTA: A Benchmark for Clinical Tool Agents

Meissa: Multi-modal Medical Agentic Intelligence

Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning

Abstract

Medical AI Agents

Explore similar work

MedCTA: A Benchmark for Clinical Tool Agents

Meissa: Multi-modal Medical Agentic Intelligence

Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning