Organizations: Duke University · Stanford University · Yale University · Princeton University · Virginia Tech · Hippocratic AI · Meta · UC Davis · Shanghai Jiao Tong University · Tongji University
Medical workflows often require multiple interdependent analyses, but existing specialist and generalist AI models are not designed to coordinate them. We introduce MMedAgent-Plus, a multimodal medical tool-use agent that combines a vision-language planner with 18 leading specialist models spanning 12 tasks and 8 modalities. Through a unified tool interface, the planner selects specialists, executes them in sequence, and synthesizes responses grounded in their outputs. The planner is instruction-tuned on 208,928 samples from 32 public medical benchmarks, covering single- and multi-tool interactions across single and multiple rounds. We separately evaluate tool orchestration, response synthesis, and end-to-end execution on held-out samples. MMedAgent-Plus achieves tool-call accuracies of 91.0%, 98.1%, and 80.7% in single-tool multi-round, multi-tool multi-round, and multi-tool single-round settings, respectively. Given correct tool outputs, MMedAgent-Plus achieves scores of 14.4-14.8 out of 15 for faithfulness, completeness, and conciseness across 600 evaluation samples. In end-to-end execution without pre-supplied tool outputs, MMedAgent-Plus correctly invokes the required specialist in 90.6% of 350 evaluation cases spanning seven tasks. Its final-answer accuracy reaches 0.98, matching the dedicated leading specialist and exceeding the strongest baseline (0.28).
Figures & tables
Figure 1: Overview of MMedAgent-Plus . (a) Medical tool library across tasks, functions, and modalities. (b) Comparison with four vision-language baselines. (c) MMedAgent-Plus work-
Figure 3: Tool orchestration performance across dialogue settings. a, Aggregate performance across single-tool multi-round, multi-tool multi-round, and multi-tool single-round dialogues, evaluated on 500 held-out dialogues per setting (Section 2 ). Whole-case accuracy (Eq. 4 ) requires the complete ordered sequence of tool calls within a dialogue to match the reference sequence, whereas tool-call accuracy (Eq. 3 ) measures correctness at the level of individual required tool invocations. Tool-level precision, recall, and F1 are summarized using micro-averaging, which pools tool decisions across all dialogues, and macro-averaging, which assigns equal weight to each tool type. b-d, Per-tool precision, recall, and F1 for b, single-tool multi-round, c, multi-tool multi-round, and d, multi-tool single-round dialogues. Each row represents a specialist tool, with connected markers indicating its precision, recall, and F1. Tools are ordered by F1 within each panel. Precision captures the extent of unnecessary tool selection, whereas recall captures missed required tool invocations. Full metric definitions are provided in Section 4.5.3 .
Figure 4: Tool- and task-level routing performance across dialogue settings. Row-normalized recall and incorrect routing assignments aggregated across three dialogue settings (1,500 held-out dialogues). a, Per-tool recall, with the dashed line indicating 0.90. b, Nonzero incorrect tool assignments, showing how missed or misrouted tool calls are distributed across alternative routing outcomes. No tool indicates that M responds without invoking a specialist. c, Per-task recall. d, Nonzero incorrect task assignments. For Registration, the proportion assigned to No tool equals 1− recall, corresponding to the missed UniGradICON invocations in b .
Model
SM (Single-Tool MR)
MM (Multi-Tool MR)
MS (Multi-Tool SR)
Avg
Faith.
Compl.
Conci.
Avg
Faith.
Compl.
Conci.
Avg
Faith.
Compl.
Conci.
LLaVA-Med [ 13 ]
2.68
3.14
2.15
2.76
1.27
1.03
1.01
1.77
2.45
2.03
2.15
3.18
GPT-5.4-mini [ 57 ]
3.80
3.46
3.74
4.21
4.04
3.72
3.88
4.54
2.53
2.11
2.40
3.08
MedGemma [ 58 ]
3.95
3.76
3.59
4.49
3.87
3.59
3.52
4.50
3.12
2.98
2.75
3.63
Qwen3-VL [ 59 ]
4.42
4.16
4.43
4.68
4.29
3.86
4.34
4.65
3.63
3.30
3.49
4.10
MMedAgent-Plus (ours)
4.84
4.85
4.90
4.78
4.79
4.68
4.87
4.81
4.92
5.00
5.00
4.77
Table 1: Final response quality across dialogue settings. Responses are evaluated for Faithfulness (Faith.) / Completeness (Compl.) / Conciseness (Conci.) on a 1-5 scale. Avg denotes the mean of the three scores. Dialogue settings includes single-tool multi-round (SM), multi-tool multi-round (MM), and multi-tool single-round (MS).
Figure 5: End-to-end performance of MMedAgent-Plus across seven representative tasks. Each task is evaluated on 50 held-out cases using one specialist tool (Table 4 ), giving 350 cases in total. a, LLM-as-a-judge scores for five text-producing tasks, showing the mean of faithfulness, completeness, and conciseness (1-5 scale) for each task. The legend reports the overall mean across the five tasks. b, Specialist invocation accuracy, reported as the fraction of cases in which M selects the required specialist, together with 95% Wilson confidence intervals and the underlying counts. c, Task-level scores from a decomposed into faithfulness, completeness, and conciseness. Points indicate means, and error bars represent 95% bootstrap confidence intervals. Shaded bands highlight the highest mean within each task. Direct-answer baselines are evaluated against the original multimodal inputs because they do not receive specialist outputs. d, Task-specific output quality compared with standalone specialists and direct-answer baselines. The Specialist output row reports metrics computed directly from specialist outputs, whereas model rows report metrics computed from final responses. Abbreviations: R-L, ROUGE-L; Acc., accuracy; RG, RadGraph F1; Ent., entity-level F1; B-4, BLEU-4; CX, CheXpert F1; μ F1, micro-F1; mF1, macro-F1; HD95, 95th-percentile Hausdorff distance. For segmentation, Dice is averaged over all 50 cases (missing masks score 0), whereas HD95 is averaged only over cases with generated masks (lower is better, 0 indicates a perfect match). Segmentation metrics are not applicable to text-only responses and are indicated by dashes.
System
Modality / clinical scope
Supported tasks
WSI-Agents [ 34 ]
Pathology (WSI)
Multi-task WSI analysis
PathAgent [ 35 ]
Pathology (WSI)
WSI analysis
RadAgents [ 38 ]
Chest X-ray
Reporting, interpretation
EviAgent [ 39 ]
Chest X-ray
Report generation
MARCH [ 40 ]
Chest CT
Report generation
MRGAgents [ 41 ]
Chest X-ray
Report generation
Table 2: Task and modality coverage of medical tool-orchestrating agents. Comparison of the reported modality or clinical scope and supported tasks of representative systems. MMedAgent is the base system extended by MMedAgent-Plus .
Category
Tool
#Samples
Single-tool Multi-round
PMC-LLaMA [ 47 ]
9,000
RaTE-NER [ 48 ]
4,500
ChatCAD-R [ 52 ]
2,528
LLaVA-Med [ 13 ]
16,687
BiomedCLIP [ 12 ]
7,923
Grounding DINO [ 62 ]
4,207
Table 3: Composition of the instruction-tuning corpus across three dialogue settings. Single-tool multi-round samples are grouped by specialist tool, whereas multi-tool settings are reported by their total sample counts. Tool-chain configurations are detailed in Supplementary Table S5 - S6 .
Task
Tool
Modality
Metric(s)
QA / Rationale Generation
PMC-LLaMA [ 47 ]
Text
ROUGE-L, Accuracy
Named Entity Recognition
RaTE-NER [ 48 ]
Text
RadGraph F1, Entity-level F1
Classification (Pathology)
CONCH [ 2 ]
Pathology
Micro F1, Macro F1
Classification
BiomedCLIP [ 12 ]
X-Ray, MRI, CT, US
Micro F1, Macro F1
Segmentation (Fundus)
IterNet [ 54 ]
Retina
Dice, HD95
Segmentation (General)
MedSAM [ 3 ]
X-Ray, MRI, CT, US
Dice, HD95
Table 4: Task-specific evaluation metrics used in end-to-end assessment. Tasks span both text and image modalities; metrics are selected to reflect clinically relevant aspects of output quality. QA, question answering; NER, named entity recognition; MRI, magnetic resonance imaging; CT, computed tomography; US, ultrasound; HD95, 95th-percentile Hausdorff distance.
Tool
Task
Checkpoint
License
Text understanding and generation
PMC-LLaMA [ 47 ]
QA & rationale generation
axiong/PMC_LLaMA_13B
OpenRAIL
RaTE-NER [ 48 ]
Named entity recognition
Angelakeke/RaTE-NER-Deberta ; token classification with span aggregation
MIT
ChatCAD-R † [ 52 ]
Retrieval-augmented generation
Released ChatCAD checkpoint: https://drive.google.com/drive/folders/1Wcfspge2K0db-Fz2K9vjjFGMKX32WCvc
Apache-2.0
Visual understanding
LLaVA-Med † [ 13 ]
Visual question answering
microsoft/llava-med-v1.5-mistral-7b . The original LLaVA-Med v1.0.0 checkpoint is no longer accessible from the original release repository.
Microsoft Research License
Table S1 : Tool cards for the 18 specialist tools in T . For each tool, we report its supported task, deployment checkpoint, and declared license, as documented in the original code repository or model card. ‘Not specified’ indicates that no explicit license was identified in the corresponding release and does not imply permission for unrestricted use. Tools marked with a dagger ( † ) are inherited from MMedAgent [ 28 ] and retain their original configurations.
Tool
Text
X-ray
CT
MRI
US
Fundus
OCT
Pathology
PMC-LLaMA [ 47 ]
✓
RaTE-NER [ 48 ]
✓
ChatCAD-R [ 52 ]
✓
LLaVA-Med [ 13 ]
✓
✓
✓
✓
BiomedCLIP [ 12 ]
✓
✓
✓
✓
CONCH [ 2 ]
✓
Table S2 : Tool-modality compatibility matrix. Checkmarks indicate supported tool-modality combinations included in the instruction-tuning corpus. Text-based tools operate on textual inputs, while image-based tools may also accept textual prompts (Section S1.3 ).
Task
Tool
Modality
Data Source
#Samples
QA / Rationale Generation
PMC-LLaMA
Text
PMC-LLaMA , PubMedQA , MedQA-USMLE
9,000
Named Entity Recognition
RaTE-NER
Text
RaTE-NER , SumPubMed
4,500
Retrieval-Augmented Generation
ChatCAD-R
Text
Merck Manual
2,528
Visual Question Answering
LLaVA-Med
MRI, CT, X-ray, Histology, Gross
PMC article 60K-IM
16,687
Classification
CONCH
Pathology
CRC100k
3,600
Classification
BiomedCLIP
MRI, CT, X-ray, Histology, Gross
PMC article 60K-IM
7,923
Table S3 : Source datasets for instruction-tuning data construction. The table lists the source datasets, associated specialist tools, input modalities, and number of instruction samples used to construct S1 . Dataset links are provided in footnotes.
Task
Tool
Request
Response
QA & rationale generation
PMC-LLaMA [ 47 ]
48
40
Named entity recognition
RaTE-NER [ 48 ]
40
40
Report generation (radiology)
LLaVA-Rad [ 14 ]
81
40
Report generation (retina OCT)
SpecialistVLMs [ 49 ]
90
41
Segmentation (ultrasound)
UltraSAM [ 56 ]
49
41
Segmentation (fundus)
IterNet [ 54 ]
65
—
Table S4 : Size of the request and response template inventories. For each task we report the number of distinct user-request phrasings and assistant-response templates from which samples are drawn. Inventories are frozen before corpus construction and are released with the generation scripts.
Tool chain
Workflow
BiomedCLIP → PMC-LLaMA → LLaVA-Rad
Identify the study, interpret it, then report
LLaVA-Rad → PMC-LLaMA
Generate a report, then answer a question about it
MedSAM → LLaVA-Rad
Segment a structure, then report on it
UltraSAM → LLaVA-Rad
Segment an ultrasound study, then report on it
IterNet → LLaVA-Rad
Extract retinal vessels, then report on them
Table S5 : Tool chains in the multi-tool single-round subset S3 . Each sequence is selected and ordered from a single user request before any specialist tool is executed.
Tool chain
Workflow
Successive requests on the same study (two invocations; 45 distinct chains)
BiomedCLIP → PMC-LLaMA
Classify the study, then answer a question about it
BiomedCLIP → RaTE-NER
Classify the study, then extract typed entities
BiomedCLIP → ChatCAD-R
Classify the study, then retrieve supporting knowledge
MedSAM → PMC-LLaMA
Segment a structure, then answer a question about it
LLaVA-Med → PMC-LLaMA
Answer a visual question, then reason over the answer
Table S6 : Representative tool chains in the multi-tool multi-round subset S2 . The first block shows dependent workflows in which a downstream text-based tool processes the output of an upstream specialist. The remaining blocks show independent tool sequences in which each turn introduces a separate request. These sequences illustrate the diversity of tool combinations rather than dependencies between successive invocations.
Table S9 : Training configuration of MMedAgent-Plus .
Figure S9 : LLM-as-a-judge scoring rubric. Each criterion is scored independently. Colour encodes quality from 1 (red) to 5 (blue).
Figure S10 : Biomedical question answering. MMedAgent-Plus invokes PMC-LLaMA and correctly concludes that optic nerve head torsion is more prominent in normal-tension glaucoma, whereas MedGemma-1.5-4B-IT and LLaVA-Med contradict the cited evidence.
Figure S11 : Biomedical question answering. For a question on nitric-oxide treatment of stored erythrocytes, MMedAgent-Plus returns a concise PMC-LLaMA-grounded answer, while the direct-answer baselines produce verbose responses that fail our reference matching despite reaching the correct yes/no conclusion.
Figure S12 : Biomedical question answering. MMedAgent-Plus correctly reports that GSTM1 and GSTT1 polymorphisms are not associated with gestational diabetes mellitus; MedGemma-1.5-4B-IT and LLaVA-Med omit the explicit negative association required by the reference answer.
Figure S13 : Radiology report generation. Despite the user mislabeling a chest radiograph as an MRI, MMedAgent-Plus invokes LLaVA-Rad and produces an accurate structured report; direct-answer baselines misidentify the modality or decline to interpret the image.
Figure S14 : Radiology report generation. On a lateral chest radiograph with low lung volumes, MMedAgent-Plus synthesizes a faithful LLaVA-Rad report, whereas Qwen3-VL-8B hallucinates a pulmonary mass and LLaVA-Med incorrectly calls the study normal.
Figure S15 : Radiology report generation. For a pacemaker chest radiograph with chronic interstitial markings, MMedAgent-Plus preserves LLaVA-Rad findings on device leads and lung parenchyma; baselines misread the study as hiatal hernia, pneumothorax, or pleural effusion.
Figure S16 : Named entity recognition. MMedAgent-Plus reproduces RaTE-NER typed spans for coronary-artery anatomy and abnormality phrases; direct-answer baselines return unstructured term lists without the required entity-type labels.
Figure S17 : Named entity recognition. MMedAgent-Plus identifies grade I L5 - S1 spondylolisthesis as a single Disease entity per RaTE-NER, while baselines split or paraphrase the span without matching the specialist schema.
Figure S18 : Named entity recognition. On the negated finding no intracranial hemorrhage , MMedAgent-Plus labels intracranial hemorrhage as Non-Disease following RaTE-NER; baselines either skip entity extraction or produce free-text interpretations.
Figure S19 : Image classification. MMedAgent-Plus invokes BiomedCLIP and correctly identifies immunohistochemistry staining in a histopathology panel; direct-answer baselines default to H&E or mixed-stain descriptions.
Figure S20 : Image classification. MMedAgent-Plus classifies the scan as lung CT via BiomedCLIP; baselines give generic “CT scan” answers that omit the organ-specific label required by the reference.
Figure S21 : Image classification. MMedAgent-Plus recognizes a COVID line chart through BiomedCLIP, whereas all direct-answer baselines misclassify the figure as a chest radiograph montage.
Figure S22 : Pathology classification. MMedAgent-Plus reports CONCH stromal-tissue (STR) predictions with clinical context; baselines provide generic histology descriptions without the tissue-class label.
Figure S23 : Pathology classification. MMedAgent-Plus identifies tumor tissue (TUM) via CONCH; direct-answer baselines describe tissue architecture but fail to assign the pathology class.
Figure S24 : Pathology classification. MMedAgent-Plus correctly reports CONCH background-tissue (BACK) with no pathological findings; baselines give non-committal or incorrect interpretations of the patch.
To make clinically grounded decisions, medical AI agents are expected to go beyond simple recognition and be capable of tool retrieval, evidence acquisition, and integration. Existing benchmarks largely evaluate isolated perception or single-turn question answering, and therefore provide limited visibility into failures of planning, tool recruitment, and rollout reliability. We introduce MedCTA, a benchmark for evaluating medical tool agents on clinician-validated, step-implicit tasks grounded in realistic multimodal clinical inputs, including radiology images, pathology slides, and reports. MedCTA comprises 107 real-world clinical tasks with clinician-verified executable trajectories over 5 deployed tools, and supports process-aware evaluation of tool selection, argument validity, execution stability, trajectory fidelity, and outcome quality. We benchmark 18 open- and closed-source multimodal models and find that even frontier systems remain brittle in multi-step clinical tool use: autonomous rollouts are dominated by protocol failures, premature stopping, and incorrect tool recruitment, while gold-standard tool routing yields large but still incomplete gains. These results show that strong backbone perception does not translate into reliable agentic behavior in clinical settings. MedCTA provides a rigorous testbed for auditing, diagnosing, and advancing trustworthy medical AI agents. The dataset and evaluation suite are available at https://ivul-kaust.github.io/MedCTA/
Tajamul Ashraf, Hyewon Jeong, Fida Mohammad Thoker +1
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Massachusetts Institute of Technology (MIT), USA
Multi-modal large language models (MM-LLMs) have shown strong performance in medical image understanding and clinical reasoning. Recent medical agent systems extend them with tool use and multi-agent collaboration, enabling complex decision-making. However, these systems rely almost entirely on frontier models (e.g., GPT), whose API-based deployment incurs high cost, high latency, and privacy risks that conflict with on-premise clinical requirements. We present Meissa, a lightweight 4B-parameter medical MM-LLM that brings agentic capability offline. Instead of imitating static answers, Meissa learns both when to engage external interaction (strategy selection) and how to execute multi-step interaction (strategy execution) by distilling structured trajectories from frontier models. Specifically, we propose: (1) Unified trajectory modeling: trajectories (reasoning and action traces) are represented within a single state-action-observation formalism, allowing one model to generalize across heterogeneous medical environments. (2) Three-tier stratified supervision: the model's own errors trigger progressive escalation from direct reasoning to tool-augmented and multi-agent interaction, explicitly learning difficulty-aware strategy selection. (3) Prospective-retrospective supervision: pairing exploratory forward traces with hindsight-rationalized execution traces enables stable learning of effective interaction policies. Trained on 40K curated trajectories, Meissa matches or exceeds proprietary frontier agents in 10 of 16 evaluation settings across 13 medical benchmarks spanning radiology, pathology, and clinical reasoning. Using over 25x fewer parameters than typical frontier models like Gemini-3, Meissa operates fully offline with 22x lower end-to-end latency compared to API-based deployment. Data, models, and environments are released at https://github.com/Schuture/Meissa.
Yixiong Chen, Xinyi Bai, Yue Pan +2
Johns Hopkins University, Baltimore, MD, USA · Cornell University, Ithaca, NY, USA
AI agents with tool-use capabilities show promise for integrating the domain expertise of various tools. In the medical field, however, tools are usually AI models that are inherently error-prone and can produce contradictory responses. Existing research on medical agents lacks sufficient understanding of the tools' realistic reliability and thus cannot effectively resolve tool conflicts. To address this gap, this paper introduces a framework that enables an agent to interact with tools and empirically learn their practical trustworthiness across different types of multimodal queries via agentic learning. As a concrete instantiation, we focus on chest X-ray analysis and present a tool-expertise-aware chest X-ray agent (TEA-CXA). When tool outputs disagree, the agent experimentally accepts or rejects multimodal tool results, receives rewards, and learns which tool to trust for each query type. Importantly, TEA-CXA extends existing codebases for reinforcement learning with multi-turn tool-calling that focus on textual inputs, to support multimodal contexts effectively. In addition, we enhance the codebase for medical use scenarios by supporting multiple tool calls in one turn, parallel tool inference, and multi-image accommodation within a single user query. Our code framework is applicable to general medical research on multi-turn tool-calling reinforcement learning in multimodal settings. Experiments show that TEA-CXA outperforms the state-of-the-art methods and a comprehensive set of baselines. Code will be released.
Zheang Huai, Honglong Yang, Xiaomeng Li
The Hong Kong University of Science and Technology, Kowloon, Hong Kong