A Multi-Agent Framework for Automated Machine Learning with Evidence-based Planning for Medical Image Analysis

Yiqing Shen; Mathias Unberath

cs.CVOct 20, 2026

A Multi-Agent Framework for Automated Machine Learning with Evidence-based Planning for Medical Image Analysis

Authors: Yiqing Shen, Mathias Unberath

Organizations: Department of Computer Science, Johns Hopkins University

Abstract

Deep learning has become the default tool for medical image analysis, driven by the open-source movement. However, achieving the best performance on a given task remains difficult, with architecture selection and hyperparameter optimization (HPO) still considered "more of an art than a science". Automated machine learning (AutoML) automates parts of model training, yet helps little in medical imaging: general-purpose approaches lack medical architectures and preprocessing, and medical-focused ones demand engineering expertise and remain confined to narrow tasks such as 3D segmentation. We introduce MedicalMLOmni, a multi-agent AutoML framework that automates model training for diverse single- and multi-modal medical tasks through a natural language interface. Specifically, MedicalMLOmni performs evidence-based planning by retrieving published results to guide model selection and configuration, and employs parallel HPO where multiple agents explore adjustment strategies concurrently. MedicalMLOmni achieves the best score on 19 of 20 benchmarks; PMC-VQA is the exception, where a vision-language model with broader pre-training leads. Backbone-matched controls attribute the gains to the framework, not the backbone. By removing coding requirements, MedicalMLOmni lowers the barrier for domain experts with rich datasets but limited engineering expertise.

Figures & tables

System	Config. source	Refinement
AutoM3L [ 32 ]	LLM prior	Bayesian opt.
MLZero [ 9 ]	LLM prior + memory	Episodic feedback
M 3 Builder [ 10 ]	Prompt templates	Exec. debugging
mAIstro [ 42 ]	Expert prompts	Exec. feedback
AutoHealth [ 46 ]	LLM prior	Pred. uncertainty
AMID [ 31 ]	LLM prior	Audit trail

Table 1: Concurrent multi-agent medical AutoML systems differ in where the training configuration originates. All of them share the agentic decomposition. Config. source: where the architecture, loss and hyperparameters come from; Refinement: how a configuration is revised after a run.

Figure 1: Overview of MedicalMLOmni showing evidence-based planning through literature retrieval, task-specific execution agents, and parallel hyperparameter optimization.

Method	AMOS [ 20 ]	BraTS23 [ 23 ]	KiTS23 [ 16 ]	BTCV [ 25 ]	ACDC [ 6 ]	TotalSeg [ 44 ]	MedSeg [ 24 ]	Touchstone [ 5 ]	Avg.
	CT/MRI	MRI	CT	CT	MRI	CT	Mixed	CT
nnU-Net [ 18 ]	0.893 $\pm$ 0.004	0.929 $\pm$ 0.003	0.831 $\pm$ 0.005	0.866 $\pm$ 0.004	0.914 $\pm$ 0.003	0.921 $\pm$ 0.004	0.884 $\pm$ 0.005	0.889 $\pm$ 0.004	0.891 $\pm$ 0.004
MedNeXt-L [ 35 ]	0.888 $\pm$ 0.005	0.936 $\pm$ 0.003	0.825 $\pm$ 0.006	0.875 $\pm$ 0.004	0.920 $\pm$ 0.003	0.898 $\pm$ 0.005	0.876 $\pm$ 0.006	0.883 $\pm$ 0.005	0.888 $\pm$ 0.005
STU-Net-L [ 17 ]	0.891 $\pm$ 0.004	0.931 $\pm$ 0.004	0.820 $\pm$ 0.007	0.852 $\pm$ 0.006	0.895 $\pm$ 0.005	0.891 $\pm$ 0.005	0.863 $\pm$ 0.007	0.871 $\pm$ 0.006	0.877 $\pm$ 0.005
SegResNet [ 33 ]	0.861 $\pm$ 0.006	0.910 $\pm$ 0.005	0.808 $\pm$ 0.008	0.819 $\pm$ 0.007	0.889 $\pm$ 0.006	0.878 $\pm$ 0.006	0.857 $\pm$ 0.008	0.856 $\pm$ 0.007	0.860 $\pm$ 0.007
VISTA3D [ 15 ]	0.850 $\pm$ 0.007	0.912 $\pm$ 0.005	0.837 $\pm$ 0.006	0.845 $\pm$ 0.006	0.903 $\pm$ 0.005	0.912 $\pm$ 0.005	0.871 $\pm$ 0.006	0.862 $\pm$ 0.007	0.874 $\pm$ 0.006

Table 2: MedicalMLOmni reaches the highest Dice on all eight segmentation benchmarks. Dice

\uparrow

as mean

\pm

standard deviation over repeated runs; the second header row gives the imaging modality. Bold: best; underline: second best.

Figure 2: Segmentation analysis. (a) Per-benchmark Dice improvement over nnU-Net with propagated standard deviation as error bars. (b) Cross-benchmark standard deviation (solid) and average per-run standard deviation (hatched) for each method, measuring performance consistency. (c) Average Dice gain over four baselines grouped by imaging modality, with error bars indicating propagated uncertainty.

Figure 3: VQA and multimodal analysis. (a) Per-benchmark VQA accuracy gain of MedicalMLOmni over the respective second-best method, with propagated standard deviation as error bars. (b) Performance profile showing the fraction of segmentation benchmarks on which each method is within a given Dice gap from the best. (c) Per-benchmark performance gain of MedicalMLOmni over three multimodal baselines, with propagated uncertainty.

Method	VQA-RAD [ 26 ]	SLAKE [ 29 ]	PathVQA [ 14 ]	PMC-VQA [ 48 ]	GEMeX [ 30 ]	MedXpert [ 50 ]	Kvasir [ 11 ]	Avg.
	X-ray	Multi	Path	Multi	X-ray	Clinical	Endo
MedVInT-TD [ 48 ]	73.9 $\pm$ 0.6	84.7 $\pm$ 0.6	55.8 $\pm$ 0.9	35.2 $\pm$ 0.9	70.8 $\pm$ 0.7	24.6 $\pm$ 1.0	61.3 $\pm$ 0.8	58.0 $\pm$ 0.8
MedVInT-TE [ 48 ]	69.5 $\pm$ 0.7	88.4 $\pm$ 0.5	48.8 $\pm$ 1.0	36.6 $\pm$ 0.9	68.2 $\pm$ 0.8	23.8 $\pm$ 1.1	48.8 $\pm$ 1.1	54.9 $\pm$ 0.9
BiomedGPT [ 47 ]	61.1 $\pm$ 0.8	86.3 $\pm$ 0.5	52.3 $\pm$ 0.9	24.1 $\pm$ 1.1	66.5 $\pm$ 0.8	21.7 $\pm$ 1.2	58.6 $\pm$ 0.9	52.9 $\pm$ 0.9
LLaVA-Med-v1.5 [ 27 ]	64.7 $\pm$ 0.7	82.1 $\pm$ 0.6	55.5 $\pm$ 0.9	29.1 $\pm$ 1.0	73.4 $\pm$ 0.6	28.6 $\pm$ 1.0	68.7 $\pm$ 0.7	57.4 $\pm$ 0.8
AutoGluon [ 40 ]	40.8 $\pm$ 1.3	59.5 $\pm$ 1.2	35.1 $\pm$ 1.5	15.8 $\pm$ 1.4	46.4 $\pm$ 1.3	14.2 $\pm$ 1.5	37.9 $\pm$ 1.4	35.7 $\pm$ 1.4

Table 3: MedicalMLOmni is best on six of the seven VQA benchmarks and on average. Accuracy

\uparrow

in % as mean

\pm

standard deviation over repeated runs; the second header row gives the image domain. Bold: best; underline: second best.

Method	BraTS-Sur [ 4 ]	MIMIC [ 22 ]	NLST [ 34 ]	ADNI [ 45 ]	ISIC [ 12 ]	Avg.
	MRI+Clin.	X-ray+Text	CT+EHR	MRI+PET	Img+Meta	rank
	Acc $\uparrow$	AUC $\uparrow$	AUC $\uparrow$	Acc $\uparrow$	pAUC $\uparrow$	$\downarrow$
Fusion Trans. [ 36 ]	0.712 $\pm$ 0.015	0.853 $\pm$ 0.011	0.847 $\pm$ 0.012	0.812 $\pm$ 0.014	0.197 $\pm$ 0.007	2.00
AutoGluon [ 40 ]	0.645 $\pm$ 0.019	0.752 $\pm$ 0.016	0.746 $\pm$ 0.018	0.718 $\pm$ 0.019	0.145 $\pm$ 0.012	3.00
AutoKeras [ 21 ]	0.632 $\pm$ 0.020	0.741 $\pm$ 0.017	0.732 $\pm$ 0.019	0.705 $\pm$ 0.020	0.138 $\pm$ 0.013	4.40
MLZero [ 9 ]	0.627 $\pm$ 0.020	0.735 $\pm$ 0.017	0.741 $\pm$ 0.018	0.698 $\pm$ 0.020	0.134 $\pm$ 0.013	4.80

Table 4: MedicalMLOmni ranks first on all five multimodal benchmarks. Primary metrics vary by task and are given as mean

\pm

standard deviation over repeated runs; average rank is computed per benchmark and then averaged. Bold: best; underline: second best.

Figure 4: Ablation analysis on: (a) Component ablation of Tab. 5 on four representative benchmarks, where

\times

3 and

\times

5 denote three and five parallel HPO agents. (b) Parallel agent scaling effects on performance and time. (c) Data efficiency across different training set sizes.

Evd.	Top- $k$	HPO	Agents	AMOS	BraTS	KiTS	BTCV	ACDC	TSeg	MSeg	Touch	Avg.
✗	✗	✗	0	.881	.922	.834	.858	.907	.908	.873	.877	.883
✓	✗	✗	0	.889	.931	.840	.867	.915	.915	.881	.885	.890
✓	✓	✗	0	.895	.938	.845	.873	.921	.922	.887	.892	.897
✓	✓	✓	1	.901	.944	.848	.881	.927	.929	.893	.898	.903
✓	✓	✓	3	.905	.948	.850	.886	.931	.934	.896	.902	.907
✓	✓	✓	5	.908	.951	.852	.890	.935	.939	.899	.905	.910

Table 5: Every component adds Dice on every segmentation benchmark. Evd.: evidence-based retrieval; Top-

k

: multi-source synthesis; HPO: hyperparameter optimization; Agents: number of parallel HPO agents. Dice

\uparrow

; bold: best; underline: second best.

Setting	Backbone	Dice $\uparrow$	VQA $\uparrow$	MM $\downarrow$
Default (no framework)	Qwen3-30B	0.795	59.3	3.80
Planning w/o evidence	Qwen3-30B	0.883	63.4	2.20
Best baseline †	Task-specific	0.898	65.1	1.40
Domain expert	Task-specific	0.901	66.5	1.20
MedicalMLOmni	Qwen3-8B	0.885	62.1	2.60
MedicalMLOmni	Llama-3.3-70B	0.907	68.2	1.10

Table 6: The framework, not the backbone, accounts for the improvement. The upper block removes MedicalMLOmni with the LLM fixed; the lower block swaps the LLM with MedicalMLOmni fixed, and the gains persist outside the Qwen3 family. Dice: segmentation; VQA: accuracy in %; MM: multimodal average rank; † tuning budget matched to our 18 GPU-hours. Bold: best; underline: second best.

Corpus	Variation	Overlap $\uparrow$	Agreement $\uparrow$	Dice $\uparrow$
Live API	Baseline	0.62	84%	0.910
Frozen	Baseline	1.00	100%	0.910
Frozen	Paraphrased	0.88	96%	0.908
Frozen	Bench. excl.	0.58	80%	0.904
Frozen	Pre-bench.	0.53	76%	0.897
Frozen	$\tau$ $\pm$ 10%	1.00	100%	0.895

Table 7: A frozen snapshot makes planning exactly reproducible. Most of the advantage survives when benchmark-specific sources are withheld. Overlap: mean Jaccard index between the top-

k

source sets of independent retrieval runs; Agreement: fraction of runs selecting the same architecture and loss; Dice: mean over the eight segmentation benchmarks.

Task family	Reference	$\Delta$	95% CI	$p$
Segmentation (8)	nnU-Net	+0.019	[+0.014, +0.024]	$<$ 0.001
VQA (7)	Qwen3-VL	+3.5%	[+1.8%, +5.2%]	0.015
Multimodal (5)	Fusion Trans. [ 36 ]	+1.0 Rank	[+0.4, +1.6]	0.022

Table S1 : MedicalMLOmni leads the strongest baseline of each task family at

p<0.05

\Delta

: mean difference; CI: 95% bootstrap confidence interval;

p

: two-sided Wilcoxon signed-rank test.

Task family	Modal architecture	Modal loss	Match
Segmentation (8)	3D U-Net	DiceCE	88%
VQA (7)	ViT+LLM (LoRA)	Cross-Entropy	71%
Multimodal (5)	Cross-Attention	Focal Loss	40%

Table S2 : Segmentation plans match the best published recipe most often, multimodal plans least often. Modal plan per task family; Match: fraction of benchmarks whose synthesized architecture and loss coincide with those of the best published recipe.

Figure S1 : Computational cost and cross-task analysis. (a) Performance vs. computational cost trade-off, where each point represents a method’s average Dice score (

\pm\sigma

) and total GPU-hours across 8 segmentation benchmarks. (b) Cross-task generalization: bar groups show normalized scores on segmentation, VQA, and multimodal tasks for the methods evaluated in all three categories, and the dashed rules mark the specialist baselines of each task, namely nnU-Net and Auto3DSeg, Qwen3-VL and LLaVA-Med, and Fusion Transformer [ 36 ] .

Stage	GPU-hours	Share
Planning and retrieval (CPU/API)	0.5	3%
Initial training	4.5	25%
HPO retraining	12.0	67%
Validation and inference	1.0	6%
Total	18.0	100%

Table S3 : HPO retraining takes two thirds of the 18 GPU-hours. GPU-hour breakdown across the eight segmentation benchmarks.

Automated Machine Learning

Automated machine learning (AutoML) builds and tunes data-to-model pipelines with minimal manual effort, balancing predictive quality, compute, and deployment constraints.

See all 96 papers

Can Coding Agents Build Robust Baselines? A Skill-Based Approach for Automating the Medical Imaging Model-Development Pipeline
Aug 24, 2026Eugenia Moris, José Ignacio OrlandoAutomated Machine Learning AI Agents for Scientific Discovery
Arionkoder LLC, Boston, MA 02114, United States · Yatiris, PLADEMA, UNICEN-CONICET, Tandil, Buenos Aires, Argentina
Towards Autonomous and Auditable Medical Imaging Model Development
Jul 12, 2026Shengyuan Liu, Jia-Xuan Jiang, Boyun Zheng +8Medical Imaging Large Language Model-Guided Optimization
The Chinese University of Hong Kong · Institute of Automation, Chinese Academy of Sciences · Independent Researcher +2
AutoMedBench: Towards Medical AutoResearch with Agentic AI Models
Jun 1, 2026Junqi Liu, Selena Song, Yuhan Wang +12AI Agent Evaluation AI Agent Benchmarks
University of California, Santa Cruz · 2NVIDIA

Aug 24, 2026J/K move · Enter open · S save

Can Coding Agents Build Robust Baselines? A Skill-Based Approach for Automating the Medical Imaging Model-Development Pipeline

Eugenia Moris, José Ignacio Orlando

Arionkoder LLC, Boston, MA 02114, United States · Yatiris, PLADEMA, UNICEN-CONICET, Tandil, Buenos Aires, Argentina

Automated Machine Learning AI Agents for Scientific Discovery Medical Image Analysis LLM Agent Workflow Optimization

Open paper AskPDF arXiv

Developing competitive deep learning baselines for medical imaging remains a highly iterative process requiring literature review, implementation, experimentation, and expert refinement. Existing automation approaches typically optimize isolated components, such as architecture search or hyperparameter tuning, rather than the complete baseline development process. We present an agentic AI Scientist workflow that combines literature-guided reasoning, automated code generation, and hypothesis-driven experimentation to generate competitive baseline models for medical imaging challenges. The framework is evaluated on four public benchmarks spanning segmentation, classification, and detection. Across all tasks, the Experimentation Pipeline consistently improves validation performance, achieving competitive leaderboard results, including 6th place on both PUMA tracks (15 teams) and 31st place on MILK10k (125 teams). On MIDOG25, the resulting model also demonstrates strong domain generalization across scanners, tumor types, and species. Using the same workflow across all challenges without task-specific redesign, we demonstrate that skill-based, literature-guided agentic workflows can substantially reduce the engineering effort required to develop competitive medical imaging baselines.

Abstract

Automated Machine Learning

Explore similar work

Can Coding Agents Build Robust Baselines? A Skill-Based Approach for Automating the Medical Imaging Model-Development Pipeline

Towards Autonomous and Auditable Medical Imaging Model Development

AutoMedBench: Towards Medical AutoResearch with Agentic AI Models