Auto-MD-bench: Evaluating Autoresearch Agents for Real-World Medical Tasks

Shenqi Zhou; Qiran Zou; Wenhao Zhao; Zhenyu Liao; Dianbo Liu

cs.LGOct 18, 2026

Auto-MD-bench: Evaluating Autoresearch Agents for Real-World Medical Tasks

Authors: Shenqi Zhou, Qiran Zou, Wenhao Zhao, Zhenyu Liao, Dianbo Liu

Organizations: National University of Singapore · Zhejiang University · Amazon

Abstract

Autonomous agents are increasingly capable of developing machine learning (ML) models end to end. In medical practice, however, strong performance on the target metric does not fully characterize how good the models are in real-world use cases. For example, medical ML models should also provide well-calibrated risk estimates and remain robust across patient populations. Yet autonomous model development is typically guided by task objectives that emphasize simple predictive discrimination (e.g. AUROC). This raises a key question: are these objectives sufficient to guide agents toward models that meet the broader requirements of real-world medical use? In this study, we introduce Auto-MD-bench, a benchmark of 15 real-world medical tasks from 10 clinical datasets, spanning survival analysis, classification and sequential prediction, with external validation across cohorts, hospitals and time periods. We evaluate four agents: AIDE, AI Scientist v2, Codex, and Autoresearch. The strongest agents achieve competitive primary scores, yet their predicted risks are less calibrated than those of the simple baseline models, which could lead to severe medical accidents. Moreover, models with nearly identical primary scores can exhibit markedly different calibration, while performance varies substantially across population shifts. These findings show that primary task performance alone is insufficient to assess autonomous clinical modeling and multiple different metrics widely used in healthcare practice need to be taken into consideration.

Figures & tables

Figure 1: Overview of Auto-MD-bench. Agents develop medical prediction models under a stated primary objective. We evaluate primary task performance, risk calibration, and performance under population shift, and analyze code and development records to help explain performance differences. External validation uses test data from a different cohort, hospital, or time period; the corresponding setting without population shift uses training and test data from the same population.

Benchmark	Domain	Tasks	Calibration	External validation
			Evaluation for medical use
MLE-bench	General ML (Kaggle)	75	✗	✗
HealthAgentBench	Healthcare workflows	54	✗	✗
MedAgentBoard	Diverse medical tasks	100	✗	✗
MedAgentGym	Biomedical data science	129 †	✗	✗
AutoHealth	Health prediction	17	✓	✗

Table 1: Coverage of evaluation dimensions relevant to medical use in related agent benchmarks. ✓: reported; ✗ : not reported. Calibration denotes explicit assessment of probability calibration; external validation uses a distinct cohort, site, or time period. † MedAgentGym reports task categories.

	Normalized score $\uparrow$		Mean rank $\downarrow$		Valid
System	penalized	non-pen.	penalized	non-pen.	runs
Simple baseline models	0.487		–		–
AIDE	0.474 $\pm$ 0.032	0.474 $\pm$ 0.032	2.93 $\pm$ 0.88	3.13 $\pm$ 0.99	100.0% $\pm$ 0.0%
Codex	0.517 $\pm$ 0.003	0.517 $\pm$ 0.003	1.60 $\pm$ 0.51	1.60 $\pm$ 0.51	100.0% $\pm$ 0.0%
Autoresearch	0.518 $\pm$ 0.006	0.518 $\pm$ 0.006	1.80 $\pm$ 0.77	1.80 $\pm$ 0.77	100.0% $\pm$ 0.0%
AI Scientist v2	0.355 $\pm$ 0.064	0.467	3.67 $\pm$ 0.82	3.47 $\pm$ 0.83	73.3% $\pm$ 13.3%

Table 2: Primary task performance on 15 tasks with Qwen3.8 Flash. Scores and valid run rates are mean

\pm

SD over three repetitions; ranks are mean

\pm

SD across tasks. Penalized scores assign zero to invalid runs; non-penalized scores exclude them. Bold: best agent value per metric.

	All	Survival analysis	Classification	Sequential prediction
AIDE	0.128	0.045	0.280	0.066
Codex	0.086	0.076	0.113	0.065
Autoresearch	0.075	0.061	0.087	0.090
AI Scientist v2	0.072	0.048	0.102	0.078
Simple baseline models	0.027	0.027	0.031	0.018

Table 3: Expected calibration error (ECE; lower is better) by agent and task class. Bold: lowest value per column.

Figure 2: Calibration error and modeling practices. (a) Contributions to the ECE gap relative to task-specific simple baseline models. (b) ECE after agents handle class imbalance, with or without recalibration. Circles show individual experiments; diamonds show group means. Group definitions are in Appendix I .

Objective	AUROC $\uparrow$	ECE $\downarrow$
AUROC	0.676 $\pm$ 0.023	0.172 $\pm$ 0.211
Brier score	0.640 $\pm$ 0.107	0.061 $\pm$ 0.157
AUROC and ECE	0.683 $\pm$ 0.016	0.016 $\pm$ 0.023

Table 4: Calibration under three evaluation objectives on UCI Diabetes (mean

\pm

SD over valid runs). Bold: best value across objectives.

External validation setting	Simple baseline models	AIDE	Codex	Autoresearch	AI Scientist v2
Rotterdam $\to$ GBSG (cohort)	0.372 (100%)	0.306 (86%)	0.385 (99%)	0.371 (97%)	0.332 (85%)
SEER (temporal)	0.724 (99%)	0.730 (97%)	0.745 (103%)	0.735 (99%)	0.670 ∗ (–)
eICU (hospital)	0.736 (99%)	0.748 (99%)	0.751 (100%)	0.751 (100%)	0.705 (100%)
Sepsis (hospital)	0.199 (52%)	0.168 (41%)	0.231 (52%)	0.268 (61%)	0.228 ∗ (63%)
Sepsis + 500 target labels	0.248 (65%)	0.217 (54%)	0.299 (67%)	0.253 (57%)	0.124 ∗ (34%)

Table 5: Performance under population shift. Entries are mean normalized scores; parentheses give ratios to the corresponding score without the population shift. Bold: best score per setting; ∗ one valid run.

Model family
Method	AIDE	Codex	Autoresearch	AI Scientist v2
Gradient-boosted trees	87% $\pm$ 0%	89% $\pm$ 8%	89% $\pm$ 8%	76% $\pm$ 13%
Neural networks	11% $\pm$ 8%	31% $\pm$ 17%	38% $\pm$ 10%	36% $\pm$ 10%
Ensembling and validation
Averaging across seeds or folds	62% $\pm$ 14%	89% $\pm$ 4%	89% $\pm$ 10%	31% $\pm$ 7%
Single model only	13% $\pm$ 12%	2% $\pm$ 4%	4% $\pm$ 8%	36% $\pm$ 10%

Table 6: Modeling methods in valid experiments (mean

\pm

SD over three repetitions; categories can overlap).

Figure 3: Modeling changes during agent search. Points show the percentage of sampled solution pairs containing each type of modeling change. Early, middle, and late indicate relative positions in the chronologically ordered sequence of solution pairs within each run, rather than equal intervals of elapsed time. Pairs may contain multiple types of change.

LLM	Normalized score $\uparrow$	ECE $\downarrow$
Qwen3.8 Flash	0.517 $\pm$ 0.003	0.086
gpt-6-luna	0.517 $\pm$ 0.004	0.041
Gemini 3.8 Flash	0.520 $\pm$ 0.006	0.029

Table 7: Codex with three LLMs. Scores are mean

\pm

SD over three repetitions; ECE is averaged across tasks. Bold: best value per metric.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Data source	Outcome	Validation	Metric	Train / test
Survival analysis
SUPPORT	Cohort	Death	Internal	$C^{td}$	7,098 / 1,775
METABRIC	Cohort	Death	Internal	$C^{td}$	1,523 / 381
Rotterdam	Cohort	Recurrence or death	Internal	$C^{td}$	2,385 / 597
Rotterdam $\to$ GBSG	Cohort $\to$ trial	Recurrence or death	External (cohort)	$C^{td}$	2,982 / 686
SEER †	Registry	Breast-cancer death	Internal	$C^{td}$	232,625 / 49,291

Appendix

Table 8: The 15 tasks in Auto-MD-bench. Sample sizes are patients unless noted; UCI Diabetes counts encounters. Matching symbols indicate shared test sets.

C^{td}

: time-dependent concordance; Utility: PhysioNet/CinC 2019 challenge score.

Dataset	Content	Access
SUPPORT ( Connors et al., 1995 )	$\sim$ 9,100 seriously ill inpatients, 5 US hospitals, 1989–94	public
METABRIC ( Curtis et al., 2012 )	$\sim$ 2,000 breast tumors with genomic profiles	public
Rotterdam, GBSG ( Royston and Altman, 2013 )	breast cancer registry (1978–93) and trial (1984–89)	public
UCI Diabetes ( Strack et al., 2014 ; Clore et al., 2014 )	101,766 encounters involving patients with diabetes, 130 US hospitals, 1999–2008	public
PhysioNet/CinC 2012 ( Silva et al., 2012 )	12,000 ICU stays, first 48 h	public
PhysioNet/CinC 2019 ( Reyna et al., 2019b ; Reyna et al., 2019a )	hourly ICU records labeled for sepsis onset, 40,336 patients, 2 hospital systems	public

Appendix

Table 9: Source datasets.

Agent	Type	Search strategy	Submission selection
Codex	coding agent	agent-directed	final submission.csv
Autoresearch	coding agent	greedy search	final submission.csv
AIDE	research agent	greedy tree search	best self-reported validation score
AI Scientist v2	research agent	best-first tree search	submission from the selected node in the latest stage with a saved submission

Appendix

Table 10: Agent systems, search strategies and submission selection.

Agent	Input	Output
AIDE	0.65 $\pm$ 0.05	0.28 $\pm$ 0.02
Codex	26.88 $\pm$ 1.86	0.17 $\pm$ 0.01
Autoresearch	51.44 $\pm$ 13.23	0.30 $\pm$ 0.08
AI Scientist v2	4.29 $\pm$ 0.63	1.69 $\pm$ 0.24

Appendix

Table 11: Tokens per run (millions; mean

\pm

SD over three repetitions).

Task	Simple baseline models	AIDE	Codex	Autoresearch	AI Scientist v2
SUPPORT	0.622	$0.639\pm 0.010$	$0.651\pm 0.000$	$0.656\pm 0.005$	$0.630\pm 0.013$
METABRIC	0.642	$0.658\pm 0.004$	$0.659\pm 0.008$	$0.676\pm 0.007$	$0.647\pm 0.018^{\dagger}$
Rotterdam	0.685	$0.677\pm 0.022$	$0.694\pm 0.002$	$0.692\pm 0.004$	$0.694\pm 0.002$
Rotterdam $\to$ GBSG	0.686	$0.653\pm 0.028$	$0.693\pm 0.007$	$0.686\pm 0.021$	$0.666\pm 0.023$
SEER	0.865	$0.749\pm 0.216$	$0.862\pm 0.019$	$0.870\pm 0.003$	$0.861\pm 0.011^{\dagger}$
SEER, later period	0.862	$0.865\pm 0.014$	$0.872\pm 0.002$	$0.868\pm 0.011$	$0.835^{\ddagger}$

Appendix

Table 12: Primary scores by task for valid runs (mean

\pm

SD).

\dagger

Two valid runs;

\ddagger

one valid run.

Task	Simple baseline models	AIDE	Codex	Autoresearch	AI Scientist v2
SUPPORT	0.032	$0.049\pm 0.027$	$0.035\pm 0.008$	$0.044\pm 0.013$	$0.057\pm 0.036$
METABRIC	0.054	$0.048\pm 0.003$	$0.132\pm 0.152$	$0.081\pm 0.033$	$0.053\pm 0.001^{\dagger}$
Rotterdam	0.042	$0.059\pm 0.045$	$0.042\pm 0.011$	$0.055\pm 0.027$	$0.082\pm 0.050$
Rotterdam $\to$ GBSG	0.039	$0.112\pm 0.096$	$0.040\pm 0.001$	$0.160\pm 0.170$	$0.103\pm 0.111$
SEER	0.006	$0.027\pm 0.040$	$0.015\pm 0.010$	$0.016\pm 0.011$	$0.008\pm 0.005^{\dagger}$
SEER, later period	0.007	$0.008\pm 0.005$	$0.253\pm 0.210$	$0.005\pm 0.001$	$0.013^{\ddagger}$

Appendix

Table 13: ECE by task for valid runs (mean

\pm

SD).

\dagger

Two valid runs;

\ddagger

one valid run.

Task	Grouping	Simple baseline models’ lowest group	Same lowest group	Agent gap	Simple baseline models’ gap
Rotterdam	Age	Oldest	92%	0.074	0.089
Rotterdam $\to$ GBSG	Age	Oldest	25%	0.019	0.033
SEER	Age	Youngest	73%	0.023	0.024
SEER	Race/ethnicity	American Indian/Alaska Native	73%	0.036	0.049
SEER, later period	Age	Youngest	80%	0.027	0.026
SEER, later period	Race/ethnicity	American Indian/Alaska Native	70%	0.035	0.048

Appendix

Table 14: Subgroup performance gaps across valid runs. “Same lowest group” is the share matching the simple baseline models’ lowest-scoring subgroup.

Agent	Objective	Valid	AUROC	Brier score	ECE
AIDE	AUROC	3/3	0.659 $\pm$ 0.042	0.335 $\pm$ 0.068	0.440 $\pm$ 0.090
AIDE	Brier score	3/3	0.606 $\pm$ 0.142	0.099 $\pm$ 0.010	0.037 $\pm$ 0.054
AIDE	AUROC and ECE	3/3	0.685 $\pm$ 0.005	0.095 $\pm$ 0.002	0.022 $\pm$ 0.027
Codex	AUROC	3/3	0.682 $\pm$ 0.011	0.161 $\pm$ 0.117	0.133 $\pm$ 0.221
Codex	Brier score	3/3	0.688 $\pm$ 0.001	0.093 $\pm$ 0.000	0.005 $\pm$ 0.002
Codex	AUROC and ECE	3/3	0.690 $\pm$ 0.002	0.093 $\pm$ 0.000	0.005 $\pm$ 0.001

Appendix

Table 15: Results by agent and evaluation objective on UCI Diabetes (mean

\pm

SD over valid runs).

Figure 4: AUROC and ECE under three evaluation objectives on UCI Diabetes. Points are valid runs; bars show means and dashed lines the simple baseline model. Colors mark the two modeling practices associated with large calibration error.

Practice	AUROC (A)	Brier score (B)	AUROC and ECE (C)
Averaged ranks as probabilities	3	0	0
Class-imbalance handling without recalibration	3	1	0
Explicit probability calibration	1	6	10

Appendix

Table 16: Modeling practices under the three UCI Diabetes objectives. Each objective has 11 valid runs; categories can overlap.

Agent	Task	Repetition	Problem
Codex	SUPPORT	2	The labels of the XGBoost Cox member are all positive, so censored patients are treated as events (about 20%)
Codex	SUPPORT	3	The labels of the XGBoost Cox member are all positive (about 20%)
Codex	SEER, later period	1	The XGBoost Cox member adds 100,000 to censoring times, so censored patients stay in the risk set throughout (about 37% on average, about half after 36 months)
Autoresearch	SEER	1	The labels of the XGBoost Cox member are not negated for censored patients (about 1.5%)
Autoresearch	SEER	3	The binary members for 24, 48 and 60 months label patients censored before the time point as survivors (about 27%)
Autoresearch	NHANES	1	The per-time-point IPCW classifiers take weights from a KM estimate that ignores the survey cycle, while censoring is determined entirely by the cycle; predicted survival in some cycles collapses to 0.1–0.3 (about 71%)

Appendix

Table 17: Censoring errors in valid survival submissions; parentheses show the affected ensemble weight.

Figure 5: Test performance over development time, averaged across runs; curves are zero before the first valid submission.

	AIDE	Codex	Autoresearch	AI Scientist v2
Versions per run	25.3	6.4	7.8	41.0
Time to first valid version (min)	30	64	17	30
Run time (min)	720	295	429	720
Ended before 12 h	0%	98%	100%	0%
Selected submission has highest test score	22%	32%	36%	21%
Reported validation $-$ test score †	$+0.047$	$+0.006$	$+0.008$	$+0.012$

Appendix

Table 18: Development and submission-selection summary.

Change	Score increase	Score decrease	Mean score change
	Share of pairs
Ensembling	58%	41%	$+0.010$
Hyperparameter tuning	50%	49%	$+0.004$
Feature engineering	51%	46%	$+0.015$
Model family	79%	18%	$+0.084$
Bug fix	66%	32%	$+0.059$

Appendix

Table 19: Score changes for sampled version pairs containing each revision type, pooled across agents. Mean score change is later minus earlier normalized test score.

Practice	AIDE	Codex	Autoresearch	AI Scientist v2
Averaged ranks as probabilities	20%	18%	9%	6%
Class-imbalance handling, no recalibration	21%	4%	13%	35%
Performance checked by age or sex	0%	4%	9%	0%
Validation under population shift	33%	89%	89%	40%
Missingness features	81%	89%	89%	58%

Appendix

Table 20: Modeling practices by agent (percentage of eligible valid runs).

Comparison	Agreement
Modeling methods, repeated annotation	95% (130 fields, 10 runs)
Modeling practices, repeated annotation	95% (288 fields, 36 runs)
Missingness features vs. earlier records	91% (100 runs)
Validation under population shift vs. earlier checks	95% (22 runs)

Appendix

Table 21: Agreement in repeated annotations and comparisons with earlier code reviews.

LLM	Normalized score $\uparrow$	ECE $\downarrow$	Valid runs	Input tokens	Output tokens
Qwen3.8 Flash	0.517 $\pm$ 0.003	0.086	100.0% $\pm$ 0.0%	26.88 $\pm$ 1.86	0.17 $\pm$ 0.01
gpt-6-luna	0.517 $\pm$ 0.004	0.041	100.0% $\pm$ 0.0%	2.77 $\pm$ 0.12	0.04 $\pm$ 0.00
Gemini 3.8 Flash	0.520 $\pm$ 0.006	0.029	100.0% $\pm$ 0.0%	7.29 $\pm$ 0.24	0.05 $\pm$ 0.00

Appendix

Table 22: Codex with three LLMs across 15 tasks. Values are mean

\pm

SD over three repetitions; ECE is averaged across tasks and token counts are millions per run.

Medical AI Agents

See all 39 papers

AutoMedBench: Towards Medical AutoResearch with Agentic AI Models
Jun 1, 2026Junqi Liu, Selena Song, Yuhan Wang +12AI Agent Evaluation AI Agent Benchmarks
University of California, Santa Cruz · 2NVIDIA
HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents
Jun 30, 2026Qianchu Liu, Sheng Zhang, Guanghui Qin +18Long-Horizon Agent Evaluation AI Agent Benchmarks
Microsoft Research
PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
May 4, 2026Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler +10AI Agent Benchmarks Electronic Health Records
Stanford University

Jun 1, 2026J/K move · Enter open · S save

AutoMedBench: Towards Medical AutoResearch with Agentic AI Models

Junqi Liu, Selena Song, Yuhan Wang, Jiawei Mao+11

University of California, Santa Cruz · 2NVIDIA

AI Agent Evaluation AI Agent Benchmarks Scientific Workflow Automation Autonomous Research Agents Medical AI Agents

Open paper AskPDF arXiv

Autonomous agents are increasingly expected to support end-to-end medical-AI research workflows, moving beyond isolated prediction tasks or short-form clinical question answering. However, existing medical agent benchmarks primarily evaluate final outputs, providing limited visibility into agent behavior within the research process. To address this gap, we present AutoMedBench, a workflow-aware benchmark for autonomous medical-AI research across diverse medical imaging and multimodal inference tasks, organizing agent execution into a unified five-stage workflow (S1-S5): Plan, Setup, Validate, Inference, and Submit. It comprises long-horizon tasks with each run averaging 33 agent turns, spanning five research tracks: segmentation, image enhancement, visual question answering (VQA), report generation, and lesion detection. Each task is evaluated under two difficulty tiers, Lite and Standard, which use the same data and metrics but differ in the amount of task-brief scaffolding, and each run is scored using both final task performance and S1-S5 stage scores, enabling stage-level analysis from the initial task brief to the final submitted artifact. Across thousands of recorded runs, stage-level scoring reveals that Validate is the weakest workflow stage on average, whereas Setup is the strongest, suggesting that current agents are better at making pipelines executable than at verifying their reliability. Post-run error analysis further shows that verification and submission failures dominate tagged errors, accounting for 37.7% and 38.1% of fired codes respectively, whereas task-understanding errors are rare at 0.9%, and runs with one fired error code have a 48% lower overall score than runs with no error code on average.

Abstract

Medical AI Agents

Explore similar work

AutoMedBench: Towards Medical AutoResearch with Agentic AI Models

HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents

PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments