BioXArena-Pro: a research-grade benchmark for autonomous biomedical machine learning agents
Authors: Loka Li, Duzhen Zhang, Xingbo Du, Leonard Song, Kun Zhang, Le Song
Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates. · The Westminster Schools, Atlanta, Georgia, United States. · Carnegie Mellon University, Pittsburgh, Pennsylvania, United States.
Coding agents are increasingly proposed as autonomous builders of biomedical models, but existing benchmarks are small, near saturation or open to leaked information. We present BioXArena-Pro, 64 research-level modelling tasks from eight biomedical domains, each rebuilt from a published study; most hold out whole biological units such as species or cohorts. Across 22 agent configurations, the best scores 55.1 of 100, and the scaffold matters almost as much as the model. We re-ran a published method for every task: where the published recipe applies unchanged, the best agent and the methods tie; on rebuilt tasks the methods trail by 8.2 points, suggesting that agents gain by adapting to new data. Pretrained models (including biological foundation models) matter where the strongest agent uses them: forbidding them decreases its score on those tasks by 9 points. Every task was audited in several AI-assisted rounds and a final human review.
Figures & tables
Figure 1: BioXArena-Pro is a larger, more heterogeneous and harder successor to BioXArena. a , The 64 tasks, one row per domain; a solid border marks a test set that holds out a biological or clinical unit (out-of-distribution split), a dashed border an in-distribution split. b , Training units per task (log scale) for BioXArena (grey, 76 tasks) and BioXArena-Pro (red, 64 tasks); violins show the distribution, points are tasks, bars are medians and the title gives the total. c , Public data per task, drawn as in b . d , Share of tasks by split design, from random, source-defined or other in-distribution splits (greys) to held-out scaffolds, entities, donors, cohorts, cell lines, species and time windows (reds); the label gives the out-of-distribution share. e , Normalized task scores ( ×100 ) of the four agent configurations run on both benchmarks, on the common BioXArena-Pro scale (Methods); each point is one task, bars are means and the number on the right is the mean over tasks.
Figure 2: Capability of agent configurations on the final release. a , Mean score over the 64 tasks for each of the 22 configurations (bars; the axis starts at 25), with 95% bootstrap CIs over tasks (5,000 resamples); failed tasks score 0 and their count is given next to the interval. Dotted line, the best mean. The two columns to the right give the agent wall-clock time summed over the 64 tasks and the input (bar) and output (small number) tokens in millions; input includes cached prompt tokens. Colour marks the backbone (one colour family per vendor, the newer backbone darker). b , Domain means per configuration, rows ordered as in a ( n=8 tasks per domain). c , Scaffold–backbone design: mean score of four model-agnostic scaffolds with four backbones, with row and column means. d , Vendor-CLI design: each coding CLI with the older and the newer of its vendor’s two backbones (arrow), and OpenCode with the newer backbone (open diamond). e , Agent wall-clock time against mean score; red, the search-based MLEvolve scaffold.
Figure 3: Reproduced published-method references match agents only where the test set stays close to the original study. a , Per task, the reproduced published-method reference (the primary method, chosen by validation on training data) against Codex CLI with GPT-6 Astra; colour marks the domain as in Fig. 1 , filled points are tier-1 methods (published for this dataset) and open points tier-2 methods (strongest published method for the same problem class, adapted); labels name selected methods that beat the agent and tasks with the largest agent lead. b , Agent minus published-method reference per task, grouped by how close our test set is to the original study: close (same data and metric family), rebuilt (new split, removed inputs or new metric) or mixed (the grade differs between instances or applies to part of the task); bars are means. c , Mean score over tasks for the published-method reference, the two strongest configurations and the per-task best of the 22 configurations, for all tasks, the published-method baseline (tier-1 methods on tasks close to the original study) and the tier-1, tier-2 and 2024–2026 subsets. All scores ×100 .
Figure 4: Pretrained models: what the configurations use and what they are worth. a , Share of runs whose final code uses each model family, per configuration (22 configurations × 64 tasks, 1,401 runs with code; families are not exclusive). b , Codex CLI with GPT-6 Astra on the 14 tasks whose default solution already relied on a pretrained model: change in score when the prompt forbids all pretrained weights and embeddings; hatched bars, tasks whose default used task-supplied pretrained embeddings rather than loaded weights. c , The eight tasks whose default solution used no pretrained model and whose solution under an encouraging prompt (with a local zoo of 99 models) adopted one: change in score. Dashed lines are means; the grey band is ± 1 s.d. of the same differences on the 42 tasks with no pretrained model in either arm. Bars are coloured by domain as in Fig. 1 .
Figure 5: Integrity defects in pre-release builds and their exploitation by agents. a , Confirmed defects per audit round, by severity ( n=168 defects in 62 of 64 tasks); the 14 serious defects of round 3 are the auditor’s nine confirmed claims of c , two further tasks that share a confirmed grader defect, one found by direct review of the task the auditor declined and two found while fixing. b , Defects by category, coloured by who found them: the initial review, the systematic scans, the adversarial AI auditor or an anomalous agent score or submission. c , Outcome of human verification for the 34 claims the AI auditor rated serious. d , Agent scores with the defect (filled) and after the fix (open) for the 13 clear or partial exploitation cases with a comparable rerun (the score after a fix is that on the task version of that round, not on the final release); Cx, Codex CLI (A, GPT-6 Astra; S, GPT-5.6 Sol); CC, Claude Code (O5.5, Opus 5.5; O5, Opus 5). e , Score of the same configuration on the first version and on the final release for every task with a valid run on both; filled points are tasks whose defects were fixed without changing the metric or scale ( n=33 per configuration), open points the rest; the title gives the mean change for Codex CLI with GPT-5.6 Sol ( −7.1 ; −3.9 for Codex CLI with GPT-6 Astra on the same 33 tasks). These shifts combine the removal of defects with other hardening of the rebuilt tasks. Scores in d and e are ×100 .
“Type” is the format of the predicted output, not the metric (Supplementary Table 6 gives the metric); abbreviations: Regr., regression; Segm., segmentation; Gener., generation. “Train/Test” gives the packaged prediction units (rows of train.csv / test.csv , or instances counted from the data files for index-style tables; ‘–’ where the training set has no separate instance count). “Split” records the held-out unit as stated in the task description; ∘ marks an in-distribution split. “Public size” is the on-disk size of the task’s public directory. “Published method” is the primary reproduced published method (tier 1, published for this dataset; tier 2, strongest published method for the problem class) and its score on the task ( ×100 ). Sources are given as in the task descriptions; full citations and licences are listed in Supplementary Table 2.
Task
Type
Metric
Train/Test
Split
Size
Source
Published method (tier, score)
Chemical biology (8 tasks)
boom-csd
Regr.
mean of clip(r)+calibrated RMSE per source x tail
15,925 / 2,000
property-tail OOD (extrapolation)
1.52M
BOOM (Antoniuk et al. 2025) density/HOF
boom_egnn (1, 77.3)
chemixhub-conductivity
Regr.
residual clip(r)+calibration vs reference per source
MoleculeNet via KANO (Fang 2023); PubChem subset; ElementKG
kano (1, 68.6)
Supplementary Table 1: Complete task catalogue of BioXArena-Pro with data statistics, split construction and reproduced published-method references.
One line per source; tasks built from several sources list them in order. “Licence” is the licence or terms of use stated on the primary release page of each source (data portal, Zenodo, Figshare, Dataverse, Hugging Face dataset card or repository LICENSE file), checked on 2026-11-09; where only a software licence is stated we say so. † marks a licence we could not verify (no licence stated by the source, or the primary page could not be read); ‡ marks a conflict between the task description shipped with the data and the primary release (wrong citation, different licence, or a record that belongs to another study), with the primary release given here. “NC” flags non-commercial terms. Full citation strings, the licence evidence and the redistribution notes are in task_sources.csv in the code repository.
Task
Source dataset(s)
Ref.
Licence
Access
Chemical biology (8 tasks)
boom-csd
BOOM (CSD density / heat of formation)
1
Apache-2.0 (repository; no separate data licence) ‡
https://github.com/FLASK-LLNL/BOOM
chemixhub-conductivity
CheMixHub
2
not stated (code MIT; data licence not stated) †
https://github.com/chemcognition-lab/chemixhub
Polymer-electrolyte conductivity compilation (Bradford et al. 2023)
NIST Standard Reference Data (SRD 147); (c) U.S. Secretary of Commerce on behalf of the United States of America, all rights reserved
https://ilthermo.boulder.nist.gov/
Supplementary Table 2: Primary data sources, references and licences of the 64 BioXArena-Pro tasks.
“Sources” counts the distinct input sources an agent receives for a task, under the definition used in the BioXArena paper (each table column group, file or asset that carries independent information counts once; identifiers and the target do not). “Modality families” lists the families those sources belong to (molecule, sequence, structure, image, table, gene expression, protein abundance, other omics, spatial coordinates, text, time series). “Multi-modal” is the judgement under the same rubric: yes, at least two families carrying independent information; borderline, two families of which one is weakly informative; no, one family. Annotated from the public task descriptions and file headers on 2026-10-22 with the BioXArena judgements as calibration; the per-task rationale is in source_modality.csv in the code repository.
Task
Sources
Input sources
Modality families
Multi-modal
Chemical biology (8 tasks)
boom-csd
2
SMILES molecular structure ; queried property (density or heat of formation)
1: molecule
no
chemixhub-conductivity
6
component SMILES list ; component mole fractions ; temperature and pressure conditions ; component molecular weights ; conditions-only reference prediction ; electrolyte class (ionic liquid vs polymer electrolyte)
2: molecule; table
yes
drugcomb-multiomics-synergy
11
drug A SMILES ; drug B SMILES ; drug protein-target annotations ; drug clinical phase ; cell-line and tissue metadata ; replicate count ; additive reference prediction (reference_zip) ; single-agent monotherapy response table ; cell-line transcript expression ; cell-line SWATH proteomics ; cell-line miRNA profile
5: molecule; gene expression; protein abundance; other omics: miRNA; table
Supplementary Table 3: Per-task input sources and modality families of BioXArena-Pro.
Scaffold
Backbone
Mean
95% CI
OK / fail
Time (h)
In (M)
Out (M)
Codex CLI
GPT-6 Astra
55.1
50.0–59.8
64 / 0
31.9
262.8
2.26
Claude Code
Claude Opus 5.5
53.2
48.3–58.0
64 / 0
30.5
45.8
1.20
OpenCode
GPT-6 Astra
53.2
48.2–58.0
64 / 0
27.2
224.7
1.57
OpenCode
Claude Opus 5.5
53.2
48.1–58.0
64 / 0
33.3
60.7
1.31
Claude Code
Claude Opus 5
52.9
48.0–57.8
64 / 0
51.0
116.5
3.01
MLEvolve
Claude Opus 5.5†
52.7
47.7–57.6
64 / 0
127.0
272.4
30.41
Supplementary Table 4: The 22 agent configurations: score, failures, agent time and tokens. Mean, mean normalized score over the 64 tasks ( ×100 ), failures scored 0; CI, 95% percentile bootstrap over tasks (5,000 resamples). Time, agent wall-clock time summed over the 64 tasks; In and Out, input and output tokens in millions (input includes cached prompt tokens as reported by each API). †, two task scores taken from MLEvolve with Opus 5 because the API served empty completions (Methods). In Claude Code with Opus 5.5, two tasks were completed by Opus 5 through the CLI’s refusal fallback (Methods). Repeat runs: the 64-task means of the three runs are 55.1, 54.1 and 54.4 for Codex CLI with GPT-6 Astra and 53.2, 53.6 and 53.8 for Claude Code with Opus 5.5; the table lists the first run. Three further leaderboard configurations (Codex CLI with GPT-6.1 Sol, 34.8; the minimal scaffold with DeepSeek V4 Pro, 33.7; and the minimal scaffold with Gemini 3.1 Pro, 33.5) fit neither design and are not analysed in the paper.
Comparison
Mean diff.
Higher
Tied
Lower
P
PHolm
Codex CLI vs OpenCode, GPT-6 Astra
+1.88
51
0
13
< 0.0001
< 0.0001
Claude Code vs OpenCode, Opus 5.5
+0.04
34
0
30
0.2326
0.6978
Gemini CLI vs OpenCode, Gemini 3.8 Flash
-1.06
37
1
26
0.4017
0.8033
OpenCode vs Minimal, GPT-6 Astra
+6.70
52
0
12
< 0.0001
< 0.0001
OpenCode vs Minimal, Opus 5.5
+6.68
51
0
13
< 0.0001
< 0.0001
OpenCode vs Minimal, Gemini 3.8 Flash
+10.32
53
1
10
< 0.0001
< 0.0001
Supplementary Table 5: Paired comparisons between configurations. Mean difference in normalized score ( ×100 ) over the 64 tasks, number of tasks on which the first configuration is higher, tied or lower, and the two-sided Wilcoxon signed-rank P value, raw and Holm-corrected over the 15 comparisons. A configuration can be higher on most tasks and still have a lower mean, because failed runs score 0.
Definitions follow the grading code of the release. Every task score is finally clipped to [0,1] and reported ×100 ; an invalid submission (schema, identifier or value errors) scores 0 for the task, and 100 is the formal maximum of every metric (no noise ceiling was measured, so it need not be reachable in practice). “Chance” is the score of an uninformed submission on the 0–100 scale, measured with the released graders (constant or reference prediction for regression and ranking scores; uniformly random labels, random rankings or random scores otherwise; mean of 3–5 seeds). The tasks fall into two families. (i) Chance-zero metrics (36 tasks): correlations reported as clip(r,0,1) , calibrated regression scores, R2 , average precision or F1 expressed as a gain over label prevalence, skill scores and scores offset against a public reference prediction, together with exact-match and segmentation scores that have no chance agreement. (ii) Metrics kept on their native scale (28 tasks): ROC-AUC and the concordance index (chance 50), and macro-F1, accuracy, average precision, reciprocal rank, Fmax, sequence recovery and sensitivity at fixed specificity, whose chance level depends on the number of classes or candidates and is listed per task. r , Pearson correlation; ρ , Spearman correlation; AP, average precision; p , positive prevalence in the test set; SD, standard deviation; n.d., not defined. a An input-independent training term-frequency prior scores 39.6 on this task. b Exact counts have no natural chance level; the all-zero template scores 0. c The training climatology scores 0; a uniform forecast scores 14.8.
Task
Primary metric and aggregation
Normalization to 0–100
Chance
Chemical biology (8 tasks)
boom-csd
Calibrated regression score: 0.5 × clip(Pearson r,0,1) + 0.5 × clip(1 - RMSE/SD(y_true),0,1). Aggregation: computed in each of 4 cells (source density/hof × low/high extrapolation tail); equal-weight mean of the 4 cells.
Components clipped inside the grader
0
chemixhub-conductivity
Calibrated regression score on residuals after subtracting the public conditions-only reference (test.csv reference_ln_conductivity) from truth and prediction: 0.5 × clip(Pearson r,0,1) + 0.5 × clip(1 - RMSE/SD(y_true - ref),0,1). Aggregation: computed within each of the 2 sources; equal-weight mean.
Offset against reference prediction, components clipped inside the grader
0
drugcomb-multiomics-synergy
Partial Pearson correlation: truth and prediction projected onto the orthogonal complement of span{cell-line indicators, public additive reference_zip}, then Pearson r of the projections. Aggregation: computed within each of 9 tissues ( ≥ 20 rows); mean over tissues.
Clip(mean r, 0, 1) in the grader (and again by the evaluator)
0
kano-molecular-property-prediction
Classification instances: average precision per scored target minus prevalence, divided by (1 - prevalence); regression instances: R 2 = 1 - MSE/Var(y) per target. Aggregation: per-target values clipped to [0,1], mean over scored targets of an instance, then equal-weight mean over 12 instances.
Gain over prevalence (classification) / R 2 (regression), each clipped to [0,1] per target
0
Supplementary Table 6: Primary metric, normalization and chance level of each task.
All scores ×100 . Tier 1, method published for the task’s dataset; tier 2, strongest published method of the same problem class. Group, comparability of our test set with the original study: close (same data and metric family), rebuilt (new split, removed inputs or new metric) or mixed (the grade differs between instances or applies to part of the task). The nine tier-1 tasks in the close group ( ∗ ) form the published-method baseline. Published, the primary reproduced published method (named in Supplementary Table 1); Codex, Codex CLI with GPT-6 Astra; Claude, Claude Code with Opus 5.5. Best, the highest score among the 22 configurations and the configuration that reached it. The scores of all 22 configurations on every task are in the leaderboard files of the code repository.
Task
Tier
Group
Published
Codex
Claude
Best
Best configuration
Chemical biology (8 tasks)
boom-csd
1
mixed
77.3
86.8
83.8
88.0
Claude Code + Opus 5
chemixhub-conductivity
1
rebuilt
54.8
56.7
56.8
57.8
MLEvolve + Gemini 3.8 Flash
drugcomb-multiomics-synergy
2
rebuilt
28.5
47.6
45.7
48.6
MLEvolve + Opus 5.5
kano-molecular-property-prediction
1
rebuilt
68.6
69.2
69.7
69.7
Claude Code + Opus 5.5
Supplementary Table 7: Per-task scores of the reproduced published-method reference and of the agents.
Reproducing and comparing deep research agents today is hard: the same backbone evaluated on the same benchmark can report different accuracies across papers because the harness and tool registry differ, and integrating a new model into a comparable evaluation surface costs weeks of model-specific engineering. These are symptoms of a broader reproducibility problem in deep research agent research. Here, we introduce BioMedArena, an open-source toolkit that addresses this reproducibility gap and provides an arena for comparing deep research agents under a shared evaluation environment. BioMedArena decouples six layers of biomedical agent evaluation -- benchmark loading, tool exposure, tool selection, harness mode, context management, and scoring -- and exposes 166 biomedical benchmarks and 75 biomedical tools across 9 functional families. Adding a new model, benchmark, or tool can be accomplished with a few-line provider adapter. Beyond evaluation infrastructure, BioMedArena ships a library of high-quality reference components: 6 agent harnesses (including our proposed Mutual-Evolve) and 6 context-management strategies, any of which can be equipped on any backbone. Equipping these components substantially improves all 12 backbones; on each of 8 representative biomedical benchmarks, the best equipped backbone surpasses prior state-of-the-art (SOTA), by 15.01 percentage points on average. The toolkit, configurations, and per-task traces are available at https://github.com/AI-in-Health/BioMedArena.
Jinge Wu, Hongjian Zhou, Mingde Zeng +8
University of Oxford · University College London · Technical University of Munich +1
Autonomous agents are increasingly capable of developing machine learning (ML) models end to end. In medical practice, however, strong performance on the target metric does not fully characterize how good the models are in real-world use cases. For example, medical ML models should also provide well-calibrated risk estimates and remain robust across patient populations. Yet autonomous model development is typically guided by task objectives that emphasize simple predictive discrimination (e.g. AUROC). This raises a key question: are these objectives sufficient to guide agents toward models that meet the broader requirements of real-world medical use? In this study, we introduce Auto-MD-bench, a benchmark of 15 real-world medical tasks from 10 clinical datasets, spanning survival analysis, classification and sequential prediction, with external validation across cohorts, hospitals and time periods. We evaluate four agents: AIDE, AI Scientist v2, Codex, and Autoresearch. The strongest agents achieve competitive primary scores, yet their predicted risks are less calibrated than those of the simple baseline models, which could lead to severe medical accidents. Moreover, models with nearly identical primary scores can exhibit markedly different calibration, while performance varies substantially across population shifts. These findings show that primary task performance alone is insufficient to assess autonomous clinical modeling and multiple different metrics widely used in healthcare practice need to be taken into consideration.
Shenqi Zhou, Qiran Zou, Wenhao Zhao +2
National University of Singapore · Zhejiang University · Amazon
We evaluate whether AI agents can match the reported findings of published biomedical studies using public data. Existing evaluations do not consistently separate analysis from prior knowledge or retrieval of the published answer. We introduce BioStudyBench, a benchmark of 25 long-horizon analysis tasks drawn from studies first published between July and September 2026, after the developer-reported knowledge cutoffs of the models we evaluate, semi-automatically filtered down from 404,019 PubMed records. In each task, the agent receives a neutral research question but no data files, so it must find and download the relevant public data, search the literature through tools that return only records dated before its cutoff, and report findings through data analysis. To measure gains over prior knowledge, we run every task both with and without access to data and tools. Across eight models, access to data and tools raises the pass rate by 47 percentage points on average over the no-data baseline. Open-weight models across sizes trail closed-weight models, with the best open-weight model passing 81.3% of tasks against 94.7% for the best closed-weight model.