BioXArena-Pro: a research-grade benchmark for autonomous biomedical machine learning agents
Organizations: Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates. · The Westminster Schools, Atlanta, Georgia, United States. · Carnegie Mellon University, Pittsburgh, Pennsylvania, United States.
Abstract
Coding agents are increasingly proposed as autonomous builders of biomedical models, but existing benchmarks are small, near saturation or open to leaked information. We present BioXArena-Pro, 64 research-level modelling tasks from eight biomedical domains, each rebuilt from a published study; most hold out whole biological units such as species or cohorts. Across 22 agent configurations, the best scores 55.1 of 100, and the scaffold matters almost as much as the model. We re-ran a published method for every task: where the published recipe applies unchanged, the best agent and the methods tie; on rebuilt tasks the methods trail by 8.2 points, suggesting that agents gain by adapting to new data. Pretrained models (including biological foundation models) matter where the strongest agent uses them: forbidding them decreases its score on those tasks by 9 points. Every task was audited in several AI-assisted rounds and a final human review.
Figures & tables
| “Type” is the format of the predicted output, not the metric (Supplementary Table 6 gives the metric); abbreviations: Regr., regression; Segm., segmentation; Gener., generation. “Train/Test” gives the packaged prediction units (rows of train.csv / test.csv , or instances counted from the data files for index-style tables; ‘–’ where the training set has no separate instance count). “Split” records the held-out unit as stated in the task description; marks an in-distribution split. “Public size” is the on-disk size of the task’s public directory. “Published method” is the primary reproduced published method (tier 1, published for this dataset; tier 2, strongest published method for the problem class) and its score on the task ( ). Sources are given as in the task descriptions; full citations and licences are listed in Supplementary Table 2. | |||||||
|---|---|---|---|---|---|---|---|
| Task | Type | Metric | Train/Test | Split | Size | Source | Published method (tier, score) |
| Chemical biology (8 tasks) | |||||||
| boom-csd | Regr. | mean of clip(r)+calibrated RMSE per source x tail | 15,925 / 2,000 | property-tail OOD (extrapolation) | 1.52M | BOOM (Antoniuk et al. 2025) density/HOF | boom_egnn (1, 77.3) |
| chemixhub-conductivity | Regr. | residual clip(r)+calibration vs reference per source | 39,089 / 4,958 | group-disjoint (component set) | 8.13M | CheMixHub 2025; ionic-liquid DB; polymer-electrolyte compilation 2023 | chemeleon_xgboost (1, 54.8) |
| drugcomb-multiomics-synergy | Regr. | projected Pearson per tissue mean | 176,729 / 93,964 | cold-drug | 243M | DrugComb v1.4 2021; NCI-ALMANAC 2017; CellMiner | deepsynergy (2, 28.5) |
| kano-molecular-property-prediction | Multi-label | AUPRC gain / R2 per target per instance | 183,881 / 22,996 | unspecified (KANO splits) | 245M | MoleculeNet via KANO (Fang 2023); PubChem subset; ElementKG | kano (1, 68.6) |
| One line per source; tasks built from several sources list them in order. “Licence” is the licence or terms of use stated on the primary release page of each source (data portal, Zenodo, Figshare, Dataverse, Hugging Face dataset card or repository LICENSE file), checked on 2026-11-09; where only a software licence is stated we say so. † marks a licence we could not verify (no licence stated by the source, or the primary page could not be read); ‡ marks a conflict between the task description shipped with the data and the primary release (wrong citation, different licence, or a record that belongs to another study), with the primary release given here. “NC” flags non-commercial terms. Full citation strings, the licence evidence and the redistribution notes are in task_sources.csv in the code repository. | ||||
|---|---|---|---|---|
| Task | Source dataset(s) | Ref. | Licence | Access |
| Chemical biology (8 tasks) | ||||
| boom-csd | BOOM (CSD density / heat of formation) | 1 | Apache-2.0 (repository; no separate data licence) ‡ | https://github.com/FLASK-LLNL/BOOM |
| chemixhub-conductivity | CheMixHub | 2 | not stated (code MIT; data licence not stated) † | https://github.com/chemcognition-lab/chemixhub |
| Polymer-electrolyte conductivity compilation (Bradford et al. 2023) | 3 | MIT (data and code) | https://github.com/learningmatter-mit/Chem-prop-pred | |
| NIST ILThermo (SRD 147) ionic-liquid conductivity | 4 | NIST Standard Reference Data (SRD 147); (c) U.S. Secretary of Commerce on behalf of the United States of America, all rights reserved | https://ilthermo.boulder.nist.gov/ | |
| “Sources” counts the distinct input sources an agent receives for a task, under the definition used in the BioXArena paper (each table column group, file or asset that carries independent information counts once; identifiers and the target do not). “Modality families” lists the families those sources belong to (molecule, sequence, structure, image, table, gene expression, protein abundance, other omics, spatial coordinates, text, time series). “Multi-modal” is the judgement under the same rubric: yes, at least two families carrying independent information; borderline, two families of which one is weakly informative; no, one family. Annotated from the public task descriptions and file headers on 2026-10-22 with the BioXArena judgements as calibration; the per-task rationale is in source_modality.csv in the code repository. | ||||
|---|---|---|---|---|
| Task | Sources | Input sources | Modality families | Multi-modal |
| Chemical biology (8 tasks) | ||||
| boom-csd | 2 | SMILES molecular structure ; queried property (density or heat of formation) | 1: molecule | no |
| chemixhub-conductivity | 6 | component SMILES list ; component mole fractions ; temperature and pressure conditions ; component molecular weights ; conditions-only reference prediction ; electrolyte class (ionic liquid vs polymer electrolyte) | 2: molecule; table | yes |
| drugcomb-multiomics-synergy | 11 | drug A SMILES ; drug B SMILES ; drug protein-target annotations ; drug clinical phase ; cell-line and tissue metadata ; replicate count ; additive reference prediction (reference_zip) ; single-agent monotherapy response table ; cell-line transcript expression ; cell-line SWATH proteomics ; cell-line miRNA profile | 5: molecule; gene expression; protein abundance; other omics: miRNA; table | yes |
| kano-molecular-property-prediction | 2 | SMILES molecular structure ; element-oriented knowledge graph and pre-computed element/functional-group embeddings | 1: molecule | no |
| Scaffold | Backbone | Mean | 95% CI | OK / fail | Time (h) | In (M) | Out (M) |
|---|---|---|---|---|---|---|---|
| Codex CLI | GPT-6 Astra | 55.1 | 50.0–59.8 | 64 / 0 | 31.9 | 262.8 | 2.26 |
| Claude Code | Claude Opus 5.5 | 53.2 | 48.3–58.0 | 64 / 0 | 30.5 | 45.8 | 1.20 |
| OpenCode | GPT-6 Astra | 53.2 | 48.2–58.0 | 64 / 0 | 27.2 | 224.7 | 1.57 |
| OpenCode | Claude Opus 5.5 | 53.2 | 48.1–58.0 | 64 / 0 | 33.3 | 60.7 | 1.31 |
| Claude Code | Claude Opus 5 | 52.9 | 48.0–57.8 | 64 / 0 | 51.0 | 116.5 | 3.01 |
| MLEvolve | Claude Opus 5.5† | 52.7 | 47.7–57.6 | 64 / 0 | 127.0 | 272.4 | 30.41 |
| Comparison | Mean diff. | Higher | Tied | Lower | ||
|---|---|---|---|---|---|---|
| Codex CLI vs OpenCode, GPT-6 Astra | +1.88 | 51 | 0 | 13 | 0.0001 | 0.0001 |
| Claude Code vs OpenCode, Opus 5.5 | +0.04 | 34 | 0 | 30 | 0.2326 | 0.6978 |
| Gemini CLI vs OpenCode, Gemini 3.8 Flash | -1.06 | 37 | 1 | 26 | 0.4017 | 0.8033 |
| OpenCode vs Minimal, GPT-6 Astra | +6.70 | 52 | 0 | 12 | 0.0001 | 0.0001 |
| OpenCode vs Minimal, Opus 5.5 | +6.68 | 51 | 0 | 13 | 0.0001 | 0.0001 |
| OpenCode vs Minimal, Gemini 3.8 Flash | +10.32 | 53 | 1 | 10 | 0.0001 | 0.0001 |
| Definitions follow the grading code of the release. Every task score is finally clipped to and reported ; an invalid submission (schema, identifier or value errors) scores 0 for the task, and 100 is the formal maximum of every metric (no noise ceiling was measured, so it need not be reachable in practice). “Chance” is the score of an uninformed submission on the 0–100 scale, measured with the released graders (constant or reference prediction for regression and ranking scores; uniformly random labels, random rankings or random scores otherwise; mean of 3–5 seeds). The tasks fall into two families. (i) Chance-zero metrics (36 tasks): correlations reported as , calibrated regression scores, , average precision or F1 expressed as a gain over label prevalence, skill scores and scores offset against a public reference prediction, together with exact-match and segmentation scores that have no chance agreement. (ii) Metrics kept on their native scale (28 tasks): ROC-AUC and the concordance index (chance 50), and macro-F1, accuracy, average precision, reciprocal rank, Fmax, sequence recovery and sensitivity at fixed specificity, whose chance level depends on the number of classes or candidates and is listed per task. , Pearson correlation; , Spearman correlation; AP, average precision; , positive prevalence in the test set; SD, standard deviation; n.d., not defined. a An input-independent training term-frequency prior scores 39.6 on this task. b Exact counts have no natural chance level; the all-zero template scores 0. c The training climatology scores 0; a uniform forecast scores 14.8. | |||
|---|---|---|---|
| Task | Primary metric and aggregation | Normalization to 0–100 | Chance |
| Chemical biology (8 tasks) | |||
| boom-csd | Calibrated regression score: 0.5 clip(Pearson r,0,1) + 0.5 clip(1 - RMSE/SD(y_true),0,1). Aggregation: computed in each of 4 cells (source density/hof low/high extrapolation tail); equal-weight mean of the 4 cells. | Components clipped inside the grader | 0 |
| chemixhub-conductivity | Calibrated regression score on residuals after subtracting the public conditions-only reference (test.csv reference_ln_conductivity) from truth and prediction: 0.5 clip(Pearson r,0,1) + 0.5 clip(1 - RMSE/SD(y_true - ref),0,1). Aggregation: computed within each of the 2 sources; equal-weight mean. | Offset against reference prediction, components clipped inside the grader | 0 |
| drugcomb-multiomics-synergy | Partial Pearson correlation: truth and prediction projected onto the orthogonal complement of span{cell-line indicators, public additive reference_zip}, then Pearson r of the projections. Aggregation: computed within each of 9 tissues ( 20 rows); mean over tissues. | Clip(mean r, 0, 1) in the grader (and again by the evaluator) | 0 |
| kano-molecular-property-prediction | Classification instances: average precision per scored target minus prevalence, divided by (1 - prevalence); regression instances: R 2 = 1 - MSE/Var(y) per target. Aggregation: per-target values clipped to [0,1], mean over scored targets of an instance, then equal-weight mean over 12 instances. | Gain over prevalence (classification) / R 2 (regression), each clipped to [0,1] per target | 0 |
| All scores . Tier 1, method published for the task’s dataset; tier 2, strongest published method of the same problem class. Group, comparability of our test set with the original study: close (same data and metric family), rebuilt (new split, removed inputs or new metric) or mixed (the grade differs between instances or applies to part of the task). The nine tier-1 tasks in the close group ( ) form the published-method baseline. Published, the primary reproduced published method (named in Supplementary Table 1); Codex, Codex CLI with GPT-6 Astra; Claude, Claude Code with Opus 5.5. Best, the highest score among the 22 configurations and the configuration that reached it. The scores of all 22 configurations on every task are in the leaderboard files of the code repository. | |||||||
|---|---|---|---|---|---|---|---|
| Task | Tier | Group | Published | Codex | Claude | Best | Best configuration |
| Chemical biology (8 tasks) | |||||||
| boom-csd | 1 | mixed | 77.3 | 86.8 | 83.8 | 88.0 | Claude Code + Opus 5 |
| chemixhub-conductivity | 1 | rebuilt | 54.8 | 56.7 | 56.8 | 57.8 | MLEvolve + Gemini 3.8 Flash |
| drugcomb-multiomics-synergy | 2 | rebuilt | 28.5 | 47.6 | 45.7 | 48.6 | MLEvolve + Opus 5.5 |
| kano-molecular-property-prediction | 1 | rebuilt | 68.6 | 69.2 | 69.7 | 69.7 | Claude Code + Opus 5.5 |