Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment
Authors: Yiqun T. Chen, Jan Matthias, Jianing Yao, Hongyu Eric Zhao, Stephanie C. Hicks
Organizations: Department of Biostatistics, Johns Hopkins University, Baltimore, MD, USA · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA · Department of Biomedical Informatics and Data Science, Johns Hopkins University, Baltimore MD, USA · Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA
AI agents are increasingly automating data analysis, but responsible adoption requires reliable performance evaluation, which is costly in specialized scientific domains. We introduce a framework that translates scientific publications into auditable benchmarks for AI agents and demonstrate its use on spatial transcriptomics alignment tasks. Across 121 tasks, 16 baseline methods, and six AI agents, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks, often using surprisingly simple heuristics. Agent performance was robust to prompting and access to pre-installed packages. Benchmark construction also uncovered errors in prior work and shortcuts in current evaluation metrics. Finally, we show that agent-generated solutions can be distilled into a reusable program that ranks in the top 10% of baseline methods on unseen data. Our framework offers a scalable way to evaluate AI agents in specialized scientific domains by leveraging and stress-testing existing papers and metrics.
Figures & tables
Figure 1: Overview of the framework for translating published benchmarks into auditable tasks for AI agents. a , End-to-end workflow for aligning spatial transcriptomics datasets. Tasks are compiled from the source publication, executed in isolated containers, and audited through their execution traces to refine task definitions, evaluation metrics, and prompts. b , Mapping from the SABench study [ 11 ] to a structured task card for the human dorsolateral prefrontal cortex (DLPFC) dataset [ 12 ] . Components of the publication are mapped to the corresponding fields in the task specification, which undergoes automated validation prior to agent execution. c , Representative tasks from four of the five task families and the total number of tasks per family. Cells are colored by cell type or anatomical region (both withheld from the agents). d , Example task auditing on three-dimensional reconstruction of a mouse hemibrain across 129 MERFISH sections [ 13 , 14 ] , showing unaligned input sections, ground-truth coordinates, the top-performing published baseline (SPACEL), and the best agent solution ( gpt-6-astra ).
Figure 2: Comparison of AI agents with published spatial alignment methods under a unified evaluation harness. a , Coordinate error (mean absolute error, MAE; log scale) of the best basic-prompt agent run versus the best published baseline across 28 tasks with ground-truth coordinates. The dashed line indicates parity. b , Expression agreement (mean of gene-wise Pearson correlation, cosine similarity, and structural similarity) of the best agent run versus the best published method across 118 tasks, distinguishing cross-platform pairs (black) from other task families (grey). c , Percentage of tasks on which a single basic-prompt run of each model matched or outperformed the best published baseline, stratified by task family for expression agreement (left) and coordinate error (right). d , Mean percentage of the 16 reference baselines (13 published packages at their optimal configuration and 3 geometric baselines) outperformed per task versus median runtime (log scale) for published packages (black), geometric baselines (open circles), and basic-prompt agent runs (colored circles; shaded region). e , Representative alignments across four tasks comparing unaligned inputs, ground-truth coordinates, the best published baseline, and the best agent solution, with MAE reported below each panel. Top, tasks where agents outperformed baselines; bottom, tasks where published baselines performed better.
Figure 3: Impact of domain-specific context on agent tool use, alignment accuracy, and computational cost. a , Percentage of scored runs that executed each published alignment package during exploration and retained it in the final submitted code under the basic and rich-context prompts. b , Change in within-task percentile under the rich-context prompt relative to the basic prompt across 121 tasks. Points denote individual tasks; horizontal bars indicate the mean and 95% cluster-bootstrap confidence interval across source datasets. c , Mean within-task percentile versus median API cost per run (US$, log scale) under the basic prompt (open circles) and rich-context prompt (filled circles). Error bars denote 95% bootstrap confidence intervals. d , Algorithmic composition of final submitted code under each prompting condition, categorized by custom geometric transformation class or published package output. e , Difference in the proportion of runs incorporating specific algorithmic steps in the final submission (rich-context minus basic prompt, in percentage points). f , Mean within-task percentile of rich-context runs whose final submission relied on custom code (open circles) versus a published package (filled circles). Error bars denote 95% bootstrap confidence intervals.
Figure 4: Concordance of proxy alignment metrics with ground-truth coordinate error. a , Taxonomy of spatial alignment evaluation metrics, their required supervision, and coverage across the 121 benchmark tasks. MAE, mean absolute error; MNN, mutual nearest neighbors; SSIM, structural similarity; MI, mutual information. b , Within-task Spearman rank correlation between each proxy metric and ground-truth coordinate error across 32 tasks with known positions (signed so that positive values denote agreement with spatial accuracy). Points represent individual tasks (real tissue, black; simulated tissue, grey); vertical bars and right-hand labels indicate median correlation and task count. c , Median within-task Spearman correlation across all pairs of evaluation metrics. d – f , Comparison of expression agreement ( d , e ) and MNN label agreement ( f ) against coordinate error (log scale) across agent runs, published methods, and geometric baselines on overlapping mouse brain sections ( d , with inset magnifying MAE <30 ) and simulated Visium spot data ( e , f ).
Figure 5: Execution dynamics, algorithmic building blocks, and metric-dependent step effects across coding agents. a , Mean share of tool calls allocated to each operational phase across models. b , Frequency of key workflow behaviors across models, including visual inspection of generated plots and iterative submission refinement ( ≥3 rewrites). c , Prevalence of core algorithmic steps in the final submitted code of basic-prompt runs across models. d , Prevalence of algorithmic steps in the top-scoring run of each task versus all other runs. e , Model- and prompt-adjusted association of each step in the final code with within-task percentile across all tasks (primary metric, left) and across the 32 tasks with ground-truth coordinates (MAE, right). Bars and whiskers show point estimates and 95% bootstrap confidence intervals; solid bars indicate confidence intervals excluding zero. f , Example winning strategy from gpt-6-astra combining an affine search on smoothed shared-gene maps with a penalized cubic B-spline warp, 25% held-out gene cross-validation, and a positive-Jacobian fold check.
Figure 6: Distillation of agent runs into a standalone alignment program evaluated on held-out tasks. a , Five-fold cross-validation design under random and task-family splits. An isolated synthesis agent (Claude Code with claude-opus-5.5 ) inspects the final scripts and training scores of claude-opus-5.5 and gpt-6-astra on four folds and writes a standalone Python program, which executes without LLM calls on the held-out fold. b , Mean percentage of the 16 reference baselines outperformed on held-out tasks, by task family, for distilled programs, individual basic-prompt agent runs, published packages, and geometric baselines (dashed lines mark the median package; n denotes task count).
Family
Tasks
Source
Content
Serial sections (sa-0xx)
35
SABench [ 11 ]
DLPFC Visium [ 12 ] , BCA, MBSA and MBSP, AMBC Xenium [ 39 ] , MHPR MERFISH [ 62 ] , SCC [ 63 ] , BARISTA-seq [ 64 ] , STARmap [ 65 ] , fly embryo (16 sections) [ 41 ] , MOp MERFISH (31 and 33 sections) [ 66 ] , Open-ST lymph node (19 sections) [ 59 ] , WMB MERFISH (129 sections and windows of 5 and 10) [ 13 ] , MOB Stereo-seq [ 67 ] and Slide-seqV2 [ 68 ] , and four rotation and crop sweeps
Motor cortex series (sa-1xx)
12
SABench [ 11 ]
MOp MERFISH, one task per sample of two mice, 4 to 6 sections each
Cross-platform pairs (sa-2xx)
45
SABench [ 11 ]
All pairs of ten platforms on adult mouse brain sections: ST [ 37 ] , Visium, Visium HD [ 69 ] , Xenium, Xenium 5K, MERFISH [ 38 ] , STARmap PLUS [ 70 ] , Slide-seq, Stereo-seq and CosMx [ 71 ]
Simulated tissue (sa-3xx)
12
Albis simulator [ 57 ]
Simulated serial sections with known positions (Supplementary Table 3 )
Rotation, crop, 3D (sa-4xx)
17
GEASO data [ 14 ] , WMB MERFISH
STARmap PLUS depth, overlap and stitching [ 70 ] ; STARmap and RIBOmap cross-modality [ 60 ] ; cross-animal and rotation sweep [ 61 ] ; spatial CUT&Tag-RNA [ 40 ] ; 129-section MERFISH hemibrain; Open-ST reconstruction; whole-brain MERFISH from raw coordinates (129, 33, 17 and 33 irregular sections)
fly embryo, human DLPFC, human breast cancer, human lymph node, human squamous cell carcinoma, mouse brain, mouse cortex, mouse hypothalamus, mouse motor cortex, mouse olfactory bulb, whole mouse brain
Supplementary Table 2 : Tissues, technologies and available scores per family. For each family, the tissues and technologies it covers and how many tasks can be scored with true positions, with labels but no true positions, or with expression agreement only. The serial-sections family also holds some cross-platform pairs and three-dimensional stacks from SABench.
Task
Regime
Key parameters
sa-301
strong mixing, cells
24,207 cells, R=2,050 µm, σ=1.5
sa-302
Xenium-like cells
600,000 cells, R=6,000 µm, σ=1.5
sa-303
Visium spots
55 µm spots on a 100 µm lattice, σ=0.3
sa-304, sa-305
Visium HD bins
16 µm ( σ=0.7 ) and 8 µm ( σ=0.8 )
sa-306, sa-307
as sa-303 and sa-304, lattice fixed to the slide
removes the lattice leak
sa-308
cells with gaps
30 slabs of 137 µm, 9 kept at irregular spacing
Supplementary Table 3 : Simulated tasks. R , tissue radius; σ , standard deviation of the batch effect.
Supplementary Figure 1 : Evaluation of published alignment packages and geometric baselines under a unified scorer. a , Task coverage across families and number of parameter configurations evaluated per method. b , Mean within-task percentile across the 16 reference baselines on the primary metric, stratified by task family. c , Number of tasks on which each published package achieved the highest baseline score, or on which a geometric baseline matched or outperformed all packages. d , Mean within-task percentile stratified by individual evaluation metric. MNN, mutual nearest neighbors; SSIM, structural similarity; MI, mutual information. e , Percentage of tasks on which the best basic-prompt agent run matched or outperformed the best published configuration across individual metrics. PCA, principal component analysis; ICP, iterative closest point.
Agent software
Model
Effort
Launch (abridged)
Claude Code 2.1.281
claude-opus-5.5
high
claude --print --output-format stream-json --model <m> --dangerously-skip-permissions -p "<prompt>"
Supplementary Table 4 : Agent harnesses, models, and launch configurations. Software versions correspond to the container environment; launch commands are abridged.
Supplementary Figure 2 : Container isolation layers and automated leakage audit results. a , Isolation architecture preventing access to held-out data and scoring scripts inside the container workspace. b , Distribution of automated audit alerts across all audited runs by category and most frequently matched keywords. c , Proportion of flagged runs in the primary analysis set and within-condition percentile difference between flagged and unflagged runs. d , Summary of dataset-level shortcuts identified through trace inspection and remediation.
Leakage mode
Observed agent behavior
Copying ground truth
Traversed parent directories despite prompt restrictions, located held-out ground-truth coordinate files, and submitted them directly.
Modifying the scorer
Located the evaluation script on the shared filesystem and modified or replicated it to inflate internal validation scores.
Cross-run copying
Inspected output directories and scripts generated by prior or concurrent agent runs on the same task and reused their solutions.
Extracting bundled outputs
Located precomputed alignment outputs bundled within cloned method repositories and submitted them as newly computed results.
Supplementary Table 5 : Modes of benchmark leakage observed prior to container isolation. Behaviors identified in execution traces during initial pilot runs on a shared filesystem.
Supplementary Figure 3 : Algorithmic steps explored and retained across models and their association with alignment scores. Rows correspond to algorithmic steps grouped by pipeline stage (omitting 7 of 46 steps explored in <3% of runs). Model columns display basic-prompt runs, showing the percentage of runs that explored each step at any point (light bars) and retained it in the final code (dark bars and numerical labels); the rich-context column pools all six evaluated models. The rightmost two columns show the within-model, within-prompt difference in percentile between runs with and without each step in the final submission across all tasks (primary metric) and across the 32 coordinate-truth tasks (coordinate error), with 95% bootstrap confidence intervals and run counts. Solid bars denote confidence intervals excluding zero.
Supplementary Figure 4 : Quantitative analysis of execution trace volume, code and text embeddings, self-evaluations, and vocabulary. a , Word count of agent narration and line count of final submitted code per run across models under the basic prompt (open boxes) and rich-context prompt (filled boxes; boxes denote interquartile range, whiskers 10th–90th percentiles). b , c , UMAP projections of final submitted code (50-dimensional TF-IDF token embeddings; b ) and agent narration ( bge-small-en-v1.5 sentence embeddings; c ), colored by model (large panels) and transformation family (inset panels). d , Proportion of runs in which the agent’s final summary claimed a high-quality alignment, stratified by within-task performance tertile (bottom, middle, and top third from light to dark). e , Most distinctive narration terms per model ranked by weighted log-odds ratio.
Supplementary Figure 5 : Run-to-run reproducibility across independent replicates. a , Comparison of replicate 2 versus replicate 1 on coordinate error (MAE) and composite score across five models, with Spearman rank correlation ρ . b , Percentage of tasks matching or outperforming the best published baseline for replicate 1, replicate 2, and the best of two replicates. c , Mean within-task percentile across models estimated independently from replicate 1 and replicate 2. d , Absolute difference between replicates as a fraction of the cross-model range per task.
Supplementary Figure 6 : Variance decomposition of alignment accuracy, cost, and runtime. a , Sequential sum-of-squares decomposition across all 2,521 scored runs, showing the fraction of variance in coordinate error, composite score, API cost, and runtime explained by task, model, prompt condition, interaction terms, and replicate residual. b , Corresponding decomposition restricted to basic-prompt runs across the five models with replicates.
AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.
Tianyu Liu, Allen Xin Wang, Antonia Panescu +30
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7
AI agents are increasingly useful for biological data analysis, but existing benchmarks mostly test broad biological knowledge, executable workflows, or localized analysis steps rather than end-to-end scientific reasoning over spatial measurements. We introduce SpatialBench-Long, a benchmark for long-horizon spatial biology in which agents must recover biological claims from raw or near-raw data and calibrated experimental context without prescribed methods. SpatialBench-Long contains 24 evaluations across primary pancreatic ductal adenocarcinoma (PDAC), engineered glioblastoma organoids and in vivo tumors, Cas9 lineage-traced lung adenocarcinoma, and mouse optic nerve aging/intervention systems, spanning CosMx, Visium, Xenium, multiplexed error-robust fluorescence in situ hybridization (MERFISH), single-cell RNA sequencing (scRNA-seq), Slide-seq, Slide-tags, histology, and lineage-recording data. Candidate claims are hardened through reproduction, independent scientist review, and trajectory inspection. Final answers are graded deterministically over controlled vocabularies and symbols with companion rubrics capturing progress through key analysis chokepoints. Across the SpatialBench-Long benchmark, three model-harness pairs tie at 8/72 runs (11.1%): Gemini 3.5 Flash / Pi terminal coding harness, GPT-5.5 / Pi, and GPT-5.5 / OpenAI Codex. SpatialBench-Long tests whether agents can move beyond executing procedural analysis to deriving accurate scientific conclusions from complex spatial measurements.
Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam.
Tianyun Zhong, Wei Wang, Jialin Li +17
University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group +4