q-bio.GNNov 9, 2026

Publications as benchmarks: auditing AI agents on spatial transcriptomics alignment

Authors: Yiqun T. Chen, Jan Matthias, Jianing Yao, Hongyu Eric Zhao, Stephanie C. Hicks

Organizations: Department of Biostatistics, Johns Hopkins University, Baltimore, MD, USA · Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA · Department of Biomedical Informatics and Data Science, Johns Hopkins University, Baltimore MD, USA · Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA

Abstract

AI agents are increasingly automating data analysis, but responsible adoption requires reliable performance evaluation, which is costly in specialized scientific domains. We introduce a framework that translates scientific publications into auditable benchmarks for AI agents and demonstrate its use on spatial transcriptomics alignment tasks. Across 121 tasks, 16 baseline methods, and six AI agents, the best agents (GPT-6-Astra and Gemini-4-Argon) matched or outperformed all baselines on 80% of tasks, often using surprisingly simple heuristics. Agent performance was robust to prompting and access to pre-installed packages. Benchmark construction also uncovered errors in prior work and shortcuts in current evaluation metrics. Finally, we show that agent-generated solutions can be distilled into a reusable program that ranks in the top 10% of baseline methods on unseen data. Our framework offers a scalable way to evaluate AI agents in specialized scientific domains by leveraging and stress-testing existing papers and metrics.

Figures & tables

Explore similar work

CardsList
  1. Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

    Jun 10, 2026Tianyu Liu, Allen Xin Wang, Antonia Panescu +30AI for ScienceScientific Agent Benchmarks

  2. Verifiable Benchmarking of Long-Horizon Spatial Biology

    May 27, 2026Ian Diks, Harihara Muralidharan, Tim Proctor +1Spatial TranscriptomicsBenchmark Design

  3. SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

    Oct 23, 2026Tianyun Zhong, Wei Wang, Jialin Li +17Scientific Agent BenchmarksLLM Agent Evaluation