Organizations: Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA · Ming Hsieh Department of Electrical and Computer Engineering, Viterbi School of Engineering, University of Southern California, Los Angeles, CA, USA · Alfred E. Mann Department of Biomedical Engineering, Viterbi School of Engineering, University of Southern California, Los Angeles, CA, USA · Department of Pediatrics, University of Chicago, Chicago, IL, USA · Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA · Comprehensive Cancer Center, UChicago Medicine, Chicago, IL, USA
Autonomous AI agents are increasingly used to automate biomedical research, creating a growing need to benchmark their capabilities on real scientific data and workflows. Their reliability in neuroimaging remains insufficiently characterized, where plausible analytical outputs can conceal errors that affect scientific conclusions. Existing studies have advanced scientific question answering and tool-based automation, but provide limited evidence on workflow reconstruction, acquisition of procedural expertise and evidence-based scientific judgment. Here we present NIBench, a neuroimaging benchmark comprising three complementary evaluations of autonomous agents. First, we benchmark execution and workflow reconstruction using 56 tasks across seven tool ecosystems, covering structural, diffusion and functional magnetic resonance imaging, positron emission tomography and multimodal analyses. Second, we examine whether agents can evolve their procedural skills through distillation of execution trajectories, evaluating the resulting skills on previously encountered tasks, unseen tasks and other models. Third, we assess scientific judgment using 36 claims from published Alzheimer's disease studies, requiring agents to analyse raw cohort data and provide reproducible evidence. We evaluate 19 language models within a shared agent framework, with selected models undergoing skill distillation and claim evaluation. Leading agents execute supplied analysis protocols nearly perfectly, whereas withholding procedural guidance widens performance differences. Distilled skills improve performance on acquisition tasks and transfer between models, but benefits on unseen tasks are inconsistent. Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and access to data does not always improve judgment.
Figures & tables
Figure 1: Overview of NIBench and the workflow evaluation tasks. a , The evaluation workflow. b , Distribution of the 56 workflow tasks by task tier, imaging modality and primary tool ecosystem. The inner and middle rings indicate task tier and imaging modality, respectively. Radial labels identify individual tasks, with text colors indicating the primary tool ecosystem. c , Example of a workflow task under protocol-guided and protocol-unspecified instructions. The protocol-unspecified condition removes the procedural instructions marked with a minus sign and adds the instruction marked with a plus sign. The task objective and output requirements remain unchanged.
Figure 2: Execution correctness with and without an analysis protocol. a , Mean correctness across 56 tasks for 11 closed-weight and eight open-weight models under protocol-guided (open circles) and protocol-unspecified (filled circles) conditions. b , Correctness by task tier for the three highest-scoring models in each group. c , d , Task-level outcomes under protocol-guided (c) and protocol-unspecified (d) conditions for the 11 closed-weight models and four highest-performing open-weight models.
Figure 3: Acquisition and transfer of procedural skills through trajectory distillation. a , Skill-distillation procedure. b , Change in correctness relative to round 0 across successive distillation rounds for the 28 acquisition tasks (left) and 28 held-out tasks (right). c , Distribution of acquisition and held-out tasks by tier, imaging modality, and primary tool ecosystem. d , Correctness by task tier for closed-weight and open-weight models at their selected distillation rounds. e , Held-out correctness using the selected distilled library compared with the expert-written library. f , Held-out correctness using different skill libraries, including the initial seed, expert-written, self-distilled, and cross-model distilled libraries.
Figure 4: Evaluation of scientific judgment. a , Overview of the claim-evaluation procedure. b , Performance of Claude Opus 5.5 and GPT-6-astra across the three scientific questions. Radial plots show conclusion agreement, evidence reproducibility and analytical-route scores. The overall score is the weighted sum of the three components. c , Confusion matrices comparing agent conclusions with expert-derived reference answers, pooled across the three questions and three runs. Outlined cells indicate incorrect conclusions, categorized by whether the agent was overly cautious or overly certain. Bars below the matrices show the corresponding distribution of no-data responses. d , Proportion of runs matching the reference answer for each claim, with data-enabled and no-data results shown separately.
Figure 5: Examples of differences in scientific judgment between agents. a , Evaluation of a claim concerning dementia progression in amyloid- and tau-positive cognitively unimpaired participants. b , Evaluation of a claim concerning the association between longitudinal tau-PET changes and cognitive decline.
Figure 6: Computational efficiency and failure patterns across language models. a , Correctness plotted against output tokens per correct task under protocol-guided and protocol-unspecified conditions. Arrows connect the two conditions for each model. Red lines indicate the Pareto frontiers, with the inset showing the highest-performing region. b , Distribution of task outcomes, including successful completion, incorrect outputs, missing outputs and timeouts, grouped by model class and instruction condition. c , Median output tokens per correct task across the four task tiers. d , Proportion of failed shell commands by failure category, averaged across models within each class. e – h , Relationships between correctness and process compliance (e), model parameter count (f), median output tokens per task (g) and median turns per task (h). Curves in e and f are locally weighted regressions (LOWESS).
Artificial-intelligence (AI) agents hold promise for automating bioimage analysis, yet no benchmark evaluates whether they can carry out real-world analyses end to end. Such analyses are hard for agents because 2D images, 3D volumes and time-lapse sequences are often too large to read as context, so an agent must choose and run an analysis through code, specialized software and rendered views. Published studies make this capability testable, because each pairs raw images with a peer-reviewed result. We introduce BIABench, a benchmark of 16 tasks reconstructed from published biological studies that retain their scientific questions, imaging data and ground truth. The tasks span eleven analysis subtasks and modalities from H&E histology to single-molecule localization microscopy. Each submission receives an outcome score, which compares the output files with the ground truth using field-standard metrics, and a process score, in which a vision-language model judges method choice and quality control against an expert-written rubric. We evaluated general-purpose and biology-specific agents across several language models, with repeated runs of every task. Routine two-dimensional tasks were solved well, but on some tasks that added a third dimension or a time axis no agent scored above 0.19. Neither biological specialization, stronger models nor detailed expert instructions closed this gap. The agents were also unreliable, with scores varying more between repeated runs of one agent than between different agents, and without ground truth a correct run could not be told from a wrong one by its process score or by the time spent. Released openly with its data and code, BIABench provides a verifiable framework for evaluating, and eventually training, agents for reliable long-horizon bioimage analysis.
Zixuan Pan, Davide Panzeri, Lukas Johanns +5
Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, USA · Leibniz-Institut für Analytische Wissenschaften – ISAS – e.V., Dortmund, Germany · Institute of Computer Science, University of Tartu, Tartu, Estonia
Autonomous agents are increasingly expected to support end-to-end medical-AI research workflows, moving beyond isolated prediction tasks or short-form clinical question answering. However, existing medical agent benchmarks primarily evaluate final outputs, providing limited visibility into agent behavior within the research process. To address this gap, we present AutoMedBench, a workflow-aware benchmark for autonomous medical-AI research across diverse medical imaging and multimodal inference tasks, organizing agent execution into a unified five-stage workflow (S1-S5): Plan, Setup, Validate, Inference, and Submit. It comprises long-horizon tasks with each run averaging 33 agent turns, spanning five research tracks: segmentation, image enhancement, visual question answering (VQA), report generation, and lesion detection. Each task is evaluated under two difficulty tiers, Lite and Standard, which use the same data and metrics but differ in the amount of task-brief scaffolding, and each run is scored using both final task performance and S1-S5 stage scores, enabling stage-level analysis from the initial task brief to the final submitted artifact. Across thousands of recorded runs, stage-level scoring reveals that Validate is the weakest workflow stage on average, whereas Setup is the strongest, suggesting that current agents are better at making pipelines executable than at verifying their reliability. Post-run error analysis further shows that verification and submission failures dominate tagged errors, accounting for 37.7% and 38.1% of fired codes respectively, whereas task-understanding errors are rare at 0.9%, and runs with one fired error code have a 48% lower overall score than runs with no error code on average.
Reliable automated research requires agents to vet data, verify analyses, and generate hypotheses grounded in trustworthy evidence, potentially reducing routine scientific workload while allowing scientists to focus on interpretation and discovery. Existing benchmarks often only assess analytical task completion or hypothesis generation separately rather than testing whether reliable evidence supports valid and novel claims. We introduce DISCERN (Data Integrity and Scientific Capability: Evidence, Reasoning, and Novelty), a controlled benchmark on real, publicly available datasets that evaluates three key levels of an automated research workflow. The first two levels test data integrity and analysis verification under confounds and tool traps, while the third tests hypothesis generation and revision under adversarial review, including counterfactual cases in which evidence consistent with real data and documented scientific phenomena conflicts with established expectations, motivating alternative explanations and testable hypotheses. Across 203 tasks, eight life-science tracks, and eight models, DISCERN shows that strong aggregate performance can mask level-specific weaknesses. Agents earn perfect scores in only 60.8% of Level 1, 34.2% of Level 2, and 0.6% of Level 3 evaluations, with penalties attributed to rejection of sound data, failure to carry recognized limitations into conclusions, and wide variation in hypothesis production. Cross-track rankings by token and code use are substantially more stable than rankings by evidence judgment, suggesting greater consistency in computational effort than in evidence-based reasoning. These profiles identify opportunities for supervised scientific assistance, but current agents do not yet demonstrate reliable autonomous analysis or discovery. Code and data: https://huggingface.co/datasets/discern-bench-anon/discern-benchmark
Nan Huang, Mario Tapia-Pacheco, Kun Zhou +4
University of California San Diego · Independent Researcher