Organizations: Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA · Ming Hsieh Department of Electrical and Computer Engineering, Viterbi School of Engineering, University of Southern California, Los Angeles, CA, USA · Alfred E. Mann Department of Biomedical Engineering, Viterbi School of Engineering, University of Southern California, Los Angeles, CA, USA · Department of Pediatrics, University of Chicago, Chicago, IL, USA · Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA · Comprehensive Cancer Center, UChicago Medicine, Chicago, IL, USA
Autonomous AI agents are increasingly used to automate biomedical research, creating a growing need to benchmark their capabilities on real scientific data and workflows. Their reliability in neuroimaging remains insufficiently characterized, where plausible analytical outputs can conceal errors that affect scientific conclusions. Existing studies have advanced scientific question answering and tool-based automation, but provide limited evidence on workflow reconstruction, acquisition of procedural expertise and evidence-based scientific judgment. Here we present NIBench, a neuroimaging benchmark comprising three complementary evaluations of autonomous agents. First, we benchmark execution and workflow reconstruction using 56 tasks across seven tool ecosystems, covering structural, diffusion and functional magnetic resonance imaging, positron emission tomography and multimodal analyses. Second, we examine whether agents can evolve their procedural skills through distillation of execution trajectories, evaluating the resulting skills on previously encountered tasks, unseen tasks and other models. Third, we assess scientific judgment using 36 claims from published Alzheimer's disease studies, requiring agents to analyse raw cohort data and provide reproducible evidence. We evaluate 19 language models within a shared agent framework, with selected models undergoing skill distillation and claim evaluation. Leading agents execute supplied analysis protocols nearly perfectly, whereas withholding procedural guidance widens performance differences. Distilled skills improve performance on acquisition tasks and transfer between models, but benefits on unseen tasks are inconsistent. Agents reproduce their reported results yet sometimes reach conclusions inconsistent with expert-derived answers, and access to data does not always improve judgment.
Figures & tables
Figure 1: Overview of NIBench and the workflow evaluation tasks. a , The evaluation workflow. b , Distribution of the 56 workflow tasks by task tier, imaging modality and primary tool ecosystem. The inner and middle rings indicate task tier and imaging modality, respectively. Radial labels identify individual tasks, with text colors indicating the primary tool ecosystem. c , Example of a workflow task under protocol-guided and protocol-unspecified instructions. The protocol-unspecified condition removes the procedural instructions marked with a minus sign and adds the instruction marked with a plus sign. The task objective and output requirements remain unchanged.
Figure 2: Execution correctness with and without an analysis protocol. a , Mean correctness across 56 tasks for 11 closed-weight and eight open-weight models under protocol-guided (open circles) and protocol-unspecified (filled circles) conditions. b , Correctness by task tier for the three highest-scoring models in each group. c , d , Task-level outcomes under protocol-guided (c) and protocol-unspecified (d) conditions for the 11 closed-weight models and four highest-performing open-weight models.
Figure 3: Acquisition and transfer of procedural skills through trajectory distillation. a , Skill-distillation procedure. b , Change in correctness relative to round 0 across successive distillation rounds for the 28 acquisition tasks (left) and 28 held-out tasks (right). c , Distribution of acquisition and held-out tasks by tier, imaging modality, and primary tool ecosystem. d , Correctness by task tier for closed-weight and open-weight models at their selected distillation rounds. e , Held-out correctness using the selected distilled library compared with the expert-written library. f , Held-out correctness using different skill libraries, including the initial seed, expert-written, self-distilled, and cross-model distilled libraries.
Figure 4: Evaluation of scientific judgment. a , Overview of the claim-evaluation procedure. b , Performance of Claude Opus 5.5 and GPT-6-astra across the three scientific questions. Radial plots show conclusion agreement, evidence reproducibility and analytical-route scores. The overall score is the weighted sum of the three components. c , Confusion matrices comparing agent conclusions with expert-derived reference answers, pooled across the three questions and three runs. Outlined cells indicate incorrect conclusions, categorized by whether the agent was overly cautious or overly certain. Bars below the matrices show the corresponding distribution of no-data responses. d , Proportion of runs matching the reference answer for each claim, with data-enabled and no-data results shown separately.
Figure 5: Examples of differences in scientific judgment between agents. a , Evaluation of a claim concerning dementia progression in amyloid- and tau-positive cognitively unimpaired participants. b , Evaluation of a claim concerning the association between longitudinal tau-PET changes and cognitive decline.
Figure 6: Computational efficiency and failure patterns across language models. a , Correctness plotted against output tokens per correct task under protocol-guided and protocol-unspecified conditions. Arrows connect the two conditions for each model. Red lines indicate the Pareto frontiers, with the inset showing the highest-performing region. b , Distribution of task outcomes, including successful completion, incorrect outputs, missing outputs and timeouts, grouped by model class and instruction condition. c , Median output tokens per correct task across the four task tiers. d , Proportion of failed shell commands by failure category, averaged across models within each class. e – h , Relationships between correctness and process compliance (e), model parameter count (f), median output tokens per task (g) and median turns per task (h). Curves in e and f are locally weighted regressions (LOWESS).
Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, USA · Leibniz-Institut für Analytische Wissenschaften – ISAS – e.V., Dortmund, Germany · Institute of Computer Science, University of Tartu, Tartu, Estonia