Toward Evaluation Frameworks for Multi-Agent Scientific AI Systems

Marcin Abram

cs.CYMar 18, 2026

Toward Evaluation Frameworks for Multi-Agent Scientific AI Systems

Authors: Marcin Abram

Organizations: Los Angeles, California 90089, USA

Abstract

We analyze the challenges of benchmarking scientific (multi)-agentic systems, including the difficulty of distinguishing reasoning from retrieval, the risks of data/model contamination, the lack of reliable ground truth for novel research problems, the complications introduced by tool use, and the replication challenges due to the continuously changing/updating knowledge base. We discuss strategies for constructing contamination-resistant problems, generating scalable families of tasks, and the need for evaluating systems through multi-turn interactions that better reflect real scientific practice. As an early feasibility test, we demonstrate how to construct a dataset of novel research ideas to test the out-of-sample performance of our system. We also discuss the results of interviews with several researchers and engineers working in quantum science. Through those interviews, we examine how scientists expect to interact with AI systems and how these expectations should shape evaluation methods.

Figures & tables

Figure 1: An architectural overview of a generalized scientific AI system for physics research assistance. The interaction layer may take multiple forms, including a conversational assistant, a research workspace, an augmented text/code editor, a background agent, or a different form. The orchestration layer coordinates actions, routes problems, maintains state, plans workflows, executes tools, etc. The data layer is not restricted to offline databases; we assume that our system can and will perform a life search on the internet. The validation layer is not executed only at the end, it might be in constant dialog with the orchestration layer providing feedback and critic of particular steps.

Figure 2: A map showing Rx errors (the nodes; darker is better) and CZ errors (the edges; darker is better) for ibm_torino (133 qubit IBM quantum processor). Figure taken from URL https://quantum.cloud.ibm.com/computers?system=ibm_torino .

Figure 3: Sentiment toward two dimensions: critical thinking ability and problem-solving ability. Results based on the conducted interviews.

Figure 4: Major themes of future extensions extracted from recent quant-ph ArXiv papers.

Scientific Agent Benchmarks

See all 36 papers

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Jun 10, 2026Tianyu Liu, Allen Xin Wang, Antonia Panescu +30AI for Science Scientific Agent Benchmarks
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7
AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents
Oct 4, 2026Dongki Kim, Namkyeong Lee, Surag Nair +7AI for Science AI Agent Evaluation
Genentech · KAIST
Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists
Oct 4, 2026Kate Zhang, Yuante LiAI Agent Evaluation Scientific Agent Benchmarks
School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213

Jun 10, 2026J/K move · Enter open · S save

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

Tianyu Liu, Allen Xin Wang, Antonia Panescu, Lisa Xinyi Chen+29

Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA+7

AI for Science Scientific Agent Benchmarks AI Agent Evaluation AI Agent Benchmarks Autonomous Scientific Discovery

Open paper AskPDF arXiv

AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.

Abstract

Scientific Agent Benchmarks

Explore similar work

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents

Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists