Scientific Agent Benchmarks

Latest papers

32 papers9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks
Oct 23, 2026Tianyun Zhong, Wei Wang, Jialin Li +17Scientific Agent Benchmarks LLM Agent Evaluation
University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group +4
PersonaScienceBench: Evaluating Scientific Agents with Diverse Simulated Researchers
Oct 23, 2026Siyang Wu, Weizhi Zhang, Chen Yang +29Scientific Agent Benchmarks LLM Agent Evaluation
University of Chicago · University of Illinois Chicago · Harvard University +15
FrontierCS 2: A Benchmark for Computer Science R&D
Oct 21, 2026Qiuyang Mang, Wenhao Chai, Runyuan He +58Scientific Agent Benchmarks Coding Agent Benchmarks
Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?
Oct 20, 2026Ming Zhong, Stacy Li, Nicholas Carlini +1AI Agents for Scientific Discovery Scientific Agent Benchmarks
Anthropic Fellows Program · Anthropic
From Research Goals to Scientific Closure: Evaluating Agents Through Evidence-Supported Physics Research
Oct 15, 2026Yijia Mao, Song Dai, Tao Chen +14Scientific Agent Benchmarks AI Agents for Scientific Discovery
Shanghai AI Laboratory · Shanghai Jiao Tong University · The HongKong University of Science and Technology(Guangzhou) +1
Into the Unknown: Benchmarking AI agents on open scientific research problems
Oct 13, 2026Seungone Kim, Jiayi Geng, Young-Jun Lee +25Scientific Agent Benchmarks AI-Assisted Scientific Research
Carnegie Mellon University · University of Minnesota · Stanford University +13
Autonomous Science Bench: Evaluating AI Agents for Autonomous Scientific Discovery
Oct 13, 2026Yibo Wen, Han LiuAutonomous Scientific Discovery Scientific Agent Benchmarks
Northwestern University
ScienceClaw: Benchmarking Continual Self-Evolution of AI-for-Science Agents Across the Natural and Social Sciences
Oct 6, 2026Mingda Zhang, Wenjin Liu, Tiesunlong Shen +6Scientific Agent Benchmarks Self-Evolving Agents
The Chinese University of Hong Kong, Shenzhen · Nanyang Technological University · Centre for Biomedical Data Science, Duke-NUS Medical School +2
BioStudyBench: Evaluating Agents on Post-Cutoff Biomedical Studies
Oct 6, 2026David Li, Shaamil Karim, Christian GensbiglerScientific Agent Benchmarks AI Agent Benchmarks
University of Waterloo
AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents
Oct 4, 2026Dongki Kim, Namkyeong Lee, Surag Nair +7AI for Science AI Agent Evaluation
Genentech · KAIST
Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists
Oct 4, 2026Kate Zhang, Yuante LiAI Agent Evaluation Scientific Agent Benchmarks
School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213
OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software
Sep 30, 2026Dingyuan Dai, Heli Qi, Lei Liu +28Computer-Use Agent Benchmarks AI for Science
University of California, Los Angeles · RIKEN AIP · Yale University +12
DISCERN: Can AI Agents Work Like Scientists and Guide Discovery?
Sep 27, 2026Nan Huang, Mario Tapia-Pacheco, Kun Zhou +4AI Agent Reliability Scientific Hypothesis Generation
University of California San Diego · Independent Researcher
Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks
Sep 7, 2026Timothy KassisPrompt Engineering LLM Agent Evaluation
K-Dense
SCIRIGOR:Evaluating Open-Ended Scientific Analysis Beyond Final Scores
Sep 5, 2026Bowen Liu, Shuo Nie, Bodong Du +1Scientific Agent Benchmarks LLM Agent Evaluation
HKUST · Alibaba Group
OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows
Sep 5, 2026Aayam Bansal, Keertan BalajiAI Agent Benchmarks AI for Science
TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents
Sep 4, 2026Zhibo Yang, Chen Zhang, Yuewei Zhang +1AI Agents for Scientific Discovery AI Agent Benchmarks
TruthInsight-AI
Science sandboxes measure the scientific capability of AI agents
Aug 31, 2026Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai +8Protein Fitness Prediction AI Agent Evaluation
The Broad Institute of MIT and Harvard; Cambridge, MA 02142, USA. · Department of Biomedical Informatics, Harvard Medical School; Boston, MA 02115, USA. · The Jackson Laboratory; Bar Harbor, ME 04609, USA. +10
EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards
Aug 24, 2026Zhiqing Cui, Xinxiang Yin, Yihong Tang +11Scientific Agent Benchmarks AI Agent Benchmarks
NUIST · HKU · McGill +7
K-Bench: measuring model performance on real scientific agent requests
Aug 21, 2026Aubrey M. Brueckner, Darshil Patel, Yuhuan He +1AI for Science LLM Agent Evaluation
1K-Dense, Inc.
ASI-Bench: At the Dawn of Artificial Superintelligence
Aug 18, 2026Junwei Zhou, Zhen Sun, Binyu Li +39Scientific Agent Benchmarks AI Agent Benchmarks
How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks
Aug 14, 2026Yanlin Fei, Nazhou Liu, Xinmiao Yu +7Scientific Agent Benchmarks Autonomous Research Agents
Prentis AI · Stanford University · Titan Holdings
SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration
Jul 23, 2026Yinhao Tang, Youqing Fang, Yanan Sun +6AI Agent Benchmarks Information Retrieval
University of Science and Technology of China · Shanghai AI Laboratory
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
Jun 23, 2026Yuru Wang, Lejun Cheng, Yuxin Zuo +14AI for Science Coding Agents
Horizon Research, Frontis.AI Tsinghua University
OpenBioRQ: Unsolved Biomedical Research Questions for Agents
Jun 20, 2026Minbyul JeongAgentic RAG AI Agent Benchmarks
Upstage AI
Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Jun 10, 2026Tianyu Liu, Allen Xin Wang, Antonia Panescu +30AI for Science Scientific Agent Benchmarks
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7
SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?
May 28, 2026Sy-Tuyen Ho, Minghui Liu, Huy Nghiem +6AI Agent Benchmarks LLM Evaluation
University of Maryland · All Purpose AI
ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research
May 28, 2026Wanghan Xu, Shuo Li, Tianlin Ye +48LLM Agent Evaluation AI Agent Benchmarks
InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
Apr 14, 2026Oliver Bentham, Vivek SrikumarLLM Evaluation Scientific Agent Benchmarks
Kahlert School of Computing University of Utah
COMPOSITE-Stem
Apr 10, 2026Kyle Waters, Lucas Nuzzi, Tadhg Looram +20Scientific Agent Benchmarks AI Agent Benchmarks
PortexAI · University of Milano-Bicocca · University of Calgary +17

Oct 23, 2026J/K move · Enter open · S save

SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

Tianyun Zhong, Wei Wang, Jialin Li, Jianhong Tu+16

University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group+4

Scientific Agent Benchmarks LLM Agent Evaluation AI Agent Benchmarks AI for Science Process-Based Agent Evaluation

Open paper AskPDF arXiv

Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activities remain difficult to assess systematically. Existing benchmarks often focus on selected scientific capabilities or task types, limiting a comprehensive characterization of scientific agents' strengths and weaknesses in realistic research workflows. We introduce SCI-AGENT-EXAM, a workflow-oriented benchmark comprising 120 expert-curated tasks across 11 scientific fields and 7 research task types defined by the Scientific Workflow Task Taxonomy (SWTT). Each task requires both executable artifacts (e.g., code and scripts) and textual artifacts (e.g., reports and explanations). To evaluate the heterogeneous task outputs, we develop a hybrid evaluation protocol combining deterministic programmatic verification for executable artifacts with a calibrated, expert-validated LLM-based judge for textual artifacts. Beyond outcome assessment, we analyze progress and solution stability along execution trajectories to characterize how agents reach their final results. Experiments with 10 frontier LLM agents reveal substantial performance gaps: GPT-6 Astra achieves the highest overall score of 80.9, yet only 26.0% of its runs exceed the 0.95 success threshold. Further analyses uncover distinct capability profiles across scientific task types and substantial variation in execution efficiency and stability. Our code and data are available at https://github.com/SKYLENAGE-AI/Sci-Agent-Exam.