Into the Unknown: Benchmarking AI agents on open scientific research problems
Authors: Seungone Kim, Jiayi Geng, Young-Jun Lee, Bo Liu, Juyoung Suk, Nicholas Lubbers, Mitchell Wood, Marcus Noack, Yong-Xin Yao, Yuan Shi, Roxana Bujack, Cristina Garcia Cardona, Austin Rodriguez, Ilon Joseph, Woojin Cho, Zora Zhiruo Wang, Chuanyang Jin, Ian Wu, Apurva Gandhi, Tianjian Li, Weiwei Sun, Carolin Lawrence, Kiril Gashteovski, Dongyeop Kang, Jay Shin, Akari Asai, Graham Neubig, Sean Welleck
Organizations: Carnegie Mellon University · University of Minnesota · Stanford University · Trillion Labs · Los Alamos National Laboratory · Sandia National Laboratory · Lawrence Berkeley National Laboratory · Ames National Laboratory · Iowa State University · University of Colorado Boulder · Michigan State University · Lawrence Livermore National Laboratory · TelePIX · Johns Hopkins University · NEC Laboratories Europe · Ss.Cyril and Methodius University of Skopje
Can AI agents carry out research on open scientific problems? We introduce Into the Unknown, a benchmark that provides an agent with a science paper along with a research direction to pursue and tasks it to write a follow-up paper. We work with scientists at US Department of Energy (DOE) national laboratories, who each contribute an unpublished follow-up direction from their own work together with the necessary conditions a submission must meet. While assessing the quality of a follow-up paper is challenging in general, using these conditions to form a rubric makes evaluation more tractable for research problems that have no reference answer. We have nine frontier agents write 15 follow-ups each, spanning quantum computing, plasma physics, computational chemistry, materials science, and scientific machine learning domains. None of the 135 submissions satisfies all the conditions, and 114 fail our integrity check, most often because the paper claims work that was never done or reports numbers that contradict the recorded evidence. Overall, agents readily produce papers that look like follow-ups, but none meets the standard of the domain scientist, leaving substantial headroom. Our code and data are publicly available.
Figures & tables
Figure 1: Specifying the follow-up direction and formulating the scientist’s bar as rubrics make evaluation of open-ended research tractable. (a) For an explicit implementation of the LCHS quantum algorithm [ novikau2026efficient ] , its first author states the direction and the bar. (b) The space of possible follow-ups S1 is unbounded; fixing the direction narrows it to S2 , and the scientist’s bar narrows it further to S3 . (c) Each clause of the bar becomes an item in the direction-specific rubric (yellow) or the integrity rubric (blue). Evaluation checks whether a submission falls in S3 .
Figure 2: Task formulation of Into the Unknown . The AI agent is given a task specification, the original paper, along with its code and data. The AI agent should instantiate the scientist’s idea (in the task specification) by submitting a follow-up paper and also provide the code and logs.
Idx.
Original paper ( follow-up direction proposed by the domain scientist )
Domain
Venue
Cit.
Lab
Table 1: The list of 15 science papers in Into the Unknown . Beneath each paper, colored in gray, is the follow-up direction its own author proposed. Group A was published in 2024 or earlier; Group B was submitted in 2026. Citation counts ( Cit. ) are as of September 14 th , 2026. Affiliated United States Department of Energy national laboratories: LANL (Los Alamos), SNL (Sandia), LBNL (Lawrence Berkeley), LLNL (Lawrence Livermore), PPPL (Princeton Plasma Physics).
AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.
Tianyu Liu, Allen Xin Wang, Antonia Panescu +30
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7
AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.
Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.
Peter Kirgis, Sayash Kapoor, Andrew Schwartz +21
Princeton University · Cornflower Labs · UK AI Security Institute +8