cs.CLSep 30, 2026
SaveEurekaBench: Measuring Agentic Ability to Discover New Scientific Insights
Organizations: Carnegie Mellon University, Language Technologies Institute · Stanford University, Department of Computer Science · Yale University, Quantitative Biology Institute · Stanford University, Department of Geophysics · Carnegie Mellon University, Department of Chemistry · Massachusetts Institute of Technology, Plasma Science and Fusion Center · Columbia University, Department of Applied Physics and Applied Mathematics · Flatiron Institute, Center for Computational Astrophysics · Princeton University, Department of Astrophysical Sciences · Princeton University, Department of Computer Science · Engram
Abstract
When Isaac Newton discovered the law of gravitation, he did so through an iterative process of analyzing observed data such as planetary patterns, finding the underlying mechanisms by describing patterns in mathematical equations, and refining his theory against the Moon's orbit, revealing the startling insight that the same force governs both falling apples and orbiting planets. Would it be possible for AI agents to make similar discoveries? To measure this ability, we introduce EurekaBench, a cross-domain benchmark that tests AI agents' ability to conduct long-horizon experiments and discover mechanisms that explain observations. We evaluate these mechanisms by the scientific insights that can be derived from them. EurekaBench contains an expert-verified set of 26 long-horizon tasks across neuroscience, computer science, chemistry, astrophysics, geophysics, and plasma physics, with a total of 306 scientific insights that the discovered mechanisms are expected to support. Our evaluation framework tests three axes of scientific discovery: agents' ability to follow known scientific constraints, the predictive accuracy of the discovered mechanisms, and whether these mechanisms yield scientific insights or inform future research. Our results show that current AI agents often overly fixate on predictive accuracy optimization, surpassing human scientists, while falling substantially short in deriving scientific insights.
Figures & tables
Figure 1: The scientific discovery process in Eurekabench . In this example from plasma physics, scientists observe that different transport models predict different fusion power outputs for the same reactor under the same operating conditions (observation) . Through iterative experiments, they find that differences in turbulent transport change temperature and density distributions, explaining the different power predictions (mechanism) . This helps them understand how turbulent transport may affect fusion power in larger reactors or under stronger magnetic fields (insight) . Discovering such insights is a “eureka moment” in science.
Figure 2: Initial observations, evaluation, and results in Eurekabench . Left: Examples of initial observations from six domains: (a) neuroscience, (b) geophysics, (c) plasma physics, (d) astrophysics, (e) computer science, and (f) chemistry. Top right: An ice shelf flow-law example illustrates how an agent uses iterative experiments to propose a mechanism, which is evaluated for scientific constraints, predictive accuracy, and scientific insights. SSA denotes the shallow-shelf approximation. Bottom right: Predictive accuracy and scientific insight scores for seven AI agents and the human reference. The strongest agents approach human predictive accuracy while attaining substantially lower insight scores.
Figure 3: Scientist agent workflow in Eurekabench . Given a task specification and a documented environment interface, the scientist agent conducts iterative experiments to discover a mechanism and submits its description, implementation, and experimental records.
Figure 4: The 9 interpretable insight questions for the Antarctic ice shelf problem in geophysics in Eurekabench . These insights explain why ice deforms differently under compression and extension, how its resistance varies with direction, and which measurements determine its viscosity.
Figure 5: Problem distribution across domains.
Figure 6: Effects of goals and insight questions on mechanism discovery. We evaluate Claude Fable 5.1 with general guidance (+goal) or insight questions (+insights); stacked bars show estimated PA and SI contributions to the conditioned score, with marking changes from baseline.
Figure 7: Insights discovered during the experimentation loop vs. those derived from the discovered mechanism. Both scores use the same problems and instances.
Figure 8: Expert review of mechanism discovery. Experts review experiment logs of Claude Fable 5.1 and GPT 6 Astra on neuroscience and plasma physics tasks, highlighting failures in mechanism discovery alongside successful hypothesis rejection and understanding of physical concepts.
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Task specification | |
|---|---|
| Context | You are a systems neuroscientist studying {problem_description} . |
| {scientific_background} {observations_block} | |
| Initial observations | However, it is unclear {initial_observations} . |
| Task | Your task is to discover the mechanism that explains this phenomenon. {mechanism_requirements} To discover this, you interrogate {environment_interface} through controlled numerical experiments designed to identify {experimental_objectives} . The structure of your predictive form needs to follow from the mechanism you uncover: for each ingredient, explain why it is included, what it traces in the circuit, and how it shapes the neuron’s activity. |
| Evaluation | Every claim in your discovered mechanism needs to be backed by data you actually collected. {predictive_accuracy_evaluation} Your experimental findings will also be reviewed by an independent researcher who has access to the same simulator, to assess whether the mechanism you discover can yield new insights. |
| Deliverables | You need to produce three deliverables. If any of them is missing or does not follow the requirements below, your solution receives a score of 0. Deliverable 1: Write your discovered mechanism to {output_dir} /mechanisms/mechanism.md. State the mechanism as the causal chain from the coarse action state of the population to the activity of one neuron; its executable form, with every ingredient and fitted constant defined and the physical process each one traces identified; why its inputs jointly determine the neuron’s activity; the concrete evidence from your experiments that supports the mechanism and its form, including the evidence that let you reject the competing explanations you considered; and the insights about how population activity organizes the activity of individual neurons that follow from what you found. |
Table 3: Task specification template for the scientist agent (neuroscience).
| Problem | Goal |
| Neural dynamics | A good mechanism is not only accurate but also extrapolatable, and it can even open up more interesting new research directions. Through such a mechanism people come to understand how the nervous system of C. elegans organizes its activity into a hierarchical locomotion structure, down to how the coarse action state shapes a single neuron. |
| ARC tokamak | A good mechanism is not only accurate but also extrapolatable, and it can even open up more interesting new research directions. Through such a mechanism people come to understand how the transport a model returns sets the profiles an ARC plasma settles into and the fusion power it produces, down to why two models that leave the plasma with the same energy confinement disagree by a factor of two in power. |
Table 4: Goal guidance used in the +goal condition.
Figure 9: PA and SI scores across domains. Predictive accuracy (a) and SI scores (b) for seven AI agents on 26 problems across six domains. Each cell shows one agent’s score on one problem, averaged over its instances using the same rules as Table 2 . Both panels use the same color scale. Gray dashes mark the excluded Kimi K3 run on catchment properties .
| Judge | Reversals (%) | ||
| Both judges (average) | 0.20 [0.07, 0.32] | 39.9 [33.2, 46.5] | 0.008 |
| Claude Opus 4.8 | 0.15 [0.03, 0.26] | 41.8 [35.3, 48.4] | 0.054 |
| GPT 5.6 Sol | 0.15 [0.01, 0.29] | 42.8 [34.3, 51.9] | 0.054 |
Table 8: PA and SI rankings. Brackets show 95% bootstrap confidence intervals; -values are computed using 100,000 within-problem permutations under the null hypothesis of no association and adjusted across the three rows using Holm’s method ( Phipson and Smyth, 2010 ; Holm, 1979 ) .
Figure 10: Missing tests of the target behavior. Original outputs from AI scientist agents.
Figure 11: Missing tests of the target behavior (continued).
Figure 12: Explaining only part of the target phenomenon. Original outputs from AI scientist agents.
Figure 13: Explaining only part of the target phenomenon (continued).
Figure 14: Explaining only part of the target phenomenon (continued).
Figure 15: Explaining only part of the target phenomenon (continued).
Figure 16: Prediction with limited explanation. Original outputs from AI scientist agents.
Figure 17: Prediction with limited explanation (continued).
Figure 18: Prediction with limited explanation (continued).
Figure 19: Prediction with limited explanation (continued).
Figure 20: Unclear explanations. Original outputs from AI scientist agents.
Figure 21: Unclear explanations (continued).
Figure 22: Overgeneralization. Original outputs from AI scientist agents.
Figure 23: Failing to identify and reduce uncertainty. Original outputs from AI scientist agents.
Figure 24: Treating task optimization as mechanism discovery. Original outputs from AI scientist agents.
Figure 25: Treating task optimization as mechanism discovery (continued).
Figure 26: Treating task optimization as mechanism discovery (continued).
Figure 27: Interesting observations without further exploration. Original outputs from AI scientist agents.
Figure 28: Conflicting results and conclusions. Original outputs from AI scientist agents.
Figure 29: Risk of circular validation. Original outputs from AI scientist agents.
Figure 30: Using controls to rule out alternative explanations. Original outputs from AI scientist agents.
Figure 31: Testing which model better predicts the observations. Original outputs from AI scientist agents.
Figure 32: Revising models after experimental feedback. Original outputs from AI scientist agents.
Figure 33: Informative observations. Original outputs from AI scientist agents.
| Scientific domains | Problem names | Sources |
| Neuroscience | Premotor states | Morrison and Young,2025 |
| Zigzag foraging | Zhao et al.,2024 | |
| Biological networks | Chen et al.,2025a | |
| Neural dynamics | Fieseler et al.,2025 | |
| Navigation strategies | Chen et al.,2026 | |
| Phase memory | Dunn et al.,2025 |
Table 9: Problems and sources in Eurekabench .