Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

Wenyu Du; Stephen Chung

cs.AIOct 6, 2026

Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

Authors: Wenyu Du, Stephen Chung

Organizations: DualverseAI · University of Hong Kong · University of Cambridge

Abstract

Recent AI systems have made rapid progress in scientific discovery when given well-defined metrics, but whether they can autonomously undertake open-ended scientific discovery remains unclear. We investigate AI's ability to tackle open-ended tasks in Station, an open-world environment in which multiple agents simulate a scientific ecosystem. To tackle challenges specific to open-ended tasks, we propose augmenting Station with two mechanisms: a Supervisor mechanism and periodic Meta Reflection, which encourage persistent exploration even when intermediate metrics are lacking. We construct open-ended tasks from three recent oral papers presented at ICLR. We give agents the main research question studied in each paper while withholding the paper's results and disabling web access. We then measure how many of the original findings-partitioned into individual criteria-agents rediscover. We find that Station rediscovers 62.7% of the criteria on average, compared with 15.4% for Codex Multiagent-v2 and 14.4-20.6% for AI Scientist-v2. Ablation and behavioral analyses indicate that adding the two mechanisms together improves research coverage and continuity. We further evaluate Station on two open-ended tasks without oracle papers and find that some of the discoveries made by the agents closely match discoveries reported by researchers after the knowledge cutoff date. Together, these results indicate that a suitable environment can enable agents to autonomously make meaningful progress in open-ended scientific discovery.

Figures & tables

Figure 1 : Decentralized and sustained exploration in Station. A centralized system (left) coordinates research directions through a central controller. Station (right) allows agents with different models and contexts to independently propose and pursue directions, while Supervisor and Meta Reflection discourages early pivot.

Task	Task type	Main research question	Experimental setup
Emergent planning in RL agents [ 11 ]	Rediscovery	Do model-free RL agents form internal plans, and how do these plans guide their behavior?	RL environment and a trained RL agent checkpoint.
Low-rank structure in LLM outputs [ 12 ]	Rediscovery	Do LLM outputs exhibit low-rank structure that can support prediction or generation?	A trained LLM checkpoint and an environment for analyzing its outputs.
Temporal representations in RNNs [ 13 ]	Rediscovery	How do RNNs represent temporal information and balance spatial and temporal memory demands?	A $k$ -delay task and an environment for training and analyzing RNNs.
Subliminal learning in LLMs	Exploratory	How do behavioral traits transfer between LLMs through semantically unrelated data?	A teacher–student fine-tuning setup and a brief summary of prior work.
Hallucination in VLMs	Exploratory	How do knowledge-dependent and visual hallucinations arise, and how can they be reduced?	A VLM evaluation setup and a proposal distinguishing the two types of hallucinations.

Table 1 : Overview of the five open-ended research tasks.

Figure 2 : Sub-discovery recovery across three rediscovery tasks. A , Emergent planning in RL agents. B , Low-rank structure in LLM outputs. C , Temporal representations in RNNs. Recovery rate is the percentage of sub-discoveries recovered within each category or across the whole task. Points and error bars show the mean and minimum–maximum range across three runs, respectively. Baselines receive cumulative experiment time matched to Station’s. Full sub-discovery definitions are provided in Appendix C .

Figure 3 : Ablation and behavioral analysis of Supervisor and Meta Reflection on the emergent-planning task. A , Sub-discovery recovery for Station and ablated Station, in which both Supervisor and Meta Reflection are removed. Points and error bars show the mean and minimum–maximum range across three runs, respectively. B , Agent research behavior during the 10 ticks following Meta Reflection.

Figure 4 : Mechanistic findings in the subliminal-learning task. A , Transfer of a teacher’s cat preference to a student through digit-only training data depends non-monotonically on LoRA rank. B , Retaining only a few leading singular modes of the early-layer MLP LoRA updates recovers most of the cat preference produced by the full update. C , Training rank-128 LoRA adapters only in layers 0–13 restores cat preference, raising cat probability above both the base model and the student trained with rank-128 LoRA in all layers. We further validate that the phenomenon generalizes to other traits (Appendix G.1 ).

Figure 5 : Reducing visual hallucinations through late-layer MLP attenuation. A , Station identifies an intervention that multiplies the MLP outputs of layers 34 and 35 in InternVL3.5-8B-Instruct by 0.5 during prompt prefill. B , The intervention increases the number of correct answers from 28 to 35 on a 40-example dataset for evaluating visual hallucinations. We further validate that the intervention improves performance on additional models and larger evaluation set (Appendix G.2 ).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

System	Output tokens (M)	Cumulative experiment time (h)
Emergent Planning in RL Agents
Station (Tick 100)	3.2–3.5	10.7–29.8
Station (Tick 300)	7.5–9.0	34.4–57.3
Codex Multiagent-v2 (GPT)	3.3–4.3	50.2–300.5
AI Scientist-v2 (GPT)	8.9–10.8	70.3–167.2
AI Scientist-v2 (Gemini)	5.3–5.9	46.0–131.8

Appendix

Table 2 : Token usage and cumulative experiment time across systems. Ranges indicate the minimum and maximum across three runs.

System	Probing	Planning features	Causal intervention	Total
Emergent Planning in RL Agents
Station (Tick 300)	99.1 [97.2, 100.0]	63.9 [50.0, 72.2]	60.5 [51.9, 66.7]	75.8 [71.7, 80.8]
Station (Tick 100)	99.1 [97.2, 100.0]	52.8 [25.0, 75.0]	63.0 [55.6, 66.7]	72.4 [62.6, 78.8]
Codex Multiagent-v2 (GPT)	16.7 [0.0, 25.0]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]	6.1 [0.0, 9.1]
AI Scientist-v2 (Claude)	52.8 [50.0, 58.3]	0.0 [0.0, 0.0]	0.0 [0.0, 0.0]	19.2 [18.2, 21.2]
AI Scientist-v2 (Gemini)	41.7 [0.0, 75.0]	8.3 [0.0, 25.0]	0.0 [0.0, 0.0]	18.2 [0.0, 27.3]

Appendix

Table 3 : Sub-discovery recovery at different Station checkpoints and across baselines. Values are mean recovery percentages across three runs, with minimum–maximum ranges in brackets.

System	Recovery (%, mean $\pm$ SE)
Station (Tick 300)	$62.7\pm 2.8$
Station (Tick 100)	$61.2\pm 4.1$
Codex Multiagent-v2 (GPT)	$15.4\pm 2.4$
AI Scientist-v2 (Claude)	$17.2\pm 1.2$
AI Scientist-v2 (Gemini)	$20.6\pm 3.8$
AI Scientist-v2 (GPT)	$14.4\pm 0.7$

Appendix

Table 4 : Average sub-discovery recovery across tasks. Values are mean

\pm

standard error, with equal weight given to each task and standard errors based on run-to-run variation within tasks.

Trait	Rank	Base (%)	All-layer LoRA (%)	Early-layer LoRA (%)
Cat	128	5.72	4.43 $\pm$ 0.29	30.23 $\pm$ 8.66
Cat	256	5.72	5.54 $\pm$ 0.27	18.81 $\pm$ 5.52
Eagle	128	1.06	2.79 $\pm$ 0.95	64.79 $\pm$ 19.93
Eagle	256	1.06	2.38 $\pm$ 0.84	62.38 $\pm$ 20.40
Owl	128	0.28	1.07 $\pm$ 0.34	24.00 $\pm$ 6.70
Owl	256	0.28	0.75 $\pm$ 0.08	13.95 $\pm$ 6.07

Appendix

Table 5 : Trait transfer with all-layer and early-layer LoRA fine-tuning. Values report target-animal preference, with mean

\pm

s.d. across fine-tuning seeds.

Early layers	Seed 0 (%)	Seed 1 (%)	Seed 2 (%)	Mean $\pm$ s.d. (%)
6	41.86	29.44	51.42	$40.91\pm 11.02$
8	52.78	40.36	49.74	$47.63\pm 6.47$
10	43.24	32.06	46.28	$40.53\pm 7.49$
12	41.90	33.60	39.44	$38.31\pm 4.26$
14 (default)	37.68	32.28	20.72	$30.23\pm 8.66$
16	29.02	23.22	25.42	$25.89\pm 2.93$

Appendix

Table 6 : Early-layer cutoff sweep for cat preference at LoRA rank 128. For each cutoff, we evaluate three seeds using the canonical evaluation protocol with 50 questions and 100 sampled responses per question, yielding 5,000 responses per seed and cutoff. Means and sample standard deviations are calculated across three seeds. The default corresponds to the Cat, rank-128 setting in Table 5 .

Model	Baseline accuracy (%)	Intervention accuracy (%)	Accuracy gain (pp)
InternVL3.5-4B	75.27 $\pm$ 0.51	78.97 $\pm$ 0.45	+3.70
InternVL3.5-8B	88.37 $\pm$ 0.83	89.17 $\pm$ 0.32	+0.80
Qwen3-VL-4B	89.80 $\pm$ 0.10	91.03 $\pm$ 0.12	+1.23
Qwen3-VL-8B	58.53 $\pm$ 0.15	71.23 $\pm$ 0.15	+12.70

Appendix

Table 7 : Cross-model validation on AMBER-1000. Values are mean

\pm

s.d. across three independent seeds. Gains are measured in percentage points (pp).

MLP scale	Seed 1 (%)	Seed 2 (%)	Seed 3 (%)	Mean $\pm$ s.d. (%)
1.00 (Baseline)	89.3	88.1	87.7	88.37 $\pm$ 0.83
0.75	89.7	89.3	89.3	89.43 $\pm$ 0.23
0.50	89.4	89.3	88.8	89.17 $\pm$ 0.32
0.25	86.7	86.2	86.3	86.40 $\pm$ 0.26

Appendix

Table 8 : Scaling sensitivity on AMBER-1000 with InternVL3.5-8B.

Explore similar work

Sep 30, 2026cs.CL

EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights

When Isaac Newton discovered the law of gravitation, he did so through an iterative process of analyzing observed data such as planetary patterns, finding the underlying mechanisms by describing patterns in mathematical equations, and refining his theory against the Moon's orbit, revealing the startling insight that the same force governs both falling apples and orbiting planets. Would it be possible for AI agents to make similar discoveries? To measure this ability, we introduce EurekaBench, a cross-domain benchmark that tests AI agents' ability to conduct long-horizon experiments and discover mechanisms that explain observations. We evaluate these mechanisms by the scientific insights that can be derived from them. EurekaBench contains an expert-verified set of 26 long-horizon tasks across neuroscience, computer science, chemistry, astrophysics, geophysics, and plasma physics, with a total of 306 scientific insights that the discovered mechanisms are expected to support. Our evaluation framework tests three axes of scientific discovery: agents' ability to follow known scientific constraints, the predictive accuracy of the discovered mechanisms, and whether these mechanisms yield scientific insights or inform future research. Our results show that current AI agents often overly fixate on predictive accuracy optimization, surpassing human scientists, while falling substantially short in deriving scientific insights.

Jiayi Geng, Zhengxuan Wu, Kevin S. Chen +12

Carnegie Mellon University, Language Technologies Institute · Stanford University, Department of Computer Science · Yale University, Quantitative Biology Institute +8

Scientific Discovery Agentic Discovery Artificial Intelligence Agents Long-Horizon Task Planning Agentic

Ask PDF

Jun 10, 2026cs.AI

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood. Existing benchmarks for AI agents rarely capture the complexity, heterogeneity, and extended reasoning required by scientific work, whereas benchmarks for scientific tasks often reduce research to static, direct problems and provide limited support for interactive evaluation. Here, we introduce SciAgentArena, a systematic benchmark for evaluating AI agents in real-world scientific research scenarios drawn from emerging needs across multiple domains. SciAgentArena comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents. Using this benchmark, we find that current agents can contribute effectively to well-specified data-analysis workflows, particularly when the task structure and evaluation criteria are clear. However, their performance remains uneven across scientific contexts: agents struggle to generate genuinely novel insights, sustain self-directed exploration, and formulate robust solutions for open-ended research questions. We further characterize common failure modes across agents and identify opportunities for improving their reliability, autonomy, and scientific reasoning. Together, SciAgentArena provides a practical framework for measuring progress in AI agents for science and for guiding the design of future agents capable of addressing complex scientific challenges. Full codes, tasks, and datasets can be accessed via this link: https://sciagentarena.github.io/.

Tianyu Liu, Allen Xin Wang, Antonia Panescu +30

Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7

Scientific Discovery Agentic Benchmarks Artificial Intelligence Agents Exploration

Ask PDF

Jun 9, 2026cs.CL

Harnessing the Collective Intelligence of AI Agents in the Wild for New Discoveries

Scientific discovery is often a collective process: researchers share partial results, inspect failed attempts, and build on each other's ideas over long time horizons. Recent AI systems have shown that language-model-based agents can make meaningful progress on open scientific problems, but most existing systems operate in isolation. In this paper, we present EinsteinArena, an agent-native platform for open distributed research and discovery. EinsteinArena provides agents with a live set of open problems, each with a solid verifier, public leaderboard, and problem-specific discussion forum where agents can ask questions and share insights. We focus on mathematical tasks that have garnered substantial research interest, where progress can be measured unambiguously. As of May 2026, agents on EinsteinArena have discovered 12 new state-of-the-art results better than any previous human or AI solutions. One notable example is the kissing number problem in dimension 11, where the platform improved the best known lower bound from 593 to 604. This advance did not come from a single agent or isolated run. Rather it arose through a sequence of submissions, public discussion, verifier refinement, and subsequent agent-to-agent borrowing of ideas. These results provide evidence that decentralized scientific discovery can emerge from open interaction among autonomous agents in the wild, demonstrating a new paradigm for collective AI-driven research.

Federico Bianchi, Yongchan Kwon, Aneesh Pappu +1

Together AI · Stanford University

Scientific Discovery Artificial Intelligence Agents Intelligence Autonomous Agents

Ask PDF