cs.AIOct 6, 2026
SaveCan AI Agents Make Open-Ended Scientific Discovery? Evidence from Station
Organizations: DualverseAI · University of Hong Kong · University of Cambridge
Abstract
Recent AI systems have made rapid progress in scientific discovery when given well-defined metrics, but whether they can autonomously undertake open-ended scientific discovery remains unclear. We investigate AI's ability to tackle open-ended tasks in Station, an open-world environment in which multiple agents simulate a scientific ecosystem. To tackle challenges specific to open-ended tasks, we propose augmenting Station with two mechanisms: a Supervisor mechanism and periodic Meta Reflection, which encourage persistent exploration even when intermediate metrics are lacking. We construct open-ended tasks from three recent oral papers presented at ICLR. We give agents the main research question studied in each paper while withholding the paper's results and disabling web access. We then measure how many of the original findings-partitioned into individual criteria-agents rediscover. We find that Station rediscovers 62.7% of the criteria on average, compared with 15.4% for Codex Multiagent-v2 and 14.4-20.6% for AI Scientist-v2. Ablation and behavioral analyses indicate that adding the two mechanisms together improves research coverage and continuity. We further evaluate Station on two open-ended tasks without oracle papers and find that some of the discoveries made by the agents closely match discoveries reported by researchers after the knowledge cutoff date. Together, these results indicate that a suitable environment can enable agents to autonomously make meaningful progress in open-ended scientific discovery.
Figures & tables
Figure 1 : Decentralized and sustained exploration in Station. A centralized system (left) coordinates research directions through a central controller. Station (right) allows agents with different models and contexts to independently propose and pursue directions, while Supervisor and Meta Reflection discourages early pivot.
| Task | Task type | Main research question | Experimental setup |
|---|---|---|---|
| Emergent planning in RL agents [ 11 ] | Rediscovery | Do model-free RL agents form internal plans, and how do these plans guide their behavior? | RL environment and a trained RL agent checkpoint. |
| Low-rank structure in LLM outputs [ 12 ] | Rediscovery | Do LLM outputs exhibit low-rank structure that can support prediction or generation? | A trained LLM checkpoint and an environment for analyzing its outputs. |
| Temporal representations in RNNs [ 13 ] | Rediscovery | How do RNNs represent temporal information and balance spatial and temporal memory demands? | A -delay task and an environment for training and analyzing RNNs. |
| Subliminal learning in LLMs | Exploratory | How do behavioral traits transfer between LLMs through semantically unrelated data? | A teacher–student fine-tuning setup and a brief summary of prior work. |
| Hallucination in VLMs | Exploratory | How do knowledge-dependent and visual hallucinations arise, and how can they be reduced? | A VLM evaluation setup and a proposal distinguishing the two types of hallucinations. |
Table 1 : Overview of the five open-ended research tasks.
Figure 2 : Sub-discovery recovery across three rediscovery tasks. A , Emergent planning in RL agents. B , Low-rank structure in LLM outputs. C , Temporal representations in RNNs. Recovery rate is the percentage of sub-discoveries recovered within each category or across the whole task. Points and error bars show the mean and minimum–maximum range across three runs, respectively. Baselines receive cumulative experiment time matched to Station’s. Full sub-discovery definitions are provided in Appendix C .
Figure 3 : Ablation and behavioral analysis of Supervisor and Meta Reflection on the emergent-planning task. A , Sub-discovery recovery for Station and ablated Station, in which both Supervisor and Meta Reflection are removed. Points and error bars show the mean and minimum–maximum range across three runs, respectively. B , Agent research behavior during the 10 ticks following Meta Reflection.
Figure 4 : Mechanistic findings in the subliminal-learning task. A , Transfer of a teacher’s cat preference to a student through digit-only training data depends non-monotonically on LoRA rank. B , Retaining only a few leading singular modes of the early-layer MLP LoRA updates recovers most of the cat preference produced by the full update. C , Training rank-128 LoRA adapters only in layers 0–13 restores cat preference, raising cat probability above both the base model and the student trained with rank-128 LoRA in all layers. We further validate that the phenomenon generalizes to other traits (Appendix G.1 ).
Figure 5 : Reducing visual hallucinations through late-layer MLP attenuation. A , Station identifies an intervention that multiplies the MLP outputs of layers 34 and 35 in InternVL3.5-8B-Instruct by 0.5 during prompt prefill. B , The intervention increases the number of correct answers from 28 to 35 on a 40-example dataset for evaluating visual hallucinations. We further validate that the intervention improves performance on additional models and larger evaluation set (Appendix G.2 ).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| System | Output tokens (M) | Cumulative experiment time (h) |
|---|---|---|
| Emergent Planning in RL Agents | ||
| Station (Tick 100) | 3.2–3.5 | 10.7–29.8 |
| Station (Tick 300) | 7.5–9.0 | 34.4–57.3 |
| Codex Multiagent-v2 (GPT) | 3.3–4.3 | 50.2–300.5 |
| AI Scientist-v2 (GPT) | 8.9–10.8 | 70.3–167.2 |
| AI Scientist-v2 (Gemini) | 5.3–5.9 | 46.0–131.8 |
Table 2 : Token usage and cumulative experiment time across systems. Ranges indicate the minimum and maximum across three runs.
| System | Probing | Planning features | Causal intervention | Total |
|---|---|---|---|---|
| Emergent Planning in RL Agents | ||||
| Station (Tick 300) | 99.1 [97.2, 100.0] | 63.9 [50.0, 72.2] | 60.5 [51.9, 66.7] | 75.8 [71.7, 80.8] |
| Station (Tick 100) | 99.1 [97.2, 100.0] | 52.8 [25.0, 75.0] | 63.0 [55.6, 66.7] | 72.4 [62.6, 78.8] |
| Codex Multiagent-v2 (GPT) | 16.7 [0.0, 25.0] | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | 6.1 [0.0, 9.1] |
| AI Scientist-v2 (Claude) | 52.8 [50.0, 58.3] | 0.0 [0.0, 0.0] | 0.0 [0.0, 0.0] | 19.2 [18.2, 21.2] |
| AI Scientist-v2 (Gemini) | 41.7 [0.0, 75.0] | 8.3 [0.0, 25.0] | 0.0 [0.0, 0.0] | 18.2 [0.0, 27.3] |
Table 3 : Sub-discovery recovery at different Station checkpoints and across baselines. Values are mean recovery percentages across three runs, with minimum–maximum ranges in brackets.
| System | Recovery (%, mean SE) |
|---|---|
| Station (Tick 300) | |
| Station (Tick 100) | |
| Codex Multiagent-v2 (GPT) | |
| AI Scientist-v2 (Claude) | |
| AI Scientist-v2 (Gemini) | |
| AI Scientist-v2 (GPT) |
Table 4 : Average sub-discovery recovery across tasks. Values are mean standard error, with equal weight given to each task and standard errors based on run-to-run variation within tasks.
| Trait | Rank | Base (%) | All-layer LoRA (%) | Early-layer LoRA (%) |
|---|---|---|---|---|
| Cat | 128 | 5.72 | 4.43 0.29 | 30.23 8.66 |
| Cat | 256 | 5.72 | 5.54 0.27 | 18.81 5.52 |
| Eagle | 128 | 1.06 | 2.79 0.95 | 64.79 19.93 |
| Eagle | 256 | 1.06 | 2.38 0.84 | 62.38 20.40 |
| Owl | 128 | 0.28 | 1.07 0.34 | 24.00 6.70 |
| Owl | 256 | 0.28 | 0.75 0.08 | 13.95 6.07 |
Table 5 : Trait transfer with all-layer and early-layer LoRA fine-tuning. Values report target-animal preference, with mean s.d. across fine-tuning seeds.
| Early layers | Seed 0 (%) | Seed 1 (%) | Seed 2 (%) | Mean s.d. (%) |
|---|---|---|---|---|
| 6 | 41.86 | 29.44 | 51.42 | |
| 8 | 52.78 | 40.36 | 49.74 | |
| 10 | 43.24 | 32.06 | 46.28 | |
| 12 | 41.90 | 33.60 | 39.44 | |
| 14 (default) | 37.68 | 32.28 | 20.72 | |
| 16 | 29.02 | 23.22 | 25.42 |
Table 6 : Early-layer cutoff sweep for cat preference at LoRA rank 128. For each cutoff, we evaluate three seeds using the canonical evaluation protocol with 50 questions and 100 sampled responses per question, yielding 5,000 responses per seed and cutoff. Means and sample standard deviations are calculated across three seeds. The default corresponds to the Cat, rank-128 setting in Table 5 .
| Model | Baseline accuracy (%) | Intervention accuracy (%) | Accuracy gain (pp) |
|---|---|---|---|
| InternVL3.5-4B | 75.27 0.51 | 78.97 0.45 | +3.70 |
| InternVL3.5-8B | 88.37 0.83 | 89.17 0.32 | +0.80 |
| Qwen3-VL-4B | 89.80 0.10 | 91.03 0.12 | +1.23 |
| Qwen3-VL-8B | 58.53 0.15 | 71.23 0.15 | +12.70 |
Table 7 : Cross-model validation on AMBER-1000. Values are mean s.d. across three independent seeds. Gains are measured in percentage points (pp).
| MLP scale | Seed 1 (%) | Seed 2 (%) | Seed 3 (%) | Mean s.d. (%) |
|---|---|---|---|---|
| 1.00 (Baseline) | 89.3 | 88.1 | 87.7 | 88.37 0.83 |
| 0.75 | 89.7 | 89.3 | 89.3 | 89.43 0.23 |
| 0.50 | 89.4 | 89.3 | 88.8 | 89.17 0.32 |
| 0.25 | 86.7 | 86.2 | 86.3 | 86.40 0.26 |
Table 8 : Scaling sensitivity on AMBER-1000 with InternVL3.5-8B.