From AutoResearch to Continual AutoResearch: Do Agents Learn Across Tasks?
Organizations: RBC Borealis · ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center
Abstract
Autoresearch has shown that agents can autonomously write code, run experiments, and iteratively improve machine learning solutions. Yet most evaluations start a new agent session from scratch for each new problem, making them costly and leaving unclear whether agents can retain and apply prior insights when adapting to new tasks. We introduce CONTINUAL AUTORESEARCH, an open-ended environment that compares persistent and reset instances of the same agent under matched task streams, data, and time budget. Continual AutoResearch covers nine task families, spanning synthetic tasks, symbolic regression, time-series forecasting, and tabular AutoML. Across five frontier agents, we measure first-solution quality, research efficiency, and final held-out performance. To distinguish insight transfer from session continuity, we run a memory ablation that vary what information agents retain from prior tasks. We show that persistent state improves research efficiency of all five agents, allowing them to find stronger solutions sooner. However, these gains do not consistently translate into better final solutions. Overall, our findings suggest that retaining experience and utilizing it effectively are distinct capabilities.
Figures & tables
| Setting | Task families |
| Synthetic AutoML (3) | Shared latent, shared sparse, shared regime |
| Symbolic regression (1) | Canonical coordinates |
| Time-series forecasting (3) | Energy systems, traffic networks, ETT cross-resolution ( Lai et al., 2018 ; Godahewa et al., 2021 ; Song et al., 2020 ; Zhou et al., 2021 ) |
| Tabular AutoML (2) | SELA evaluation subset (AMLB/Kaggle), MLE-Bench Lite ( Chi et al., 2024 ; Gijsbers et al., 2024 ; Chan et al., 2025 ) |
| Model | Stateless outcome | Outcome gain | Init. gain | Time-conv. gain | Outcome win rate | Init. win rate |
| Opus 5 | 79.9 | -0.9 [+0.0] | +1.3 [+0.5] | +7.5 [+10.3] | 34.3% | 46.3% |
| GPT-5.6-Sol | 80.5 | -0.8 [+0.2] | -1.0 [+1.2] | +6.1 [+7.4] | 59.3% | 69.4% |
| Kimi K3 | 73.1 | +0.4 [+2.0] | +1.8 [+2.0] | +21.3 [+27.5] | 62.0% | 64.8% |
| Sonnet 4.6 | 70.3 | +7.6 [+1.3] | +7.0 [+7.0] | +20.9 [+20.6] | 66.7% | 75.0% |
| GPT-5.4 | 72.9 | +0.2 [-0.2] | +3.0 [+3.1] | +3.0 [+2.0] | 51.9% | 75.0% |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Equation template | Rel./all | Control |
| 1 | 3/9 | No | |
| 2 | 3/10 | No | |
| 3 | 4/12 | No | |
| 4 | 4/14 | Yes | |
| 5 | 5/16 | No |
| Model | Tasks | Outcome gain | Init. gain | Time-conv. gain | Outcome win rate | Negative transfer | Valid S/R | Failed S/R |
| Opus 5 | 32 | -0.9 [+0.0] | +1.3 [+0.5] | +7.5 [+10.3] | 34.3% | 65.7% | 147/149 | 13/16 |
| GPT-5.6-Sol | 32 | -0.8 [+0.2] | -1.0 [+1.2] | +6.1 [+7.4] | 59.3% | 40.7% | 199/397 | 18/7 |
| Kimi K3 | 32 | +0.4 [+2.0] | +1.8 [+2.0] | +21.3 [+27.5] | 62.0% | 38.0% | 108/109 | 2/14 |
| Sonnet 4.6 | 32 | +7.6 [+1.3] | +7.0 [+7.0] | +20.9 [+20.6] | 66.7% | 33.3% | 180/160 | 15/71 |
| GPT-5.4 | 32 | +0.2 [-0.2] | +3.0 [+3.1] | +3.0 [+2.0] | 51.9% | 48.1% | 153/200 | 20/23 |
| Task nature | Families | Tasks | Outcome gain | Init. gain | Time-conv. gain | Outcome win rate | Negative transfer |
| Classification | 5 | 18 | +0.8 [+0.4] | +3.4 [+2.0] | +16.4 [+18.4] | 60.0% | 40.0% |
| Regression | 3 | 10 | +0.8 [-0.1] | +3.4 [+1.7] | +3.7 [+7.4] | 50.0% | 50.0% |
| Symbolic reg. | 1 | 4 | +5.2 [-2.9] | -5.6 [+1.6] | +12.6 [+5.0] | 55.0% | 45.0% |
| Setting | Task family | Tasks | Outcome gain | Init. gain | Time-conv. gain | Outcome win rate | Negative transfer | Valid S/R | Failed S/R |
|---|---|---|---|---|---|---|---|---|---|
| Claude Code / Opus 5 | |||||||||
| All settings | Overall | 32 | -0.9 [+0.0] | +1.3 [+0.5] | +7.5 [+10.3] | 34.3% | 65.7% | 147/149 | 13/16 |
| Synth. AutoML | Shared latent | 4 | -0.1 [-0.1] | +0.1 [+0.2] | +3.1 [+4.2] | 25.0% | 75.0% | 13/13 | 0/0 |
| Synth. AutoML | Shared regime | 4 | +0.5 [+0.7] | +12.7 [+9.6] | +24.0 [+17.9] | 75.0% | 25.0% | 23/20 | 2/2 |
| Synth. AutoML | Shared sparse | 4 | -3.5 [-3.6] | -1.2 [-1.2] | +24.5 [+22.1] | 0.0% | 100.0% | 16/16 | 0/1 |
| Symbolic reg. | Canonical coord. | 4 | +1.3 [+0.0] | +3.7 [-0.1] | +29.0 [+29.0] | 25.0% | 75.0% | 26/20 | 8/0 |
| Agent | Time to first valid (min) S/R | Valid candidates S/R | First-solution gain | Full-budget gain | Gain with same candidate count | Gain excluding first-solution latency |
| Opus 5 | 7.7/11.1 | 4.4/4.9 | +1.3 | +7.5 | -0.5 | -0.3 |
| GPT-5.6-Sol | 6.1/8.5 | 6.7/11.8 | -1.0 | +6.1 | +0.1 | -1.1 |
| Kimi K3 | 9.5/18.7 | 3.5/3.6 | +1.8 | +21.3 | +1.7 | +1.5 |
| Sonnet 4.6 | 7.7/16.3 | 5.5/4.7 | +7.0 | +20.9 | +6.5 | +6.6 |
| GPT-5.4 | 8.4/9.2 | 4.7/6.3 | +3.0 | +3.0 | +1.2 | +0.5 |
| Agent | 25% | 50% | 75% | 100% |
| Opus 5 | +18.3 | +19.8 | -2.8 | -0.8 |
| GPT-5.6-Sol | +11.4 | +9.5 | +3.5 | -1.2 |
| Kimi K3 | +44.6 | +26.1 | +13.2 | +1.7 |
| Sonnet 4.6 | +26.7 | +21.3 | +24.6 | +6.9 |
| GPT-5.4 | +18.9 | +1.6 | -6.2 | -0.5 |
| Mean | +24.0 | +15.7 | +6.5 | +1.2 |
| Agent | Candidate 1 | Candidate 2 | Candidate 3 |
| Opus 5 | +0.7 (32) | -0.5 (32) | -0.4 (28) |
| GPT-5.6-Sol | -0.8 (32) | +0.6 (31) | -0.3 (29) |
| Kimi K3 | +1.7 (32) | +1.6 (29) | +0.5 (10) |
| Sonnet 4.6 | +6.8 (32) | +4.2 (28) | +12.2 (18) |
| GPT-5.4 | +3.7 (32) | +0.9 (32) | -1.0 (24) |
| Agent | Paired follow-up tasks | Improving follow-ups S/R | First-to-best improvement S/R | Incremental area gain |
| Opus 5 | 32 | 29.9%/44.4% | 1.8/3.3 | -1.0 |
| GPT-5.6-Sol | 31 | 31.5%/36.8% | 5.3/5.7 | -0.3 |
| Kimi K3 | 29 | 34.8%/25.3% | 0.5/0.6 | -0.2 |
| Sonnet 4.6 | 28 | 32.0%/26.4% | 4.2/4.1 | -0.2 |
| GPT-5.4 | 32 | 27.2%/43.6% | 1.1/5.3 | -3.2 |
| Stream length | Init. gain | Time-conv. gain | Outcome gain | Valid attempts | First/selected RMSE |
| 3 | 3 | ||||
| 5 | 3 | ||||
| 7 | 2 |
| Model | Tasks | Outcome gain M/S | Outcome gain M/N | Init. gain M/S | Time-conv. gain M/S |
| Opus 5 | 4 | +1.0 [+0.7] | +0.2 [+0.4] | +2.6 [+2.7] | -15.5 [-13.2] |
| GPT-5.6-Sol | 4 | -3.2 [-0.2] | +0.3 [-0.1] | +3.9 [+0.2] | -27.3 [-28.7] |
| Sonnet 4.6 | 4 | -0.2 [-1.2] | +14.7 [+8.3] | -2.9 [-1.5] | +25.6 [+26.8] |
| GPT-5.4 | 4 | +0.2 [-0.1] | +5.0 [+2.3] | +5.9 [+7.8] | -0.9 [-1.6] |
| Agent | 25% | 50% | 75% | 100% |
| Opus 5 | -51.8 | -18.6 | +3.0 | +1.6 |
| GPT-5.6-Sol | -14.1 | -53.3 | -35.3 | -3.6 |
| Sonnet 4.6 | +29.2 | +46.0 | +15.2 | -0.2 |
| GPT-5.4 | -15.3 | +4.4 | +0.3 | -0.1 |
| Mean | -13.0 | -5.4 | -4.2 | -0.6 |
Autonomous Research Agents
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.
See all 350 papers