Autoresearch has shown that agents can autonomously write code, run experiments, and iteratively improve machine learning solutions. Yet most evaluations start a new agent session from scratch for each new problem, making them costly and leaving unclear whether agents can retain and apply prior insights when adapting to new tasks. We introduce CONTINUAL AUTORESEARCH, an open-ended environment that compares persistent and reset instances of the same agent under matched task streams, data, and time budget. Continual AutoResearch covers nine task families, spanning synthetic tasks, symbolic regression, time-series forecasting, and tabular AutoML. Across five frontier agents, we measure first-solution quality, research efficiency, and final held-out performance. To distinguish insight transfer from session continuity, we run a memory ablation that vary what information agents retain from prior tasks. We show that persistent state improves research efficiency of all five agents, allowing them to find stronger solutions sooner. However, these gains do not consistently translate into better final solutions. Overall, our findings suggest that retaining experience and utilizing it effectively are distinct capabilities.
Figures & tables
Figure 1: Overview of Continual AutoResearch . Stateful and stateless agents process the same ordered task stream under matched environments, and per-task budgets. Each task consists of an autoresearch loop, but only the stateful agent retains experience across tasks. Private evaluation of final artifacts enables paired measurement of cross-task gains in performance, and efficiency.
Setting
Task families
Synthetic AutoML (3)
Shared latent, shared sparse, shared regime
Symbolic regression (1)
Canonical coordinates
Time-series forecasting (3)
Energy systems, traffic networks, ETT cross-resolution ( Lai et al., 2018 ; Godahewa et al., 2021 ; Song et al., 2020 ; Zhou et al., 2021 )
Tabular AutoML (2)
SELA evaluation subset (AMLB/Kaggle), MLE-Bench Lite ( Chi et al., 2024 ; Gijsbers et al., 2024 ; Chan et al., 2025 )
Table 1: Continual AutoResearch evaluation coverage. The evaluation spans four settings and nine task families, ranging from controlled synthetic and symbolic tasks with known recurring structure to real-world forecasting and tabular AutoML streams.
Figure 2: Persistent-state search advantage over time. Mean best hidden reward as a function of normalized task-budget elapsed. Task-level best-so-far curves are constructed post hoc, averaged within each family, and then averaged with equal family weight. Persistent execution generally finds useful solutions earlier, although the size and duration of the advantage vary across agents.
Model
Stateless outcome
Outcome gain
Init. gain
Time-conv. gain
Outcome win rate
Init. win rate
Opus 5
79.9
-0.9 [+0.0]
+1.3 [+0.5]
+7.5 [+10.3]
34.3%
46.3%
GPT-5.6-Sol
80.5
-0.8 [+0.2]
-1.0 [+1.2]
+6.1 [+7.4]
59.3%
69.4%
Kimi K3
73.1
+0.4 [+2.0]
+1.8 [+2.0]
+21.3 [+27.5]
62.0%
64.8%
Sonnet 4.6
70.3
+7.6 [+1.3]
+7.0 [+7.0]
+20.9 [+20.6]
66.7%
75.0%
GPT-5.4
72.9
+0.2 [-0.2]
+3.0 [+3.1]
+3.0 [+2.0]
51.9%
75.0%
Table 2: Model-level results over 32 transfer tasks per agent. Reset outcome is the family-balanced mean final reward under reset execution. Gains are stateful-minus-reset normalized reward points reported as mean [median]. Win rates report the family-balanced fraction of tasks on which stateful execution outperforms reset execution.
Figure 3: Effect of per-task wall-clock budget on Shared Regime. Stateful-minus-reset final-outcome, initialization, and time-convergence gains over transfer tasks (positions 2–5), in normalized reward points. Points show matched repetitions; markers and bars show their mean and sample standard deviation across three repetitions. Each budget is a separate run, rather than a checkpoint taken from a longer run.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Equation template
Rel./all
Control
1
c+au12+qu1sin(u2)+rcos(v3+u1u2)
3/9
No
2
c+aexp(−u12)(1+qcos(u2))+rsin(v3u1)
3/10
No
3
c+aexp[−(u12+u22+u32)](1+qcos(v4))
4/12
No
4
c+au12+ku13+qsin(u2)+rcos(v3v4)
4/14
Yes
5
c+acos(u12+u22+u32)+qsin(v4v5)
5/16
No
Appendix
Table 3: Construction of the canonical-coordinates symbolic-regression stream. “Rel./all” gives the number of relevant and total features.
Model
Tasks
Outcome gain
Init. gain
Time-conv. gain
Outcome win rate
Negative transfer
Valid S/R
Failed S/R
Opus 5
32
-0.9 [+0.0]
+1.3 [+0.5]
+7.5 [+10.3]
34.3%
65.7%
147/149
13/16
GPT-5.6-Sol
32
-0.8 [+0.2]
-1.0 [+1.2]
+6.1 [+7.4]
59.3%
40.7%
199/397
18/7
Kimi K3
32
+0.4 [+2.0]
+1.8 [+2.0]
+21.3 [+27.5]
62.0%
38.0%
108/109
2/14
Sonnet 4.6
32
+7.6 [+1.3]
+7.0 [+7.0]
+20.9 [+20.6]
66.7%
33.3%
180/160
15/71
GPT-5.4
32
+0.2 [-0.2]
+3.0 [+3.1]
+3.0 [+2.0]
51.9%
48.1%
153/200
20/23
Appendix
Table 4: Model-level results over transfer-eligible tasks. Gains are normalized reward points and are reported as mean [median]. Aggregates use equal task-family weights. S/R denotes stateful/reset.
Task nature
Families
Tasks
Outcome gain
Init. gain
Time-conv. gain
Outcome win rate
Negative transfer
Classification
5
18
+0.8 [+0.4]
+3.4 [+2.0]
+16.4 [+18.4]
60.0%
40.0%
Regression
3
10
+0.8 [-0.1]
+3.4 [+1.7]
+3.7 [+7.4]
50.0%
50.0%
Symbolic reg.
1
4
+5.2 [-2.9]
-5.6 [+1.6]
+12.6 [+5.0]
55.0%
45.0%
Appendix
Table 5: Transfer by task nature. Classification combines the synthetic and realistic AutoML streams, regression combines the time-series streams, and symbolic regression is reported separately. Gains are normalized reward points reported as mean [median] over model–family means, with each model–family pair equally weighted. Task counts exclude the first task of each stream and are reported per model. Win and negative-transfer rates pool transfer-task outcomes across the five models.
Setting
Task family
Tasks
Outcome gain
Init. gain
Time-conv. gain
Outcome win rate
Negative transfer
Valid S/R
Failed S/R
Claude Code / Opus 5
All settings
Overall
32
-0.9 [+0.0]
+1.3 [+0.5]
+7.5 [+10.3]
34.3%
65.7%
147/149
13/16
Synth. AutoML
Shared latent
4
-0.1 [-0.1]
+0.1 [+0.2]
+3.1 [+4.2]
25.0%
75.0%
13/13
0/0
Synth. AutoML
Shared regime
4
+0.5 [+0.7]
+12.7 [+9.6]
+24.0 [+17.9]
75.0%
25.0%
23/20
2/2
Synth. AutoML
Shared sparse
4
-3.5 [-3.6]
-1.2 [-1.2]
+24.5 [+22.1]
0.0%
100.0%
16/16
0/1
Symbolic reg.
Canonical coord.
4
+1.3 [+0.0]
+3.7 [-0.1]
+29.0 [+29.0]
25.0%
75.0%
26/20
8/0
Appendix
Table 6: Complete stateful-versus-reset results over transfer-eligible tasks. Gains are normalized reward points and are reported as mean [median]. S/R denotes stateful/reset, and TS denotes time series. Overall rows use equal family weights; family rows summarize tasks. Task 1 is excluded.
Figure 4: Run-to-run variability for GPT-5.6-Sol across three repeated stateful–reset comparisons. Dots show individual runs, circles their mean, and error bars ±1 standard deviation; positive values favor stateful execution.
Figure 5: Outcome transfer gain measures the difference in final selected hidden-test reward between persistent and reset execution on the same task (stateful minus stateless). Results cover 32 transfer tasks per agent, excluding the first task of every stream. Values are normalized reward points ( 100× reward difference), so positive values favor persistent state. Sonnet 4.6 shows the largest aggregate gain ( +7.6 points); GPT-5.4 and Kimi K3 remain near zero, while Opus 5 and GPT-5.6-Sol are slightly negative.
Figure 6: Initialization gain compares first valid solutions; time-convergence gain summarizes the best-so-far reward over the task budget. Open circles are task-family means, and diamonds are equal-family agent means. Gains are stateful minus stateless normalized reward points, so zero indicates no difference and positive values favor stateful execution. Mean time-convergence gain is positive for all five agents, and mean initialization gain is positive for four of five.
Figure 7: Absolute outcome performance under reset and persistent execution. Values are normalized reward points, averaged within each task family and then across families with equal weight. The analysis covers 32 transfer-eligible tasks per agent and excludes the first task of every stream. Opus 5 and GPT-5.6-Sol achieve the strongest reset performance, whereas Sonnet 4.6 benefits substantially from persistent state and nearly closes the absolute-performance gap.
Figure 8: Outcome transfer gain decomposed by agent and task family. Each cell reports the mean persistent-minus-reset outcome reward over the transfer-eligible tasks in that family, in normalized reward points; positive values favor persistent state. Cell labels show the uncapped values, while color intensity uses a signed square-root scale capped at ±20 points to preserve visibility of smaller effects. Transfer is strongly heterogeneous across both agents and recurring task structures.
Figure 9: Curves show the mean best hidden reward attained as a function of normalized task-budget elapsed. Task-level best-so-far curves are constructed post hoc from private-evaluator scores, averaged within each family, and then averaged with equal family weight. Persistent execution generally finds useful solutions earlier, but the size and duration of the advantage vary across agents.
Figure 10: Task-position-resolved search advantage on the controlled synthetic streams. Each curve reports the stateful-minus-stateless best hidden reward, in normalized reward points, over the normalized task budget. Differences are computed within the same task and then averaged with equal weight across the shared-latent, shared-regime, and shared-sparse families. Later tasks exhibit a pronounced and sustained stateful advantage for several agents, although the pattern is not universal across models.
Figure 11: Growth in persistent-state search advantage across the controlled synthetic streams. For each task position, the stateful-minus-stateless reward gap is integrated over the normalized task budget and compared with the corresponding initial-task value from the same model and family. Gray lines show the shared-latent, shared-regime, and shared-sparse families; colored diamonds show equal-family means. Positive values indicate that persistent state provides a larger search advantage on later tasks.
Agent
Time to first valid (min) S/R
Valid candidates S/R
First-solution gain
Full-budget gain
Gain with same candidate count
Gain excluding first-solution latency
Opus 5
7.7/11.1
4.4/4.9
+1.3
+7.5
-0.5
-0.3
GPT-5.6-Sol
6.1/8.5
6.7/11.8
-1.0
+6.1
+0.1
-1.1
Kimi K3
9.5/18.7
3.5/3.6
+1.8
+21.3
+1.7
+1.5
Sonnet 4.6
7.7/16.3
5.5/4.7
+7.0
+20.9
+6.5
+6.6
GPT-5.4
8.4/9.2
4.7/6.3
+3.0
+3.0
+1.2
+0.5
Appendix
Table 7: Decomposition and robustness of time-convergence gains over 32 transfer tasks per agent. Time to first valid solution and valid candidate counts are family-balanced per-task means reported as stateful/reset (S/R). First-solution gain compares the hidden reward of the first valid solutions. Full-budget gain is the headline time-convergence result. The same-candidate-count control uses only the first min(nS,nR) valid candidates from each paired run. The final control starts each trajectory at its own first valid solution, removing first-solution latency while preserving first-solution quality. All gains are normalized reward points; positive values favor stateful execution.
Agent
25%
50%
75%
100%
Opus 5
+18.3
+19.8
-2.8
-0.8
GPT-5.6-Sol
+11.4
+9.5
+3.5
-1.2
Kimi K3
+44.6
+26.1
+13.2
+1.7
Sonnet 4.6
+26.7
+21.3
+24.6
+6.9
GPT-5.4
+18.9
+1.6
-6.2
-0.5
Mean
+24.0
+15.7
+6.5
+1.2
Appendix
Table 8: Stateful-minus-reset difference in the best hidden reward achieved by each budget checkpoint. Values are normalized reward points aggregated with equal task-family weights over 32 transfer tasks per agent. Positive values favor stateful execution; the mean gives equal weight to each agent.
Agent
Candidate 1
Candidate 2
Candidate 3
Opus 5
+0.7 (32)
-0.5 (32)
-0.4 (28)
GPT-5.6-Sol
-0.8 (32)
+0.6 (31)
-0.3 (29)
Kimi K3
+1.7 (32)
+1.6 (29)
+0.5 (10)
Sonnet 4.6
+6.8 (32)
+4.2 (28)
+12.2 (18)
GPT-5.4
+3.7 (32)
+0.9 (32)
-1.0 (24)
Appendix
Table 9: Persistent-minus-reset best-reward gain after the specified number of valid candidates. Values are normalized reward points; parentheses give paired task coverage.
Agent
Paired follow-up tasks
Improving follow-ups S/R
First-to-best improvement S/R
Incremental area gain
Opus 5
32
29.9%/44.4%
1.8/3.3
-1.0
GPT-5.6-Sol
31
31.5%/36.8%
5.3/5.7
-0.3
Kimi K3
29
34.8%/25.3%
0.5/0.6
-0.2
Sonnet 4.6
28
32.0%/26.4%
4.2/4.1
-0.2
GPT-5.4
32
27.2%/43.6%
1.1/5.3
-3.2
Appendix
Table 10: Productivity after the first valid solution over transfer-eligible tasks. Improving follow-ups are the fraction of later valid candidates that establish a new best. First-to-best improvement and incremental area gain are normalized reward points. S/R denotes stateful/reset.
Stream length
Init. gain
Time-conv. gain
Outcome gain
Valid attempts
First/selected RMSE
3
+3.56
−1.75
−1.82
3
0.432/0.427
5
+5.53
−3.61
−0.05
3
0.393/0.392
7
+5.34
−31.44
−0.27
2
0.396/0.396
Appendix
Table 11: Effect of stream length on a common final Weather forecasting task. Stream length includes the final task. GPT-5.4 gains are normalized reward points relative to a fresh run. First and selected scores are test RMSE, where lower is better. Each condition contains one run.
Model
Tasks
Outcome gain M/S
Outcome gain M/N
Init. gain M/S
Time-conv. gain M/S
Opus 5
4
+1.0 [+0.7]
+0.2 [+0.4]
+2.6 [+2.7]
-15.5 [-13.2]
GPT-5.6-Sol
4
-3.2 [-0.2]
+0.3 [-0.1]
+3.9 [+0.2]
-27.3 [-28.7]
Sonnet 4.6
4
-0.2 [-1.2]
+14.7 [+8.3]
-2.9 [-1.5]
+25.6 [+26.8]
GPT-5.4
4
+0.2 [-0.1]
+5.0 [+2.3]
+5.9 [+7.8]
-0.9 [-1.6]
Appendix
Table 12: Memory-ablation results on the four transfer-eligible tasks in the Energy-system stream, reported as mean [median] normalized reward points. M/S compares matched with shuffled cards; M/N compares matched cards with an independent no-memory reference. Positive values favor matched cards. Task 1 is excluded.
Agent
25%
50%
75%
100%
Opus 5
-51.8
-18.6
+3.0
+1.6
GPT-5.6-Sol
-14.1
-53.3
-35.3
-3.6
Sonnet 4.6
+29.2
+46.0
+15.2
-0.2
GPT-5.4
-15.3
+4.4
+0.3
-0.1
Mean
-13.0
-5.4
-4.2
-0.6
Appendix
Table 13: Matched-minus-shuffled difference in the best hidden reward achieved by each budget checkpoint on the four transfer-eligible Energy-system tasks. Values are normalized reward points; positive values favor matched memory. The best-so-far reward is zero before the first valid submission.
Autonomous research agents use language models and tools to search literature, design and run experiments, analyze results, and support scientific discovery.