Memory-based self-improving agents--those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank--have shown great promise in recent literature. However, the reliability aspects of these methods have been critically overlooked. In this work, we conduct a comprehensive re-evaluation of two memory-based methods, broadening the scope of evaluation along two axes: (1) including multiple self-improving runs to quantify variance, and (2) shuffling the tasks to investigate the effect of task order. Through these experiments, we make two observations that expose the fragility of current methods: First, agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise. Second, the agent's improvement is highly dependent on task order. Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success. To better understand this fragility, we manually examine the agents' memory and hypothesize that task and environment underspecification contribute to this fragility. We validate this hypothesis by incorporating information that enables better specification, such as detailed rubrics and environment feedback, into the memory construction process. While this added information partially closes the performance degradation in previous experiments, significant gaps still remain, suggesting that other uncharacterized factors contribute to this fragility. Looking ahead, our work advocates for more rigorous evaluation protocols for self-improving agents by reporting results across multiple runs and stress-testing them under challenging conditions. Moreover, our findings on underspecification call for systems and interfaces that enable effective human oversight, preventing agents from failing in unforeseeable ways.
Figures & tables
Figure 1: (a) Overview of Memory-based Self-improving Methods. In this setting, agents will perform a sequence of tasks and generate memories at the end of each tasks. The memories will be saved to a memory bank and will be retrieved for performing future tasks. (b)(c) Example memories. We consider two established methods (AWM [ 11 ] and RBank [ 12 ] ) in our evaluation.
Method
WebArena
VisualWebArena
SCUBA
Shopping
Admin
GitLab
Map
Reddit
Multisite
Classifieds
Shopping
Reddit
Admin
Sales
Service
(187)
(182)
(180)
(109)
(106)
(48)
(234)
(466)
(210)
(158)
(64)
(45)
Standard deviation of pass@1 across 3 runs (%)
Baseline
1.53
1.19
1.98
1.30
1.78
0.98
1.01
0.83
1.03
0.30
1.95
2.77
AWM
1.26 (-18%)
2.99 (+152%)
1.20 (-39%)
1.98 (+53%)
2.48 (+39%)
0.98 (0%)
1.79 (+78%)
1.01 (+22%)
2.38 (+131%)
2.09 (+600%)
1.28 (-35%)
2.77 (0%)
RBank
1.33 (-13%)
2.12 (+79%)
3.34 (+69%)
3.89 (+200%)
2.22 (+25%)
4.28 (+336%)
1.23 (+22%)
1.58 (+91%)
1.47 (+43%)
1.58 (+429%)
1.47 (-24%)
3.78 (+36%)
Table 1: Variance-related metrics across 3 runs. Memory-based self-improving agent methods introduce increased variance in 17 out of 24 ( ≈ 71%) cases, among which 11 cases show a relative increase of more than 50%.
Table 3
Figure 2: Implicit curriculum in the default task order. We visualize the moving average (window=30) of task pass rates as the no-memory baseline agent performs tasks in the default order. In (a)(b), success rates drop on tasks with larger task IDs. In (c), we observe a more complex oscillating pattern. In general, tasks of different difficulties are not distributed evenly in the default order.
Figure 3: Performance using different task orders. We use the default task order and two shuffled orders. Model performance degrade significantly in 6 out of 8 cases when shuffled orders are used.
Figure 4: Mentions of Selected Keywords in Retrieved Memory. In (a)(b), we show that while the environment does not support API-based actions, agents increasingly save and retrieve memories about completing tasks with APIs. In (c), we show that in travel time related tasks in the map domain, the agent proposes to use Haversine Estimation instead of relying on the websites route engine, and this unintended strategy is amplified throughout the course of self-improvement.
Rubrics and Scores (+Rub):
Evaluator: string_match | Score: 0.0 (FAIL)
Prediction: "Top-2 best-selling products in 2022: 1) Quest Lumaflex™ Band, 5 units; 2) Cruise Dual Analog Watch, 4 units."
Table 4: Three types of additional information used in § 4.2 . We hypothesize such information can reduce underspecification when constructing memories. Examples are truncated due to space limit.
Figure 5: Addressing Underspecification with Additional Information. (a) We apply the three types of information individually and altogether. (b) We experiment with +All using three task orders.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Shopping
Admin
GitLab
Map
Reddit
Multisite
All
(187)
(182)
(180)
(109)
(106)
(48)
(812)
Avg. Pass
45.8
62.6
53.9
53.2
69.5
34.0
54.8
Pass@3
55.1
72.0
63.3
67.0
82.1
47.9
65.4
Passˆ3
34.8
52.2
45.0
40.4
53.8
20.8
43.3
Appendix
Table 5: Evaluation of the no-memory baseline agent on WebArena.
Metric
VisualWebArena
SCUBA
Classifieds
Shopping
Reddit
All
Admin
Sales
Service
All
(234)
(466)
(210)
(910)
(158)
(64)
(45)
(267)
Avg. Milestone
-
-
-
-
46.7
61.8
42.3
49.6
Avg. Pass
60.1
57.6
43.3
54.9
26.2
39.6
28.1
29.7
Pass@3
73.9
69.1
51.4
66.3
39.2
57.8
40.0
43.8
Passˆ3
44.9
45.1
33.8
42.4
13.9
20.3
13.3
15.4
Appendix
Table 6: Evaluation of the no-memory baseline agent on VisualWebArena and SCUBA.
Method
Shopping
Admin
GitLab
Map
Reddit
Multisite
All
(187)
(182)
(180)
(109)
(106)
(48)
(812)
No Memory
45.8/1.53/3.74
62.6/1.19/2.75
53.9/1.98/4.44
53.2/1.30/2.75
69.5/1.78/3.77
34.0/0.98/2.08
54.8
Task Order: Ordinal
AWM
48.3/1.26/2.67
59.7/2.99/6.59
51.7/1.20/2.78
50.5/1.98/4.59
72.3/2.48/5.66
32.6/0.98/2.08
54.1
RBank
52.8/1.33/3.21
60.6/2.12/4.95
59.3/3.34/7.78
48.6/3.89/8.26
71.1/2.22/4.72
27.8/4.28/10.42
56.3
Task Order: Shuffle-1
Appendix
Table 7: Per-domain pass rate (%) on WebArena. Each cell reports mean/std/best–worst gap, based on three runs of memory construction. The right-most column (“All”) reports the overall pass rate.
Method
Classifieds
Shopping
Reddit
All
(234)
(466)
(210)
(910)
No Memory
60.1/1.01/2.14
57.6/0.83/1.93
43.3/1.03/2.38
54.9
Task Order: Ordinal
AWM
61.3/1.79/4.27
55.3/1.01/2.15
45.1/2.38/5.71
54.5
ReasoningBank
59.5/1.23/2.99
58.3/1.58/3.86
45.4/1.47/3.33
55.6
Task Order: Shuffle 1
Appendix
Table 8: Per-domain pass rate (%) on VisualWebArena. Each cell reports mean/std/best–worst gap, based on three runs of memory construction. The right-most column (“All”) reports the overall pass rate.
Method
Admin
Sales
Service
All
(158)
(64)
(45)
(267)
Baseline
46.7/0.25/0.59
61.8/1.57/3.36
42.3/1.40/3.02
49.6
Synapse [ 68 ]
51.3/1.39/3.40
64.9/2.64/5.94
48.6/5.24/12.51
54.1
AWM
45.8/1.40/3.25
60.8/2.98/7.19
49.6/4.15/9.13
50.1
ReasoningBank
50.7/0.19/0.47
62.4/2.70/6.59
36.3/2.85/6.11
51.1
Appendix
Table 9: Per-domain milestone score (%) on SCUBA. Each cell reports mean/std/best–worst gap, based on three runs of memory construction. The right-most column (“All”) reports the overall pass rate.
Method
Admin
Sales
Service
All
(158)
(64)
(45)
(267)
Baseline
26.2/0.30/0.63
39.6/1.95/4.69
28.1/2.77/6.67
29.7
Synapse [ 68 ]
30.4/1.37/3.16
43.2/2.66/6.25
28.9/1.81/4.44
33.2
AWM
22.4/2.09/4.43
37.5/1.28/3.12
34.1/2.77/6.67
28.0
ReasoningBank
30.2/1.58/3.80
40.1/1.47/3.12
21.5/3.78/8.89
31.1
Appendix
Table 10: Per-domain pass rate (%) on SCUBA (binary success: milestone score =1.0 ). Each cell reports mean/std/best–worst gap, based on three runs of memory construction. The right-most column (“All”) reports the overall pass rate.
Method
Shopping
Admin
GitLab
Map
Reddit
Multisite
All
(187)
(182)
(180)
(109)
(106)
(48)
(812)
n=3 runs
No Memory
45.8/1.5/3.7
62.6/1.2/2.8
53.9/2.0/4.4
53.2/1.3/2.8
69.5/1.8/3.8
34.0/1.0/2.1
54.8
RBank (Ordinal)
52.8/1.3/3.2
60.6/2.1/5.0
59.3/3.3/7.8
48.6/3.9/8.3
71.1/2.2/4.7
27.8/4.3/10.4
56.3
n=6 runs
No Memory
45.8/1.7/4.3
61.7/1.8/5.5
53.7/1.5/4.4
52.0/2.2/6.4
68.1/2.7/7.6
34.0/1.0/2.1
54.2
Appendix
Table 11: WebArena Results, Three Additional Runs using GPT-5-mini . Each cell reports avg/std/best-worst gap across n runs, and the right-most column (“All”) reports the overall pass rate.
Method
Shopping
Admin
GitLab
Map
Reddit
Multisite
All
(187)
(182)
(180)
(109)
(106)
(48)
(812)
GPT-OSS-120B
No Memory
35.5/1.1/2.7
46.7/0.4/0.8
45.4/0.9/2.2
33.0/1.5/3.7
45.9/2.5/5.7
24.3/3.9/8.3
40.6
RBank (Ordinal)
38.0/0.9/2.1
49.0/1.7/4.1
42.4/0.3/0.6
30.0/2.2/4.6
48.1/5.4/13.2
18.8/1.7/4.2
40.5
RBank (Shuffle-1)
34.3/1.5/3.2
40.4/2.6/5.8
34.5/1.7/3.9
33.9/2.2/5.5
49.4/1.2/2.8
22.9/3.4/8.3
37.0
Gemini-2.5-Pro
Appendix
Table 12: WebArena Results using Additional Models. Each cell reports avg/std/best-worst gap across three runs, and the right-most column (“All”) reports the overall pass rate.
Method
Calendar
CSM
Drive
Email
HR
Hybrid
ITSM
Teams
All
GPT-OSS-120B
No memory
21.3/2.7/6.6
21.4/0.8/1.9
41.2/2.0/4.7
54.2/2.5/6.0
12.4/0.9/2.0
23.6/1.5/3.7
5.8/0.8/1.9
35.5/4.3/9.8
24.5
With memory
26.2/2.7/6.6
19.7/2.8/6.8
33.9/6.3/14.1
52.7/1.4/3.0
5.9/1.4/2.9
25.6/2.0/4.9
1.3/0.5/1.0
36.1/3.5/8.2
22.3
GPT-5.4-mini
No memory
38.3/0.8/1.6
21.4/1.6/3.9
47.9/2.7/6.3
58.7/2.5/6.0
20.6/1.4/2.9
24.8/4.0/9.8
25.6/3.7/7.8
21.3/2.3/4.9
30.5
With memory
36.1/1.3/3.3
25.9/1.7/3.9
51.6/3.4/7.8
56.7/1.2/3.0
16.7/2.1/4.9
23.6/5.8/13.4
24.6/1.2/2.9
15.3/4.7/11.5
29.7
Appendix
Table 13: Enterprise-Ops Gym Results. Each cell reports avg/std/best-worst gap across three runs, and the right-most column (“All”) reports the overall pass rate.
Task Query
I have jaw bruxism problem, show me something that could alleviate the problem.
Ground-Truth Evaluator
“Jaw bruxism” and “mouth guard” must be included in the content of the final page.
Agent’s Final Message
(The agent didn’t browse the shopping website and ends the task after 1 step.) Here are practical things that can help reduce jaw bruxism (teeth grinding/clenching). This is general information — consult your dentist or doctor for personalized advice: 1) Night guard / oral splint - Best: a custom-fitted occlusal guard from your dentist (protects teeth and reduces muscle stress). [text omitted for brevity…]
Agent’s Memory
Appendix
Table 14: Example of Ambiguous Task Query: WebArena Task 118. The task is intended to be a web navigation task but is interpreted as a question answering task by the agent.
Task Query
How much I spend on 4/19/2023 on shopping at One Stop Market?
Ground-Truth Evaluator
The answer string must include “0”.
Agent’s Final Message
I could not find any orders dated 4/19/2023 in your One Stop Market account. I scanned the full order history (37 orders shown after setting per-page to 50) and the extraction returned no matches for 4/19/2023. Therefore, according to this account, the amount spent on 4/19/2023 at One Stop Market is $0.00.
Author Comment
Appendix
Table 15: Example of Evaluator Bug: WebArena Task 329. A correct answer may fail strict string-matching rubric, and further lead to the creations of irrelevant memories.
Task
Check if Duquesne University in Pittsburgh can be reached in one hour by car from Pittsburgh Airport.
Rubric
Answer string must include "yes".
Agent’s Answer
Result and conclusion:
What I could not do: The OpenStreetMap directions page did not show a numeric route or duration because it displayed messages indicating JavaScript was not available, so a live OSRM route summary could not be extracted.
Appendix
Table 16: Example of Using Haversine Formula: WebArena Task 38. The agent receives an reward for including “yes” in its final answer, while the strategy to reach the answer is unintended.
AI agents increasingly operate in environments where they can diagnose failures and improve through experience, yet existing evaluations largely measure what an agent can do at a fixed point in time rather than how effectively it learns. Evaluating self-improvement requires answering three questions: does future performance improve and generalize beyond the interactions that enabled learning; how efficiently are new capabilities acquired; and where does the self-improvement process break down? To answer these questions, we study self-improvement in a controlled setting where agents amortize past experience into reusable artifacts that are inherited by future instances. At each checkpoint, we measure performance on training and held-out environment interactions while accounting for learning cost. We introduce agent plasticity, the efficiency with which an agent converts experience into gains in future held-out performance. Across multiple environments, frontier models exhibit sharply different improvement trajectories despite comparable opportunities to learn. Some achieve substantial and persistent gains, while others remain near or below their initial performance, and gains within the training regime often transfer only partially to out-of-distribution conditions. Endpoint capability and acquisition efficiency also diverge: the agent that ultimately performs best need not be the one that improves most efficiently. Tracing failures through the improvement loop further reveals different candidate bottlenecks. Agents with low plasticity often fail to reuse relevant artifacts, whereas more plastic agents may still fail despite reusing relevant artifacts, pointing to limitations in artifact quality, generalization, or application. Evaluating self-improving agents requires measuring not only what they can do, but how effectively they become better through experience.
Harman Singh, Anton Bakhtin, Rulin Shao +8
UC Berkeley · Meta Superintelligence Labs · University of Washington +1
Self-improving autonomous agents are moving from research prototypes to deployed systems. The primary goal is controllable evolution, or adaptation, from experience with minimal or even no human input. This survey frames modern self-improving agents as adaptive systems that convert experience into accumulated capability gains. We offer a system-level framework that represents a modern agent as a configuration coupling a foundation model with an operational scaffold of prompts, memory, tools, and control logic. Within this framework, self-improvement is formalized as a self-induced update operator that obtains and commits updates to model parameters or scaffold components. We organize prior work by update target and by the signals that drive change, then review applications and discuss evaluation, before closing with open problems and future directions. For convenience, we track technical updates on https://github.com/selfimproving-agent/awesome-Self-Improving-Agents.
Zhe Ren, Yimeng Chen, Dandan Guo +9
School of Artificial Intelligence, Jilin University · King Abdullah University of Science and Technology (KAUST) · Independent Researcher +2
Environments are the foundation of recursive self-improvement: they provide the problems agents work on and the feedback used to evaluate progress. Yet constructing challenging research environments with reliable evaluation still depends on domain experts, limiting their scale and disciplinary coverage. We introduce RSI-Forge, a multi-agent pipeline that turns published papers into executable environments for self-improvement. Three agents coordinate construction, reproduction, and review to produce tasks with automated evaluators; each paper's method is independently reimplemented to establish a baseline score. We present 210 environments across 18 fields, including 90 reviewed by independent human domain experts. Both experts and agent judges give high ratings to the potential for improving the provided starting solutions and the evaluators' ability to distinguish solution quality, whereas experts are more critical of shortcut resistance, faithfulness to the source paper, and whether a single idea can exhaust a task. To validate their use for repeated improvement, we evaluate four models over 3 successive attempts on 120 environments, with each attempt inheriting prior code and notes while model weights remain fixed. At least one model improves after the first attempt in 84% of environments. Models also outperform the reproduced paper methods in 68 of the 120 environments, demonstrating room for gains beyond these baselines. Transcript analysis identifies work beyond parameter tuning in 95% of these successful attempts. Analysis of the resulting trajectories shows that models scoring lower on these tasks explore less, more often accept gains smaller than the reported standard error, and rely more heavily on tuning to the development set. RSI-Forge provides a scalable approach to constructing research environments for training and evaluating self-improving agents.
Renxiong Wang, Darvin Yi, Abril Herrlein +16
Scale AI · University of California, Santa Cruz · University of North Carolina at Chapel Hill