LLM agents can use feedback from repeated attempts to improve their decisions without updating model parameters. However, positive action credits can continue to favor actions that repeatedly achieve the same score, even when higher scores are achievable. Under a fixed interaction budget, trying new actions in the final attempts can also lower performance. We introduce EARN, a method that uses past experience to guide further progress and retain earlier performance through conditional reuse of effective action sequences. EARN reduces action credits for reaching scores repeatedly attained in earlier episodes and assigns larger credits to action segments that reach higher scores. These credits adjust the probabilities of LLM-proposed actions. During the final episodes, EARN directly replays stored actions when the current context matches the recorded context and the actions remain available, bypassing LLM candidate generation. We evaluate performance at the end of the interaction budget using the average score over the final four episodes. Across Jericho, ScienceWorld, and Crafter, EARN achieves the best score on this metric in 31 of 33 task--budget settings among the evaluated methods, including ties. Across the evaluated Jericho budgets, replay bypasses 78.9--86.3% of candidate-generation calls in the final four episodes, averaged across games.
Figure 2: An illustrative example of EARN’s block-wise and step-wise credit attribution. Environment scores are illustrative. Both attribution rules use a frontier of 10 in this example; the step-wise calculation includes only score-increase signals.
Episodes / Steps
Method
DeepHome
Detective
Zork I
Zork III
ZTUU
Last-4
Last
Last-4
Last
Last-4
Last
Last-4
Last
Last-4
Last
30/30
Memory
13.8
14.0
70.0
80.0
5.0
5.0
1.8
2.0
3.8
5.0
Reflexion
6.8
7.0
20.0
20.0
2.5
0.0
0.8
1.0
0.0
0.0
AWM
14.0
14.0
50.0
50.0
10.0
10.0
0.0
0.0
0.0
0.0
EvoTest
14.8
15.0
82.5
90.0
40.0
40.0
2.3
2.0
2.5
5.0
JitRL
19.0
18.0
172.5
190.0
42.3
50.0
2.5
2.0
6.3
10.0
Table 1: Jericho results under different episode/step budgets. Last-4 denotes the average score over the final four episodes, and Last denotes the final episode’s score ( ↑ ). Bold values indicate the best results within each budget.
Episodes
Method
Conductivity
Unknown Conductivity
Plant
Power
Melting Known
Last-4
Last
Last-4
Last
Last-4
Last
Last-4
Last
Last-4
Last
10
Memory
19.0
19.0
13.3
10.0
11.3
13.0
58.0
58.0
100.0
100.0
AWM
19.0
19.0
14.5
19.0
11.0
13.0
20.0
20.0
93.3
100.0
Reflexion
10.0
10.0
10.0
10.0
3.3
1.0
10.0
10.0
86.5
73.0
EvoTest
27.8
29.0
24.3
29.0
11.8
13.0
58.0
58.0
93.3
100.0
JitRL
47.5
60.0
60.0
60.0
34.8
37.0
22.8
0.0
84.8
100.0
Table 2: ScienceWorld results across task variations, i.e., instances of the same task with different objects, materials, or environmental configurations. Last-4 denotes the mean solving score over the final four episodes, and Last denotes the final episode’s score ( ↑ ). Each episode allows 100 steps.
Table 3: Crafter achievement counts (0–22), with 100 steps per episode. Last-4 denotes the final-four average; Last denotes the final episode’s score ( ↑ ).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
Problem Formulation
K,Tk,W
Episode budget, steps in episode k , and number of final evaluation episodes.
sk,t,ak,t,yk,t
Execution context, action, and environment feedback at step t of episode k .
θ,xk,t
Frozen LLM parameters and the prompt at step t of episode k .
Sk,t,Sk
Total score after step t and episode score assigned by the scoring criterion.
Jterm
Expected average episode score over the final W episodes.
Appendix
Table 6: Main notation used in EARN.
Game
30/30
50/50
70/70
Last-4
Last
Final/Run
Last-4
Last
Final/Run
Last-4
Last
Final/Run
DeepHome
93.08
100.00
15.84
84.88
84.00
9.05
90.00
90.00
5.22
Detective
100.00
100.00
14.13
97.87
97.87
9.24
97.22
97.22
5.46
Zork I
76.67
90.00
10.98
88.00
88.00
8.22
78.28
90.91
5.35
Zork III
84.85
90.91
10.67
50.55
48.65
4.21
92.69
82.76
3.96
ZTUU
73.33
73.33
10.32
73.23
82.00
8.07
73.17
50.00
4.72
Appendix
Table 7: Candidate-generation call savings (%) for EARN on five Jericho games. Last-4 averages the final four per-episode rates; Last reports the final episode. Final/Run counts only replayed steps in the final four episodes, divided by all executed steps in the run. Average gives equal weight to each game. Budget labels give episodes/maximum steps per episode.
Figure 5: ZTUU case study in the 30-episode, 30-step setting. The upper panels illustrate the two credit sources and a later combined-memory decision. The lower panel separately shows conditional replay and fallback in the final episode. The probability change reflects accumulated memory, not an isolated contribution from either displayed episode.
Figure 6: EARN general prompt structure. Observations are textual in Jericho and ScienceWorld and include visual input in Crafter.
Step-level credit evaluation can conflate numerical association with ordering and branch recovery with position-selection value. We test these distinctions in ALFWorld with two independently seeded replay banks: 27,520 continuations over 172 fixed turns. At 16 continuations per arm per bank, cross-bank replay-contrast associations are Pearson 0.736 and pooled Spearman 0.330. On 30 historical failures, A-count-selected branches at confidence-/implicit-selected positions gain 8.54 percentage points over same-position factual reruns, but their 3.33-point advantage over frozen-random positions under the same alternative-selection rule remains unresolved. Numerical association alone does not establish reliable pooled ordering; branch recovery alone does not establish better score-informed position selection. Historical judge alignment is modestly positive with an unresolved archived-control increment. Same-cohort C/D information-source comparisons and a short, sparse-exposure training pilot provide scope checks; incremental hindsight training utility remains unestablished. Findings are conditional on the tested policies, intervention support and finite budgets.
Haiyue Zhang
University of Southern California, Los Angeles, CA, USA
Large language model (LLM) agents have shown strong decision-making capabilities in long-horizon interactive tasks, yet they still struggle to effectively leverage failed trajectories: full retries incur high interaction costs, while experience retrieval tends to dilute critical experience signals. To address this, we propose PivoARL, a self-feedback retry framework for experience exploitation in LLM agents. PivoARL identifies the pivotal erroneous turn through structured reflection and performs local retry only from the corresponding pivotal state, thereby reusing the correct prefix and reducing redundant interactions. From an information-gain perspective, we further show that pivotal retry concentrates useful experience signals near the error boundary, mitigating the signal dilution caused by state-agnostic experience utilization. Based on this insight, we design a pivotal-aware credit assignment mechanism that rewards correct prefixes while isolating erroneous suffixes, and optimize reflection quality through implicit reflection returns. We conduct a systematic evaluation on 4 agent tasks and 7 search-based QA benchmarks. Results show that PivoARL achieves significant improvements on Pass@2/3 across all tasks, with an average gain of about 11.5% over MetaRL. Moreover, benefiting from contrastive preference signals induced by pivotal turns, PivoARL also consistently improves Pass@1 on over 80% of the tasks. On Minesweeper environment, PivoARL improves over GiGPO by more than 45% and reduces interaction turns by about 42% on average compared with full-retry methods. Code is available at https://github.com/yuki-younai/PivoARL.
LLM agents have made impressive progress in coding and mathematical reasoning, motivating growing interest in learning from long-horizon interaction. Memory supports this process, but plausible lessons can be redundant or reinforce inappropriate actions. Can a memory's source decision provide a practical test for whether to retain it? We localize consequential trajectory steps, extract candidate memories, and replay each source decision with and without its candidate under the same context. This reveals two properties: most candidates barely shift the probed distribution, and both successful and failed source actions can be reinforced or suppressed. These observations motivate Anchor, which tests shift magnitude and source-role alignment before retention. Admitted memories guide later trajectories, enabling experience-driven improvement under a frozen policy. On held-out ALFWorld and WebShop tasks, Anchor improves success rate and mean score by up to 19.9% and 17.2%, respectively, relative to the strongest baseline for each backbone. With Qwen3.5-27B, it retains 41% fewer ALFWorld cards than unfiltered retention and outperforms all four WebShop cost-comparison baselines while using fewer tokens.
Xinxiang Yin, Xinglang Zhang, Haodong Chen +7
The University of Hong Kong · Huazhong University of Science and Technology · The Hong Kong University of Science and Technology +4