LLM agents can use feedback from repeated attempts to improve their decisions without updating model parameters. However, positive action credits can continue to favor actions that repeatedly achieve the same score, even when higher scores are achievable. Under a fixed interaction budget, trying new actions in the final attempts can also lower performance. We introduce EARN, a method that uses past experience to guide further progress and retain earlier performance through conditional reuse of effective action sequences. EARN reduces action credits for reaching scores repeatedly attained in earlier episodes and assigns larger credits to action segments that reach higher scores. These credits adjust the probabilities of LLM-proposed actions. During the final episodes, EARN directly replays stored actions when the current context matches the recorded context and the actions remain available, bypassing LLM candidate generation. We evaluate performance at the end of the interaction budget using the average score over the final four episodes. Across Jericho, ScienceWorld, and Crafter, EARN achieves the best score on this metric in 31 of 33 task--budget settings among the evaluated methods, including ties. Across the evaluated Jericho budgets, replay bypasses 78.9--86.3% of candidate-generation calls in the final four episodes, averaged across games.
Figure 2: An illustrative example of EARN’s block-wise and step-wise credit attribution. Environment scores are illustrative. Both attribution rules use a frontier of 10 in this example; the step-wise calculation includes only score-increase signals.
Episodes / Steps
Method
DeepHome
Detective
Zork I
Zork III
ZTUU
Last-4
Last
Last-4
Last
Last-4
Last
Last-4
Last
Last-4
Last
30/30
Memory
13.8
14.0
70.0
80.0
5.0
5.0
1.8
2.0
3.8
5.0
Reflexion
6.8
7.0
20.0
20.0
2.5
0.0
0.8
1.0
0.0
0.0
AWM
14.0
14.0
50.0
50.0
10.0
10.0
0.0
0.0
0.0
0.0
EvoTest
14.8
15.0
82.5
90.0
40.0
40.0
2.3
2.0
2.5
5.0
JitRL
19.0
18.0
172.5
190.0
42.3
50.0
2.5
2.0
6.3
10.0
Table 1: Jericho results under different episode/step budgets. Last-4 denotes the average score over the final four episodes, and Last denotes the final episode’s score ( ↑ ). Bold values indicate the best results within each budget.
Episodes
Method
Conductivity
Unknown Conductivity
Plant
Power
Melting Known
Last-4
Last
Last-4
Last
Last-4
Last
Last-4
Last
Last-4
Last
10
Memory
19.0
19.0
13.3
10.0
11.3
13.0
58.0
58.0
100.0
100.0
AWM
19.0
19.0
14.5
19.0
11.0
13.0
20.0
20.0
93.3
100.0
Reflexion
10.0
10.0
10.0
10.0
3.3
1.0
10.0
10.0
86.5
73.0
EvoTest
27.8
29.0
24.3
29.0
11.8
13.0
58.0
58.0
93.3
100.0
JitRL
47.5
60.0
60.0
60.0
34.8
37.0
22.8
0.0
84.8
100.0
Table 2: ScienceWorld results across task variations, i.e., instances of the same task with different objects, materials, or environmental configurations. Last-4 denotes the mean solving score over the final four episodes, and Last denotes the final episode’s score ( ↑ ). Each episode allows 100 steps.
Table 3: Crafter achievement counts (0–22), with 100 steps per episode. Last-4 denotes the final-four average; Last denotes the final episode’s score ( ↑ ).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
Problem Formulation
K,Tk,W
Episode budget, steps in episode k , and number of final evaluation episodes.
sk,t,ak,t,yk,t
Execution context, action, and environment feedback at step t of episode k .
θ,xk,t
Frozen LLM parameters and the prompt at step t of episode k .
Sk,t,Sk
Total score after step t and episode score assigned by the scoring criterion.
Jterm
Expected average episode score over the final W episodes.
Appendix
Table 6: Main notation used in EARN.
Game
30/30
50/50
70/70
Last-4
Last
Final/Run
Last-4
Last
Final/Run
Last-4
Last
Final/Run
DeepHome
93.08
100.00
15.84
84.88
84.00
9.05
90.00
90.00
5.22
Detective
100.00
100.00
14.13
97.87
97.87
9.24
97.22
97.22
5.46
Zork I
76.67
90.00
10.98
88.00
88.00
8.22
78.28
90.91
5.35
Zork III
84.85
90.91
10.67
50.55
48.65
4.21
92.69
82.76
3.96
ZTUU
73.33
73.33
10.32
73.23
82.00
8.07
73.17
50.00
4.72
Appendix
Table 7: Candidate-generation call savings (%) for EARN on five Jericho games. Last-4 averages the final four per-episode rates; Last reports the final episode. Final/Run counts only replayed steps in the final four episodes, divided by all executed steps in the run. Average gives equal weight to each game. Budget labels give episodes/maximum steps per episode.
Figure 5: ZTUU case study in the 30-episode, 30-step setting. The upper panels illustrate the two credit sources and a later combined-memory decision. The lower panel separately shows conditional replay and fallback in the final episode. The probability change reflects accumulated memory, not an isolated contribution from either displayed episode.
Figure 6: EARN general prompt structure. Observations are textual in Jericho and ScienceWorld and include visual input in Crafter.
Step-level credit evaluation can conflate numerical association with ordering and branch recovery with position-selection value. We test these distinctions in ALFWorld with two independently seeded replay banks: 27,520 continuations over 172 fixed turns. At 16 continuations per arm per bank, cross-bank replay-contrast associations are Pearson 0.736 and pooled Spearman 0.330. On 30 historical failures, A-count-selected branches at confidence-/implicit-selected positions gain 8.54 percentage points over same-position factual reruns, but their 3.33-point advantage over frozen-random positions under the same alternative-selection rule remains unresolved. Numerical association alone does not establish reliable pooled ordering; branch recovery alone does not establish better score-informed position selection. Historical judge alignment is modestly positive with an unresolved archived-control increment. Same-cohort C/D information-source comparisons and a short, sparse-exposure training pilot provide scope checks; incremental hindsight training utility remains unestablished. Findings are conditional on the tested policies, intervention support and finite budgets.
Haiyue Zhang
University of Southern California, Los Angeles, CA, USA
Large language model (LLM) agents have shown strong decision-making capabilities in long-horizon interactive tasks, yet they still struggle to effectively leverage failed trajectories: full retries incur high interaction costs, while experience retrieval tends to dilute critical experience signals. To address this, we propose PivoARL, a self-feedback retry framework for experience exploitation in LLM agents. PivoARL identifies the pivotal erroneous turn through structured reflection and performs local retry only from the corresponding pivotal state, thereby reusing the correct prefix and reducing redundant interactions. From an information-gain perspective, we further show that pivotal retry concentrates useful experience signals near the error boundary, mitigating the signal dilution caused by state-agnostic experience utilization. Based on this insight, we design a pivotal-aware credit assignment mechanism that rewards correct prefixes while isolating erroneous suffixes, and optimize reflection quality through implicit reflection returns. We conduct a systematic evaluation on 4 agent tasks and 7 search-based QA benchmarks. Results show that PivoARL achieves significant improvements on Pass@2/3 across all tasks, with an average gain of about 11.5% over MetaRL. Moreover, benefiting from contrastive preference signals induced by pivotal turns, PivoARL also consistently improves Pass@1 on over 80% of the tasks. On Minesweeper environment, PivoARL improves over GiGPO by more than 45% and reduces interaction turns by about 42% on average compared with full-retry methods. Code is available at https://github.com/yuki-younai/PivoARL.
Large language models (LLMs) are widely used to power agents that interact with tools and environments to solve complex tasks. Repeated attempts improve the quality of tool-using agents, but continued execution incurs additional costs even when it is unlikely to improve the best result ultimately returned. This raises the question of when to stop an attempt to reduce execution cost while preserving task-level Best-of-N quality. To address this question, we introduce GainCtrl, a gain-based execution controller for LLMs. In GainCtrl, a lightweight CPU predictor uses interaction histories to estimate expected positive gains over retained scores, which are then used to guide stopping decisions with calibrated thresholds to limit the loss in mean Best-of-N task quality. We evaluate the controller through trajectory replay across three environments and three LLM actors. Comparisons of interpolated quality-cost Pareto frontiers show that GainCtrl generally uses fewer tokens than baselines at matched target quality. Fixed policies also achieve quality close to that of full execution while substantially reducing token consumption. The predictor runs at millisecond latency on a single CPU thread, supporting repeated monitoring without access to LLM internals.
Kangwen Zhao, Jianfeng Cai, Ruopei Sun +4
University of Science and Technology of China · Ant Digital Technologies, Ant Group