RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback

Xiaoying Zhang; Zichen Liu; Yipeng Zhang; Xia Hu; Wenqi Shao

cs.AIMar 9, 2026

RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback

Authors: Xiaoying Zhang, Zichen Liu, Yipeng Zhang, Xia Hu, Wenqi Shao

Organizations: Shanghai AI Lab · National University of Singapore · Independent Researcher

Abstract

Standard reinforcement learning (RL) for large language model (LLM) agents primarily optimizes extrinsic task rewards, often favoring isolated task completion over continual adaptation. This paradigm can cause premature convergence to suboptimal policies and leaves useful experience only implicitly encoded in model parameters, limiting its retrieval and reuse for future decisions. We introduce RetroAgent, an online RL framework that trains agents to master interactive environments not merely by solving tasks, but by evolving across episodes. Inspired by human retrospective self-improvement, RetroAgent augments extrinsic rewards with hindsight-generated dual intrinsic feedback: (1) Intrinsic Numerical Feedback, which rewards beneficial exploration by measuring incremental subtask progress relative to prior attempts; and (2) Intrinsic Language Feedback which distills successes and failures into reusable textual lessons for explicit experience reuse. To leverage these lessons effectively, we propose Similarity & Utility-Aware Upper Confidence Bound (SimUtil-UCB), a retrieval strategy that balances semantic relevance, historical utility, and exploration. Across four challenging agentic benchmarks, RetroAgent achieves new state-of-the-art performance, outperforming GRPO by +18.3% on ALFWorld, +15.4% on WebShop, +27.1% on Sokoban, and +8.9% on MineSweeper, while demonstrating strong test-time adaptation and out-of-distribution generalization.

Figures & tables

Figure 1: (a) Overview of RetroAgent . After each episode, the agent reflects on its trajectory to generate dual intrinsic feedback, promoting exploration and facilitating experiential learning. (b) Initialized from Qwen-2.5-7B-Instruct, RetroAgent substantially outperforms the GRPO baseline and achieves new SOTA across four agentic benchmarks.

Figure 2: Overview of the RetroAgent framework. After each episode, a self-reflection mechanism analyzes the trajectory to produce two forms of intrinsic feedback: $(\textup{\it i})$ Intrinsic Numerical Feedback , which quantifies incremental subtask completion relative to prior attempts, rewarding promising exploratory behaviors that may not yet yield task success; and $(\textup{\it ii})$ Intrinsic Language Feedback , which distills actionable lessons from past successes and failures into a memory buffer, retrieved via the proposed SimUtil-UCB strategy to effectively leverage accumulated experiences on similar tasks.

Method	ALFWorld	WebShop		Sokoban	MineSweeper
Method	Success (%)	Score (%)	Success (%)	Success (%)	Success (%)
Qwen-2.5-7B-Instruct (Zero-Shot)	$16.9_{\pm 1.8}$	$4.5_{\pm 1.8}$	$0.8_{\pm 0.0}$	$2.6_{\pm 0.5}$	$6.5_{\pm 1.6}$
Prompting-based Methods
ReAct ∗ ( Yao et al., 2022c )	31.2	46.2	19.5	3.9	7.0
Reflexion ∗ ( Shinn et al., 2023 )	42.7	58.1	28.8	4.3	7.4
Fine-tuning with RL

Table 1: Main results across four benchmarks, averaged over three independent runs (mean

\pm

standard deviation). All improvements are statistically significant with

p<0.01

. Results marked with ∗ are cited from prior work ( Xia et al., 2026 ; Feng et al., 2025 ; Wang et al., 2025b ) . Unless otherwise specified, all training frameworks use the GRPO algorithm. “Success” and “Score” denote Success Rate and Task Score, respectively. w/ Teacher Model indicates methods that require a teacher model for skill induction.

Panel of Figure 3 — Figure 3: Test-time adaptation in an in-distribution (ID) setting on WebShop and an out-of-distribution (OOD) setting on ALFWorld.

Method	Memory Retrieval	WebShop
Method	Memory Retrieval	Discovery@1 (%)	Discovery@2 (%)	Discovery@3 (%)
GRPO (Baseline)	–	$66.9_{\pm 1.2}$	$87.8_{\pm 1.8}$	$97.1_{\pm 0.5}$
RetroAgent (In-Context)	$\times$	$76.8_{\pm 1.6}$	$91.9_{\pm 1.2}$	$98.4_{\pm 0.0}$
RetroAgent (RL-Trained)	$\times$	$77.1_{\pm 1.6}$	$91.7_{\pm 1.2}$	$99.0_{\pm 0.5}$
RetroAgent (In-Context)	$\checkmark$	$78.9_{\pm 3.6}$	$93.0_{\pm 1.4}$	$97.9_{\pm 0.5}$
RetroAgent (RL-Trained)	$\checkmark$	$\mathbf{82.3_{\pm 1.6}}$	$\mathbf{93.0_{\pm 0.8}}$	$\mathbf{99.0_{\pm 0.5}}$

Table 2: Impact of memory retrieval on test-time adaptation.

Panel of Figure 4 — Figure 4: Robustness to challenging tasks on MineSweeper.

Panel of Figure 5 — Figure 5: Accuracy of subtask completion scores generated via single-trajectory (single) vs. pairwise-trajectory (pairwise) induction for Qwen-2.5-7B-Instruct on WebShop.

Method	Hallucination Rate (%)		Estimated Utility Score (%)
	Failure ( $\downarrow$ )	Success ( $\downarrow$ )	Failure			Success
	Failure ( $\downarrow$ )	Success ( $\downarrow$ )	Low ( $\downarrow$ )	Med ( $-$ )	High ( $\uparrow$ )	Low ( $\downarrow$ )	Med ( $-$ )	High ( $\uparrow$ )
Single Induction	8.8	15.1	8.8	78.2	12.9	12.2	75.6	12.2
Pairwise Induction	3.8	11.9	3.1	76.7	20.1	6.2	76.2	17.6

Table 3: Quality of lessons ( i.e., memories) generated via single-trajectory vs. pairwise-trajectory induction, as assessed by GPT-4o.

Method	Augmentation Ratio	WebShop
	Augmentation Ratio	Task Score (%)	Success Rate (%)
GRPO	—	75.5 ±3.6	66.9 ±1.2
+ Single Induction	100% (Full Group)	81.3 ±2.6	70.3 ±2.1
+ Pairwise Induction	100% (Full Group)	82.3 ±1.3	72.9 ±1.6
+ Pairwise Induction	0 50% (Half Group)	$\mathbf{82.4}_{\pm 2.9}$	$\mathbf{75.3}_{\pm 4.3}$

Table 4: Effect of induction method and augmentation ratio on GRPO performance. Augmentation Ratio denotes the fraction of sampled trajectories per prompt that receive memory-augmented generation; the remaining trajectories are sampled without augmentation.

Method	Discounted Returns	Reward Type	WebShop
Method	Discounted Returns	Reward Type	Task Score (%)	Success Rate (%)
GRPO (Baseline)	–	Extrinsic	$75.5_{\pm 3.6}$	$66.9_{\pm 1.2}$
GRPO	✓	Extrinsic	$84.2_{\pm 0.2}$	$74.7_{\pm 2.7}$
+ Progress-Guided Rewards	✓	Extrinsic	$84.2_{\pm 1.7}$	$75.0_{\pm 3.1}$
+ Capability-Evolution Rewards	✓	Extrinsic & Intrinsic	$\mathbf{88.2}_{\pm 2.1}$	$\mathbf{79.7}_{\pm 3.1}$

Table 5: Impact of discounted returns and intrinsic reward shaping on GRPO. Capability-evolution rewards denote the intrinsic numerical feedback described in Section 3.2 .

Panel of Figure 6 — Figure 6: Valid-set performance dynamics on WebShop when augmenting GRPO with intrinsic numerical feedback (a) or intrinsic language feedback (b).

Method	Discounted Returns	Retrieval Strategy	WebShop
Method	Discounted Returns	Retrieval Strategy	Task Score (%)	Success Rate (%)
GRPO (Baseline)	–	–	$75.5_{\pm 3.6}$	$66.9_{\pm 1.2}$
GRPO	✓	–	$84.2_{\pm 0.2}$	$74.7_{\pm 2.7}$
+ Memory Retrieval	✓	Similarity	$79.1_{\pm 7.1}$	$70.1_{\pm 5.5}$
+ Memory Retrieval	✓	Similarity & Utility	$78.4_{\pm 11.4}$	$69.5_{\pm 8.7}$
+ Memory Retrieval	✓	SimUtil-UCB	$\mathbf{86.4\pm 1.8}$	$\mathbf{78.6\pm 1.6}$

Table 6: Impact of intrinsic language feedback on GRPO using different memory-retrieval strategies. SimUtil-UCB denotes the our proposed memory retrieval strategy (Section 3.3 ).

Panel of Figure 7 — Figure 7: Distribution of accumulated memory usage counts across retrieval strategies on WebShop, estimated via kernel density estimation (KDE) ( Chen, 2017 ) . Each panel shows how frequently stored memory instances are accessed under a given strategy.

Method	Intrinsic Feedback	Self-Reflection Mechanism	WebShop
Method	Intrinsic Feedback	Self-Reflection Mechanism	Task Score (%)	Success Rate (%)
GRPO (Baseline)	–	–	$75.5_{\pm 3.6}$	$66.9_{\pm 1.2}$
+ Capability-Evolution Rewards	Numerical	Pairwise Induction	$88.2_{\pm 2.1}$	$79.7_{\pm 3.1}$
+ SimUtil-UCB Memory Retrieval	Language	Pairwise Induction	$86.4_{\pm 1.8}$	$78.6_{\pm 1.6}$
RetroAgent (In-Context)	Dual	Pairwise Induction	$87.6_{\pm 2.1}$	$78.9_{\pm 3.6}$
RetroAgent (RL-Trained)	Dual	Pairwise Induction	$87.0_{\pm 1.4}$	$77.1_{\pm 1.0}$

Table 7: Individual and combined effects of intrinsic numerical and language feedback under different self-reflection mechanisms on WebShop. Rows above the dashed line ablate each feedback type in isolation; rows below combine both (Dual).

Panel of Figure 8 — Figure 8: In-context vs. RL-trained self-reflection mechanisms in RetroAgent on WebShop.

Method	Intrinsic Feedback	Vendi Score ( $\uparrow$ )
Method	Intrinsic Feedback	Successful Traj.	Failed Traj.
Qwen-2.5-7B-Instruct	–	0.00 *	1.89
GRPO (Baseline)	–	1.85	1.71
+ Capability-Evolution Rewards	Numerical	2.04	1.82
+ SimUtil-UCB Memory Retrieval	Language	2.13	1.97
RetroAgent (In-Context Self-Reflection)	Dual	2.01	1.78

Table 8: Impact of intrinsic feedback on trajectory diversity on WebShop, measured by the Vendi Score ( Friedman and Dieng, 2023 ) . A score of 0.00 for Qwen-2.5-7B-Instruct indicates that fewer than two successful trajectories were generated, precluding diversity measurement.

Panel of Figure 9 — Figure 9: Sensitivity of RetroAgent to two key coefficients on the WebShop test set, reported in terms of task score and success rate (averaged over three runs).

Figure 10: Training time (wall-clock hours) on WebShop. “Time to Match GRPO” denotes the time required for each RetroAgent variant to reach the peak performance of the GRPO baseline.

Method	ALFWorld	WebShop		Sokoban	MineSweeper
Method	Success (%)	Score (%)	Success (%)	Success (%)	Success (%)
Llama-3.1-8B-Instruct (Zero-shot)	$29.2_{\pm 0.9}$	$0.2_{\pm 0.4}$	$0.1_{\pm 0.2}$	$5.7_{\pm 0.5}$	$7.0_{\pm 0.8}$
GRPO (Baseline)	$72.7_{\pm 2.3}$	$78.0_{\pm 2.3}$	$67.6_{\pm 2.8}$	$12.2_{\pm 1.2}$	$42.4_{\pm 2.5}$
LaMer ( Jiang et al., 2025 )	$76.0_{\pm 1.8}$	-	$70.3_{\pm 3.6}$	$15.9_{\pm 2.4}$	$32.0_{\pm 3.4}$
GiGPO ( Feng et al., 2025 )	$90.9_{\pm 3.6}$	$87.8_{\pm 2.3}$	$77.7_{\pm 3.9}$	$13.5_{\pm 1.2}$	$48.2_{\pm 2.0}$
RetroAgent (In-Context)	$\mathbf{93.1_{\pm 1.5}}$	$87.8_{\pm 1.8}$	$71.9_{\pm 3.6}$	$\mathbf{39.1_{\pm 1.3}}$	$52.3_{\pm 1.6}$

Table 9: Performance of RetroAgent on Llama-3.1-8B-Instruct across four agentic benchmarks. All improvements are statistically significant (

p<0.01

Panel of Figure 11 — Figure 11: Performance comparison of various methods across different model scales.

Figure 12: Qualitative comparison of RetroAgent (in-context self-reflection) on the WebShop validation set between training step 65 (failed trajectory, left) and training step 150 (successful trajectory, right). For conciseness, only action tokens and their corresponding probabilities are shown at each decision step.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Qwen-2.5-7B-Instruct	Llama-3.1-8B-Instruct	Description
Training Configuration
Training batch size	16	16	Accumulated batch size per update
Validation batch size	128	128	Batch size for validation
Learning rate	$10^{-6}$	$10^{-6}$	Optimizer learning rate
Max prompt length	16 384	16 384	Maximum input context length (tokens)
Max response length	2 048	2 048	Maximum generated response length (tokens)

Appendix

Table 10: Default hyperparameters and training configurations for RetroAgent across all environments.

Panel of Figure 13 — Figure 13: Comparison of intrinsic discrimination capability for Qwen-2.5-7B-Instruct using single vs. pairwise induction, measured by subtask completion scores on WebShop.

Instruction: You are an expert autonomous agent operating in the WebShop e - commerce environment. Your task is to: Find me officially licensed, machine wash men’s t-shirts with polyester heathers, heathers cotton, cotton heather, needle sleeve, classic fit with color: pink, and fit type: youth, and size: small, and price lower than 50.00 dollars.

Past reflections on similar tasks: Action Insight: The search terms used did not match the available products, leading to no suitable options being found. | Navigation Insight: Efficient use of filters could have potentially narrowed down the search, but the specific dimensions and color were not met in any of the results. Warning: These lessons may be outdated. Use them only if they align with your current observation.

Observation 1: Your current observation is: ‘Search’. Your admissible actions of the current situation are: [’search[<your query>]’, ’click[search]’] Now it’s your turn to take one action for the current step. You should first reason step-by-step about the current situation, then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.

RetroAgent : <think> To achieve the goal of finding officially licensed, machine wash men’s t-shirts with specific attributes, I need to construct a search query that includes all the required attributes… I will construct the search query to include all these attributes and then execute the search. </think> Action: search[officially licensed men’s t-shirts polyester heathers heathers cotton cotton heather needle sleeve classic fit color pink fit type youth fit type size small price < 50.00]

Observation 2: You are now at step 2 and your current observation is: ’Back to Search’ [SEP] ’Page 1 (Total results: 50)’ [SEP] ’Next

>

’ [SEP] ’B09GL561XH’ [SEP] ’Disney Mickey And Friends Happy Birthday Mickey Confetti T-Shirt’ [SEP] ’

22.99’ [SEP] ’B07XPR3R7N’ [SEP] ’Marvel Avengers: Endgame Captain America America’s Language T-Shirt’ [SEP] ’

22.99’ … [SEP] ’B09Q8RD8YN’ [SEP] ’Bungo Stray Anime Dogs Anime Character, Long Sleeve, Sweatshirt, Hoodie, T shirt’ [SEP] ’$19.99’.

RetroAgent : <think> The current page does not list any t-shirts that are officially licensed… The best action at this point is to click on the first product listed to see if it is the correct product… </think> Action: click[b09gl561xh]

Observation 3: You are now at step 3… ’fit type’ [SEP] ’men’ [SEP] ’women’ [SEP] ’youth’ [SEP] ’color’ [SEP] ’white’ … [SEP] ’pink’ [SEP] ’size’ [SEP] ’small’ … ’Disney Mickey And Friends Happy Birthday Mickey Confetti T-Shirt’ [SEP] ’Price: $22.99’.

Appendix

Table 11: Successful trajectory generated by RetroAgent (in-context self-reflection) at training step 150 on the WebShop validation set.

Observation 1: Your current observation is: ‘Search’. Your admissible actions of the current situation are: [’search[<your query>]’, ’click[search]’] . Now it’s your turn to take one action for the current step. You should first reason step-by-step about the current situation, then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.

GRPO: <think> To achieve the goal of finding officially licensed, machine wash men’s t-shirts with specific attributes, I need to construct a search query that includes all the required attributes… I will construct the search query to include all these attributes and then execute the search. </think> Action: search[officially licensed men’s t-shirts polyester heathers heathers cotton cotton heather needle sleeve classic fit color pink fit type youth fit type size small price < 50.00]

Observation 2: You are now at step 2 and your current observation is: ’Back to Search’ [SEP] ’Page 1 (Total results: 50)’ [SEP] ’Next >’ [SEP] ’B08HF132PN’ [SEP] ’Flamingo Pink waterbird costume Gift Premium T-Shirt’ [SEP] ’

19.99’ [SEP] ’B07JVVDJ6L’ [SEP] ’Canadian Tuxedo T Shirt - Novelty Denim Tux T-Shirt’ [SEP] ’

19.99’ [SEP] ’B09GL561XH’ [SEP] ’Disney Mickey And Friends Happy Birthday Mickey Confetti T-Shirt’ [SEP] ’

22.99’ … [SEP] ’B07HRFSNL4’ [SEP] ’Nautica Men’s Solid Crew Neck Short-Sleeve Pocket T-Shirt’ [SEP] ’

16.05 to $40.98’.

GRPO: <think> The current observation indicates that the search results are being displayed, and the first result is "B08HF132PN" with the title "Flamingo Pink waterbird costume Gift Premium T-Shirt" and a price of $19.99. However, this product does not match the specific criteria we are looking for, as it is a costume t-shirt rather than a men’s t-shirt, and it is not officially licensed. Given the current search results, the next logical step is to click on the first result to see if it matches the criteria… The most appropriate action here is to click on the first result, "B08HF132PN", to check if it matches the criteria. </think> Action: click[b08hf132pn]

Observation 3: You are now at step 3 and your current observation is: ’Back to Search’ [SEP] ’< Prev’ [SEP] ’fit type’ [SEP] ’men’ [SEP] ’women’ [SEP] ’youth’ [SEP] ’size’ [SEP] ’small’ … ’Flamingo Pink waterbird costume Gift Premium T-Shirt’ [SEP] ’Price: $19.99’ [SEP] ’Rating: N.A.’ [SEP] ’Description’ [SEP] ’Features’ [SEP] ’Reviews’ [SEP] ’Buy Now’.

GRPO: <think> 1. The current observation shows that the search results are being displayed… 2. The product does not match the search criteria… 3. The next step is to refine the search results to ensure that the search criteria are met. The most appropriate action here is to click on "youth" to filter the results to show only products that are intended for youth. </think> Action: click[youth]

Appendix

Table 12: Failed trajectory generated by RetroAgent (in-context self-reflection) at training step 65 on the WebShop validation set.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Retrospective Progress-Aware Self-Refinement for LLM Agent Training
Jun 12, 2026Xinbei Ma, Congmin Zheng, Jiyang Qiu +11Reward Shaping LLM Agent Training
Shanghai Jiao Tong University · OPPO Research Institute
Retrieval-Augmented LLM Agents: Learning to Learn from Experience
Mar 18, 2026Thomas Palmeira Ferraz, Romain Deffayet, Vassilina Nikoulina +2Fine-Tuning LLM Agent Training
NAVER LABS Europe, Meylan, France
Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry
Jul 4, 2026Weiyang Guo, Zesheng Shi, Longhui Zhang +3Reinforcement Learning Credit Assignment in RL
Harbin Institute of Technology, Shenzhen, China

Jun 12, 2026J/K move · Enter open · S save

Retrospective Progress-Aware Self-Refinement for LLM Agent Training

Xinbei Ma, Congmin Zheng, Jiyang Qiu, Jiale Hong+10

Shanghai Jiao Tong University · OPPO Research Institute

Reward Shaping LLM Agent Training Long-Horizon LLM Agents Policy Learning RL for LLM Agents Self-Reflection in Language Model Agents

Open paper AskPDF arXiv

Long-horizon LLM-based agents receive rich environmental observations during interaction, yet outcome rewards provide limited explicit supervision about how individual actions advance task completion. We investigate whether agents can turn this interaction evidence into useful training signals through retrospective progress assessment. A WebShop pilot shows that direct progress prompting reduces task success, whereas hindsight-annotated demonstrations improve it. We introduce RePro, Retrospective Progress-Aware Training, with a forward-then-reflect rollout: the agent estimates progress while acting, then reassesses each step using the completed trajectory and outcome. After warmup with externally generated demonstrations, policy optimization combines self-generated progress differences, online-retrospective alignment, and format rewards with environment feedback, requiring neither a separate process reward model nor ongoing teacher annotation. Experiments on WebShop, ALFWorld, and Sokoban show that RePro enhances the Qwen family's performance, with up to 11.57% success rate gains.

Abstract

Reinforcement Learning

Explore similar work

Retrospective Progress-Aware Self-Refinement for LLM Agent Training

Retrieval-Augmented LLM Agents: Learning to Learn from Experience

Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry