Internalizing Agency from Reflective Experience

Rui Ge; Yichao Fu; Yuyang Qian; Junda Su; Yiming Zhao; Peng Zhao; Hao Zhang

cs.AIMar 17, 2026

Internalizing Agency from Reflective Experience

Authors: Rui Ge, Yichao Fu, Yuyang Qian, Junda Su, Yiming Zhao, Peng Zhao, Hao Zhang

Organizations: University of California San Diego · Shanghai Jiao Tong Univsersity · Nanjing University

Abstract

Large language models are increasingly deployed as autonomous agents that must plan, act, and recover from mistakes through long-horizon interaction with environments that provide rich feedback. However, prevailing outcome-driven post-training methods (e.g., RL with verifiable rewards) primarily optimize final success signals, leaving rich environment feedback underutilized. Consequently, they often lead to distribution sharpening: the policy becomes better at reproducing a narrow set of already-successful behaviors, while failing to improve the feedback-grounded agency needed to expand problem-solving capacity (e.g., Pass@k) in long-horizon settings. To address this, we propose LEAFE (Learning Feedback-Grounded Agency from Reflective Experience), a framework that internalizes recovery agency from reflective experience. Specifically, during exploration, the agent summarizes environment feedback into actionable experience, backtracks to earlier decision points, and explores alternative branches with revised actions. We then distill these experience-guided corrections into the model through supervised fine-tuning, enabling the policy to recover more effectively in future interactions. Across a diverse set of interactive coding and agentic tasks under fixed interaction budgets, LEAFE consistently improves Pass@1 over the base model and achieves higher Pass@k than outcome-driven baselines (GRPO) and experience-based methods such as Early Experience, with gains of up to 14% on Pass@128.

Figures & tables

Figure 1: Internalizing feedback-grounded agency improves model capability (i.e., Pass@K) in long-horizon interaction, while outcome-only training (e.g., GRPO) yields limited gains beyond the base model.

Figure 2: Illustration of the LEAFE framework. Stage 1 : During experience collection, the assistant periodically reviews the current trajectory and identifies a suboptimal round (denoted as red-colored

\tau

). It then produces the actionable experience

e

, which is concatenated with the restored history to facilitate subsequent attempts. Stage 2 : During experience distillation, the model optimizes a joint loss using two datasets: randomly sampled rehearsal pairs to maintain capabilities, and counterfactual pairs (original prompts paired with experience-improved actions) to internalize diverse exploration. For simplicity, we depict one branching event from the rollback exploration tree.

	WebShop				AlfWorld
	Qwen2.5-7B		Llama3.1-8B		Qwen2.5-7B		Llama3.1-8B
Method	Pass@1	Pass@128	Pass@1	Pass@128	Pass@1	Pass@128	Pass@1	Pass@128
Base (No FT)	0.05	5.20	0.00	1.80	26.07	78.57	29.82	74.29
EarlyExp	61.55	84.60	51.13	77.80	72.23	92.86	74.15	95.71
GRPO-RLVR	67.45	85.40	54.95	79.40	69.46	91.43	72.50	94.29
ACE	68.65	86.80	54.35	79.80	66.34	89.63	70.32	92.50

Table 1: Main results on four agent benchmarks. We report Pass@1(%) and Pass@128(%) for two models per dataset. LEAFE consistently outperforms baselines across all tasks in terms of Pass@128.

	Qwen2.5-72B		Llama3-70B
Method	Pass@1	Pass@128	Pass@1	Pass@128
Base (No FT)	10.00	33.94	7.35	24.85
GRPO-RLVR	20.45	36.97	13.64	27.88
LEAFE	17.12	47.88	14.09	33.94

Table 2: Main results on CodeContests with larger backbone models. LEAFE yields substantial improvements in Pass@128 compared with the GRPO-RLVR baseline. Note: EarlyExp and ACE are not reported on CodeContests because it is designed for interactive environments and does not directly apply to code execution feedback without a different implementation.

Model	IS	IR	Ours (Stage 1)
Qwen2.5-32B	48.92	51.48	55.52
Qwen2.5-72B	48.65	49.52	54.30
Llama3-70B	30.20	38.10	42.50

Table 3: CodeContests Pass@128 under different sampling strategies with Independent Sampling (IS), Iterative Refinement (IR), and our stage 1.

Figure 3: Scaling results on different benchmarks. We plot the Pass@

k

success rate as a function of the number of samples

k

. Our method (red) consistently achieves higher efficiency and performance ceilings across all tasks compared to the baselines.

Model	pass@1		pass@128
Model	$\mathcal{L}_{\text{reh}}$	$\mathcal{L}_{\text{cf}}+\mathcal{L}_{\text{reh}}$	$\mathcal{L}_{\text{reh}}$	$\mathcal{L}_{\text{cf}}+\mathcal{L}_{\text{reh}}$
Qwen2.5-7B	27.67	27.88	59.33	62.00
Llama3.1-8B	22.54	22.70	57.33	59.33
Qwen2.5-14B	37.17	36.50	67.33	72.00

Table 4: Experience-to-Policy distillation improves Pass@128.

Method	Qwen2.5-32B	Qwen2.5-72B	Llama3-70B
Base (No-train)	85.45	83.33	78.31
GRPO	81.22 -4.2	81.22 -2.1	74.07 -4.2
Ours	85.45 +0.0	85.13 +1.8	79.63 +1.3

Table 5: OOD Generalization on MBPP (trained on CodeContests). Values are Pass@128 (%). Red superscripts indicate the absolute drop relative to the corresponding base model, while blue superscripts indicate the absolute gain. GRPO suffers clear degradation under distribution shift, whereas LEAFE better preserves OOD performance and can even surpass the base model.

Task	Model	Ours	Ours + Enhancement
ALFWorld	Qwen2.5-7B	67.50 / 94.29	67.86 / 95.71 (w/ EE)
ALFWorld	Llama3.1-8B	71.79 / 96.43	74.46 / 93.57 (w/ EE)
CodeContests	Qwen2.5-32B	9.34 / 34.35	14.09 / 33.94 (w/ RL)
	Qwen2.5-72B	11.15 / 40.00	17.12 / 47.88 (w/ RL)
	Llama3-70B	9.33 / 30.30	14.09 / 33.94 (w/ RL)

Table 6: Ablation study on the impact of auxiliary training target (i.e., EarlyExp(EE) and GRPO(RL)). Results are reported as Pass@

1

/ Pass@

128

(%). While additional training target consistently enhances greedy precision (Pass@

1

), it may occasionally constrain the exploration ceiling (Pass@

128

Figure 4: Main results on WebShop and SciWorld. Bars represent Pass@1 (solid) and Pass@k (hatched) (%). Ours consistently outperforms GRPO and other baselines across different model architectures and scales.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: A example on Sokoban illustrating Stage 1 of LEAFE. Starting from a failed trajectory, the agent reflects on the interaction history, identifies an earlier suboptimal decision(step 3), and generates a compact experience summary for rollback-based revision. The environment is then reset to the selected step, the prior history is replayed(step 1-2), and a new branch is explored under the guidance of the reflected experience. Repeating this failure → reflection → rollback → correction process enables the agent to recover from early mistakes and eventually reach a successful solution.

Dataset	#Training Set	#Test Set	#EarlyExp samples	#LEAFE samples
ALFWorld	3553	140	10k	12k
ScienceWorld	2000	150	6k	12k
WebShop	5000	500	15k	10k
Sokoban	5000	500	10k	10k
CodeContests	5000	165	N/A	10k

Appendix

Table 7: Dataset statistics and approximate numbers of training samples used by auxiliary supervision. Training/test set sizes are exact instance counts, while EarlyExp and LEAFE refer to the average scale (order of magnitude) of distilled samples used for SFT across runs.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 585 papers

ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents
Oct 22, 2026Rumei Li, Xunliang CaiLong-Horizon LLM Agents RL for LLM Agents
Meituan, China
Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry
Jul 4, 2026Weiyang Guo, Zesheng Shi, Longhui Zhang +3Reinforcement Learning Credit Assignment in RL
Harbin Institute of Technology, Shenzhen, China
Agent-Reflex: Teaching Language Agents to Act Reflectively
Oct 13, 2026Guanting Dong, Xinyu Yang, Jiajie Jin +9LLM Agent Training Self-Reflection in Language Model Agents
Gaoling School of Artificial Intelligence, Renmin University of China · National University of Singapore

Oct 22, 2026J/K move · Enter open · S save

ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents

Rumei Li, Xunliang Cai

Meituan, China

Long-Horizon LLM Agents RL for LLM Agents RL for Tool Use Long-Horizon RL Self-Reflection in Language Model Agents Credit Assignment in RL

Open paper AskPDF arXiv

With external tools, large language models (LLMs) can tackle complex tasks through sustained environmental interaction. LLM-based agents are therefore increasingly used to tackle complex long-horizon tasks in real-world environments. However, these tasks involve sequential, interdependent decisions and changes in environmental states. Early execution errors affect subsequent interaction histories and environmental states, and their effects accumulate along the decision chain. To address this challenge, we propose Reflection-Guided Policy Optimization (ReflectPO), a reinforcement learning framework for autonomous error recovery in long-horizon tasks. ReflectPO comprises two core components. The first is Reflection-Guided Sampling. We use branch differences and repeated visits in interaction trajectories to construct hints for error recovery. The second is Tree-Structured Credit Assignment. We organize sampled trajectories into trees and assign action-level credit using parent--child reward differences and recovery gains. Experiments over two long-horizon tool-use benchmarks show that ReflectPO outperforms multiple existing baselines in most domains. It also achieves the best overall performance on average. Further analyses show that ReflectPO effectively improves agents' ability to recognize their own deviations, recover from erroneous states, and continue making progress toward task completion.

Abstract

LLM Agent Training

Explore similar work

ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents

Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry

Agent-Reflex: Teaching Language Agents to Act Reflectively