Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Wuhan AI Research
Reinforcement learning with outcome-based objectives such as GRPO enables LLM-based agents to solve complex, long-horizon tasks, yet the reusable exploration patterns embedded in interaction trajectories are largely discarded after a single policy update. Existing experience-augmented approaches retrieve historical guidance at inference time, but they apply experiences without accounting for the policy's evolving capability and create persistent dependencies on external retrieval. We propose EDGE (Experience-Distillation for Guided Exploration), a framework that treats retrieved experiences as temporary training-time scaffolds and progressively internalizes their benefits into the parametric policy. Concretely, EDGE partitions each rollout group into experience-conditioned and experience-free trajectories to estimate and admit only positive marginal gains without extra sampling, then distills the induced behavior into the base policy via a reverse-KL objective on its own empirical support. A co-evolutionary experience bank further synthesizes guidance from emerging failure modes and prunes obsolete entries as the policy evolves. Across embodied, web, and search-based QA tasks, EDGE improves over strong RL baselines by up to 12.5 points and remains effective without inference-time scaffolds or a proprietary reflector. The code is available at https://github.com/xvolcano02/EDGE.
Figures & tables
Figure 1: Overview of the EDGE framework. (a) Experience-guided exploration scaffolding uses contrastive rollouts to estimate the marginal gain Δe of a retrieved experience. (b) Gain-gated distillation internalizes beneficial scaffold behavior into the base policy only when Δe>0 . (c) The experience bank co-evolves with the policy via failure-driven expansion and utility-based pruning, yielding a retrieval-free policy at deployment.
Type
Method
ALFWorld
WebShop
Pick
Look
Clean
Heat
Cool
Pick2
All
Score
Succ.
Closed-Source Model
Prompting
GPT-4o
75.3
60.8
31.2
56.7
21.6
49.8
48.0
31.8
23.7
Prompting
Gemini-2.5-Pro
92.8
63.3
62.1
69.0
26.6
58.7
60.3
42.5
35.9
Qwen2.5-1.5B-Instruct
Prompting
Base Model
5.9
5.5
3.3
9.7
4.2
0.0
4.1
23.1
5.2
Table 1: Performance on ALFWorld and WebShop. We report the average success rate (%) per subtask and overall for ALFWorld, and both the average score and success rate (%) for WebShop. Results of EDGE and other experience-augmented training methods are averaged over 3 random seeds (mean ± std). † denotes results replicated from Xia et al. (2026) and Lu et al. (2026a) .
Method
NQ
TriviaQA
PopQA
HotpotQA
2Wiki
MuSiQue
Bamboogle
Avg.
Qwen3-4B-Instruct-2507
17.8
42.3
27.2
23.6
21.5
5.6
7.4
20.8
Search-R1
40.6
61.2
43.7
35.2
36.3
12.4
41.6
38.7
SkillRL (GPT-4o)
42.4
59.6
44.3
42.1
43.4
15.9
43.2
41.5
EDGE (GPT-4o)
48.0
63.4
46.9
42.4
48.1
17.8
45.6
44.6
EDGE (self)
44.8
64.3
45.4
43.6
45.5
16.2
44.2
43.4
Table 2: Performance (%) on seven search-based QA benchmarks with Qwen3-4B-Instruct-2507. “GPT-4o” and “self” denote the reflector used to synthesize experiences. Best results are bolded.
Figure 2: Performance retention after scaffold removal at inference time. EDGE preserves 96.0% of its scaffolded performance without external experiences, compared with 82.9% for SkillRL and 92.3% for EMPO 2 , indicating more effective internalization into the parametric policy at the 7B scale.
Figure 3: Training dynamics of EDGE vs. GRPO on ALFWorld with Qwen2.5-7B-Instruct. EDGE achieves higher validation success while reducing both environment steps and trajectory length more rapidly, indicating that scaffolded exploration is progressively internalized into more efficient experience-free behavior.
Method
Pick
Look
Clean
Heat
Cool
Pick2
All
GRPO
92.8
85.7
89.3
75.7
74.5
67.7
82.1
EDGE
96.0
85.1
93.4
90.0
92.9
81.6
90.4
w/o ExpPruning
90.5
82.2
90.1
84.3
87.6
76.8
86.7
w/o Distillation
91.2
81.5
91.8
78.7
77.2
71.1
83.6
w/o Gain-Gating
83.1
72.5
79.7
65.3
61.8
61.2
72.3
Table 3: Ablation results on ALFWorld with Qwen2.5-7B-Instruct. We report success rate (%) per subtask and overall.
Figure 4: Experience gain dynamics during training.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Value
RL-Training
Actor learning rate
1e−6
Maximum prompt length
4096
Maximum response length
512
Training batch size
16
Rollout group size ( G )
8
Appendix
Table 4: RL Hyperparameters
Method
Rollout
Old Prob.
Ref. Prob.
Update
Reflector
GRPO
263.41
14.10
14.23
55.38
–
SkillRL
334.13
18.45
18.51
64.74
17.12
EMPO 2
318.69
17.37
17.65
63.17
27.56
EDGE
304.56
17.26
17.67
61.45
16.78
Appendix
Table 5: Per-step wall-clock time (seconds) on ALFWorld with Qwen2.5-7B-Instruct.
Case
Failure mode
Reflected experience
Later evidence
1. Clean bowl
Navigates to destination before finding object
Locate object before placing
Retrieved on a related task at step 155; U=0.125
2. Pillow on sofa
Issues take from source lacking target
Verify source before taking
Utility rises to 0.578 by step 200 (78 retrievals)
3. Pick2 search
Unstructured search before collecting targets
Search systematically before storing
Retrieved on a related Pick2 task at step 166
4. No-op movement
Repeats navigation after arrival
Avoid repeated no-op moves
Near-threshold utility; recovers to U=0.452 by step 200
Appendix
Table 6: Experience-bank co-evolution examples. Each case is extracted from saved failure trajectories, LLM reflection logs, retrieval logs, and utility traces.
Figure 5: Sensitivity of validation success rate to distillation weight λ .
Figure 6: Experience bank evolution: early-stage expansion. Case 1 illustrates destination-first wandering corrected by an object-first scaffold; Case 2 illustrates hallucinated source actions corrected by an observation-gated rule.
Figure 7: Experience bank evolution: late-stage refinement and non-stationary utility. Case 3 illustrates specialized multi-object coordination guidance; Case 4 illustrates an entry whose utility is initially near the pruning threshold but recovers as the policy’s failure distribution shifts.
Figure 8: ALFWorld rollout prompts with and without retrieved experience.
Figure 9: Prompt for updating the ALFWorld state-aware experience bank from contrasted failed and successful trajectories. The model is instructed to generate compact, state-aware experiences that can guide future retrieval and decision making.
Figure 10: WebShop rollout prompts with and without retrieved experience.
Figure 11: Prompt for updating the WebShop shopping state-aware experience bank from contrasted failed and successful trajectories. The model is instructed to generate compact, state-aware experiences that can guide future retrieval and shopping decisions.
In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization. Existing skill-centric methods improve exploration by optimizing, filtering, or internalizing reusable skills. However, they remain centered on the skills themselves rather than being designed as adaptive training-time support for the evolving policy. To address this, we propose a policy-centric training paradigm that reframes skills as a dynamic training scaffold. Our framework, PATS, converts rollout groups from the latest policy into evidence cards and uses task-specific evaluation to adjust the context used in subsequent rollouts. Concrete guidance helps weak policies to complete challenging tasks. As policy improves, redundant context is revised or removed to reduce reliance on explicit guidance while preserving useful rollout variation. The policy is optimized with environmental rewards using standard RLVR, and the training scaffold is discarded at deployment. Across ALFWorld, WebShop, and seven search-augmented QA benchmarks, PATS achieves performance competitive with SOTA baselines while using 25%-50% fewer tokens.
Yipeng Shi, Zhipeng Ma, Yue Wang +4
School of Software & Microelectronics, Peking University · 2Tencent
Agentic reinforcement learning (RL) for Large Language Models (LLMs) critically depends on the exploration capability of the base policy, as training signals emerge only within its in-capability region. For tasks where the base policy cannot reach reward states, additional training or external guidance is needed to recover effective learning signals. Rather than relying on costly iterative supervised fine tuning (SFT), we exploit the abundant action data generated in everyday human interactions. We propose ActGuide-RL, which injects action data as plan-style reference guidance, enabling the agentic policy to overcome reachability barriers to reward states. Guided and unguided rollouts are then jointly optimized via mixed-policy training, internalizing the exploration gains back into the unguided policy. Motivated by a theoretical and empirical analysis of the benefit-risk trade-off, we adopt a minimal intervention principle that invokes guidance only as an adaptive fallback, matching task difficulty while minimizing off-policy risk. On search-agent benchmarks, ActGuide-RL substantially improves over zero RL (+10.7 pp on GAIA and +19 pp on XBench with Qwen3-4B), and performs on par with the SFT+RL pipeline without any cold start. This suggests a new paradigm for agentic RL that reduces the reliance on heavy SFT data by using scalable action guidance instead.
Yuxiang Ji, Zengbin Wang, Yong Wang +6
1Xiamen University · 2AMAP, Alibaba Group · Work done during internship at AMAP, Alibaba Group +1
Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.