Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · Wuhan AI Research
Reinforcement learning with outcome-based objectives such as GRPO enables LLM-based agents to solve complex, long-horizon tasks, yet the reusable exploration patterns embedded in interaction trajectories are largely discarded after a single policy update. Existing experience-augmented approaches retrieve historical guidance at inference time, but they apply experiences without accounting for the policy's evolving capability and create persistent dependencies on external retrieval. We propose EDGE (Experience-Distillation for Guided Exploration), a framework that treats retrieved experiences as temporary training-time scaffolds and progressively internalizes their benefits into the parametric policy. Concretely, EDGE partitions each rollout group into experience-conditioned and experience-free trajectories to estimate and admit only positive marginal gains without extra sampling, then distills the induced behavior into the base policy via a reverse-KL objective on its own empirical support. A co-evolutionary experience bank further synthesizes guidance from emerging failure modes and prunes obsolete entries as the policy evolves. Across embodied, web, and search-based QA tasks, EDGE improves over strong RL baselines by up to 12.5 points and remains effective without inference-time scaffolds or a proprietary reflector. The code is available at https://github.com/xvolcano02/EDGE.
Figures & tables
Figure 1: Overview of the EDGE framework. (a) Experience-guided exploration scaffolding uses contrastive rollouts to estimate the marginal gain Δe of a retrieved experience. (b) Gain-gated distillation internalizes beneficial scaffold behavior into the base policy only when Δe>0 . (c) The experience bank co-evolves with the policy via failure-driven expansion and utility-based pruning, yielding a retrieval-free policy at deployment.
Type
Method
ALFWorld
WebShop
Pick
Look
Clean
Heat
Cool
Pick2
All
Score
Succ.
Closed-Source Model
Prompting
GPT-4o
75.3
60.8
31.2
56.7
21.6
49.8
48.0
31.8
23.7
Prompting
Gemini-2.5-Pro
92.8
63.3
62.1
69.0
26.6
58.7
60.3
42.5
35.9
Qwen2.5-1.5B-Instruct
Prompting
Base Model
5.9
5.5
3.3
9.7
4.2
0.0
4.1
23.1
5.2
Table 1: Performance on ALFWorld and WebShop. We report the average success rate (%) per subtask and overall for ALFWorld, and both the average score and success rate (%) for WebShop. Results of EDGE and other experience-augmented training methods are averaged over 3 random seeds (mean ± std). † denotes results replicated from Xia et al. (2026) and Lu et al. (2026a) .
Method
NQ
TriviaQA
PopQA
HotpotQA
2Wiki
MuSiQue
Bamboogle
Avg.
Qwen3-4B-Instruct-2507
17.8
42.3
27.2
23.6
21.5
5.6
7.4
20.8
Search-R1
40.6
61.2
43.7
35.2
36.3
12.4
41.6
38.7
SkillRL (GPT-4o)
42.4
59.6
44.3
42.1
43.4
15.9
43.2
41.5
EDGE (GPT-4o)
48.0
63.4
46.9
42.4
48.1
17.8
45.6
44.6
EDGE (self)
44.8
64.3
45.4
43.6
45.5
16.2
44.2
43.4
Table 2: Performance (%) on seven search-based QA benchmarks with Qwen3-4B-Instruct-2507. “GPT-4o” and “self” denote the reflector used to synthesize experiences. Best results are bolded.
Figure 2: Performance retention after scaffold removal at inference time. EDGE preserves 96.0% of its scaffolded performance without external experiences, compared with 82.9% for SkillRL and 92.3% for EMPO 2 , indicating more effective internalization into the parametric policy at the 7B scale.
Figure 3: Training dynamics of EDGE vs. GRPO on ALFWorld with Qwen2.5-7B-Instruct. EDGE achieves higher validation success while reducing both environment steps and trajectory length more rapidly, indicating that scaffolded exploration is progressively internalized into more efficient experience-free behavior.
Method
Pick
Look
Clean
Heat
Cool
Pick2
All
GRPO
92.8
85.7
89.3
75.7
74.5
67.7
82.1
EDGE
96.0
85.1
93.4
90.0
92.9
81.6
90.4
w/o ExpPruning
90.5
82.2
90.1
84.3
87.6
76.8
86.7
w/o Distillation
91.2
81.5
91.8
78.7
77.2
71.1
83.6
w/o Gain-Gating
83.1
72.5
79.7
65.3
61.8
61.2
72.3
Table 3: Ablation results on ALFWorld with Qwen2.5-7B-Instruct. We report success rate (%) per subtask and overall.
Figure 4: Experience gain dynamics during training.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Value
RL-Training
Actor learning rate
1e−6
Maximum prompt length
4096
Maximum response length
512
Training batch size
16
Rollout group size ( G )
8
Appendix
Table 4: RL Hyperparameters
Method
Rollout
Old Prob.
Ref. Prob.
Update
Reflector
GRPO
263.41
14.10
14.23
55.38
–
SkillRL
334.13
18.45
18.51
64.74
17.12
EMPO 2
318.69
17.37
17.65
63.17
27.56
EDGE
304.56
17.26
17.67
61.45
16.78
Appendix
Table 5: Per-step wall-clock time (seconds) on ALFWorld with Qwen2.5-7B-Instruct.
Case
Failure mode
Reflected experience
Later evidence
1. Clean bowl
Navigates to destination before finding object
Locate object before placing
Retrieved on a related task at step 155; U=0.125
2. Pillow on sofa
Issues take from source lacking target
Verify source before taking
Utility rises to 0.578 by step 200 (78 retrievals)
3. Pick2 search
Unstructured search before collecting targets
Search systematically before storing
Retrieved on a related Pick2 task at step 166
4. No-op movement
Repeats navigation after arrival
Avoid repeated no-op moves
Near-threshold utility; recovers to U=0.452 by step 200
Appendix
Table 6: Experience-bank co-evolution examples. Each case is extracted from saved failure trajectories, LLM reflection logs, retrieval logs, and utility traces.
Figure 5: Sensitivity of validation success rate to distillation weight λ .
Figure 6: Experience bank evolution: early-stage expansion. Case 1 illustrates destination-first wandering corrected by an object-first scaffold; Case 2 illustrates hallucinated source actions corrected by an observation-gated rule.
Figure 7: Experience bank evolution: late-stage refinement and non-stationary utility. Case 3 illustrates specialized multi-object coordination guidance; Case 4 illustrates an entry whose utility is initially near the pruning threshold but recovers as the policy’s failure distribution shifts.
Figure 8: ALFWorld rollout prompts with and without retrieved experience.
Figure 9: Prompt for updating the ALFWorld state-aware experience bank from contrasted failed and successful trajectories. The model is instructed to generate compact, state-aware experiences that can guide future retrieval and decision making.
Figure 10: WebShop rollout prompts with and without retrieved experience.
Figure 11: Prompt for updating the WebShop shopping state-aware experience bank from contrasted failed and successful trajectories. The model is instructed to generate compact, state-aware experiences that can guide future retrieval and shopping decisions.