Scaling the training of Large Language Model (LLM) agents typically requires a large collection of executable environments, whose construction, maintenance, and operation constrain the expansion of training experience. Existing offline trajectories offer an opportunity to reuse interaction experience, yet effectively refining the current policy without revisiting these environments remains challenging: supervised fine-tuning (SFT) can lead to catastrophic forgetting, while online reinforcement learning requires real-time interaction with the environment. To address this challenge, we propose Step-Advantage Policy Distillation (SAPD), a method for learning from offline trajectories using relative step advantages. SAPD decomposes multi-turn trajectories into individual decision contexts for on-policy distillation, allowing the student to generate responses conditioned on recorded histories, and introduces relative step advantage signals to stabilize training. The entire process requires neither new environment observations nor additional multi-turn interactions. Experiments on ALFWorld, WebShop, and Search-based QA show that SAPD improves performance across different teacher configurations, mitigates catastrophic forgetting compared with SFT, and runs at least 4.7× faster per training step than online OPD. We will release our code soon.
Figures & tables
Figure 1: Overview of our environment-free policy refinement framework.
ALFWorld
Search-based QA
WebShop
Method
Pick
Look
Clean
Heat
Cool
Pick2
Avg.
NQ
Triv
Pop
Hotp
2Wk
MuS
Bam
Avg.
Success
SFT
100.0
100.0
94.1
73.7
88.2
84.2
90.6
32.8
47.4
38.3
29.9
36.5
11.1
28.0
37.0
51.6
GRPO
85.3
53.7
84.5
78.2
59.7
53.5
72.8
34.1
54.5
37.8
32.4
31.9
10.3
26.4
37.7
56.8
GRPO teacher model: Qwen2.5-3B-Instruct
GRPO
96.8
88.2
92.9
91.7
90.0
70.0
89.1
43.4
60.8
46.2
39.3
35.3
14.8
66.9
44.5
64.8
OPD
83.4
60.7
72.7
49.1
42.0
13.3
57.0
11.1
25.6
16.9
8.6
15.4
1.7
12.8
16.1
29.9
Table 1: Main results on ALFWorld, Search-based QA, and WebShop with SFT and GRPO teachers. We report success rates on ALFWorld and WebShop and accuracy on Search-based QA. The best results are bolded, and the second-best results are underlined.
Table 3
Figure 2: Training efficiency and evaluation performance of SAPD and Online OPD.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Training pool
Offline validation pool
Benchmark
Trajectories
Prefixes
Trajectories
Prefixes
ALFWorld
2,382
21,300
123
1,132
WebShop
909
3,384
45
166
Search-based QA
8,146
22,826
428
1,198
Appendix
Table 4: Statistics of the offline trajectory pools. Offline validation pools are distinct from the environment episodes and QA evaluation sets used to measure task performance.
Setting
ALFWorld
WebShop
Search-based QA
Student model
Qwen2.5-1.5B-Instruct
Teacher model
Qwen2.5-3B-Instruct
Teacher training
SFT or GRPO
Teacher updates
Frozen throughout distillation
Learning rate
10−6
10−6
10−6
Trajectories per training batch
16
16
128
Appendix
Table 5: Model settings and default training and evaluation configurations.
Figure 3: Evaluation success rates of SAPD and OPD+Rule during training with SFT teachers.
ALFWorld
WebShop
Method
Pick
Look
Clean
Heat
Cool
Pick2
Avg.
Success
ReOPD
92.2
61.1
93.0
88.0
72.9
25.4
75.0
66.7
SAPD
94.2
60.0
85.5
94.0
74.7
20.4
76.6
70.8
Appendix
Table 6: Performance between SAPD and ReOPD. Avg. reports overall success across all evaluation episodes, giving greater weight to categories with more episodes. Best results are bold.
Figure 4: ALFWorld evaluation performance with a 3B student and a GRPO-trained 3B teacher using offline SFT prefixes.
SFT teacher
GRPO teacher
Scoring scope
ALFWorld
WebShop
Search
ALFWorld
WebShop
Search
Action-only
96.6
59.4
38.8
69.3
53.4
41.3
Full-token
95.9
50.0
38.6
60.8
52.3
41.6
Appendix
Table 7: Comparison of action-only and full-token scoring for step advantages. Both variants use offline SFT prefixes and update the complete response. Results are reported in percentages, with the better result in each setting shown in bold.
On-policy post-training can improve large language models by learning from their own trajectories, but requires costly rollout generation. We ask whether fixed demonstrations can support competitive off-policy learning through better supervision. Our premise is that their usefulness depends not only on the training trajectories, but also on whether supervision provides informative preferences among continuations and connects this guidance to the reasoning decision being learned. We introduce Step-Aligned Privileged Distillation (SAPD), a rollout-free self-distillation method that turns demonstrations into step-aligned distributional supervision. Its key insight is to use the known progression of a reference solution to associate each reasoning transition with targeted privileged guidance, rather than treating the solution as undifferentiated context. On mathematical reasoning benchmarks, SAPD outperforms supervised fine-tuning and label smoothing on average while remaining competitive with on-policy reinforcement learning and self-distillation. Analyses support both the value of context-dependent distributional guidance and the benefit of aligning privileged information with the current step. SAPD also largely preserves out-of-domain coding performance and achieves approximately 2x training-loop speedups over the on-policy baselines. These findings suggest that carefully constructed supervision can make fully off-policy post-training a competitive and computationally efficient alternative. Our code is available at https://github.com/Miaow-Lab/SAPD.
Tianle Wang, Jiayu Liu, Ruizhi Zhao +1
Department of Data Science, City University of Hong Kong · Hong Kong Institute of AI for Science, City University of Hong Kong · Shenzhen University
We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4× faster per rollout than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.
Training multi-turn LLM agents with reinforcement learning typically relies on trajectory-level rewards, which assign a uniform advantage to every step and cannot identify which decisions led to success or failure. Self-distillation methods can provide finer-grained supervision by augmenting RL with privileged information. However, existing approaches usually apply the same type of privileged information to every step in an indistinguishable manner, ignoring a key asymmetry: routine steps need little additional guidance, while critical error steps require corrective direction that environment feedback alone cannot provide. We propose AHEAD, a step-aware framework that matches different supervision sources to different step types. The teacher receives environment feedback on all steps as a grounded dense signal, and additionally receives LLM-generated corrective hints on error steps to supply the direction that environment feedback lacks. The method introduces minimal changes to the standard GRPO algorithm. Across ALFWorld, WebShop, and Search-based QA, and across three model scales, AHEAD raises task success (+13.3 points on ALFWorld and +11.0 on WebShop at 7B over GRPO), reaches a given success rate in fewer training steps, and solves tasks within tighter interaction budgets than outcome-only RL and prior self-distillation baselines.