Small language models are attractive backbones for interactive agents, but direct distillation from strong teacher trajectories often turns rich multi-turn behavior into one-shot imitation targets. This is inefficient in long-horizon environments, where early decisions shape later states and rewards. We propose Prefix-GRPO, a reinforcement learning framework that decomposes teacher trajectories into replay-aligned prefix queries and online continuations. Each prefix is replayed in the environment to recover a valid intermediate state, after which the student continues online interaction and receives task reward. Unlike response-only GRPO, Prefix-GRPO also applies clipped policy updates to historical assistant tokens inside the replayed prefix, using a policy-distilled SFT checkpoint to estimate their old log-probabilities. This unifies prefix learning and continuation learning within the same policy-optimization form. Experiments on TextCraft, BabyAI, and ALFWorld show that Prefix-GRPO improves small-model agents over distillation and standard RL baselines, while ablations show that replay alone is insufficient without explicit prefix-token optimization. The implementation and reproduction scripts are available at https://github.com/HappynessI/Prefix_GRPO.
Figures & tables
Figure 1: Overview of Prefix-GRPO. A distilled teacher trajectory is decomposed into prefix queries that can be replayed in the environment; each prefix is replayed to recover a cut state, after which the student continues online interaction. The optimization objective updates both the generated continuation tokens and the historical assistant prefix tokens.
Figure 2: Group-query prefix construction. Prefix-GRPO selects multiple cut points from a teacher trajectory, replays the prefix actions before each cut point, and turns the recovered intermediate states into independent training queries. This construction allows one teacher trajectory to provide several replay-aligned RL starting states rather than a single imitation target.
Method
Depth1
Depth2
Depth3
Depth4
Overall
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Base Model
Base Model
79.03
90.32
21.95
41.46
1.50
8.33
0.00
0.00
33.87
47.47
Distillation
SFT
99.60
100.00
96.65
100.00
53.50
76.00
0.00
0.00
83.87
91.00
RL from Base Model
Table 1: Task-wise Avg@8 and Pass@8 on TextCraft. Bold numbers mark the best Overall Avg@8 and Overall Pass@8 .
Method
GoTo
Pickup
AOD
Find Room
SLoc
Overall
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Base Model
Base Model
65.07
100.00
40.40
73.33
66.74
100.00
35.70
60.00
34.52
80.00
58.75
90.00
Distillation
SFT
91.14
98.18
83.33
100.00
85.00
100.00
56.25
100.00
70.00
80.00
84.44
97.78
RL from Base Model
Table 2: Task-wise Avg@8 and Pass@8 on BabyAI official task families. Bold numbers mark the best Overall Avg@8 and Overall Pass@8 .
Method
Simple Place
Clean Place
Two-Obj Place
Cool Place
Heat Place
Examine
Overall
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Base Model
Base Model
57.61
84.78
19.59
59.46
6.39
28.89
11.16
32.14
13.00
44.00
21.71
63.16
23.56
53.00
Distillation
SFT
63.59
89.13
30.74
72.97
25.00
66.67
33.48
71.43
16.50
56.00
61.84
100.00
38.56
75.50
RL from Base Model
Table 3: Task-wise Avg@8 and Pass@8 on ALFWorld official task families. Bold numbers mark the best Overall Avg@8 and Overall Pass@8 .
Construction
Avg@1
Pass@2
Pass@4
Pass@8
Fixed-Ratio (0.1/0.3/0.5)
75.82
83.02
88.74
93.75
Fixed-Ratio (0.25/0.5/0.7)
80.02
87.32
92.10
94.79
Raw-Entropy Top-3
78.07
85.60
89.20
90.82
Entropy-Change Top-3
81.63
88.18
92.71
96.00
Table 4: Construction-family comparison on TextCraft. All rows use replay-validated prefix data under the current training pipeline.
Method
Avg@1
Pass@2
Pass@4
Pass@8
Replay only, no prefix-token loss
66.25
71.39
75.20
78.79
Prefix-GRPO with SFT old-logprob anchor
81.63
88.18
92.71
96.00
Table 5: Replay-only training versus explicit prefix-token optimization on TextCraft. Both rows use the same data, rollout budget, optimization budget, and evaluation protocol; the replay-only variant excludes prefix tokens from the policy loss.
Variant
Avg@1
Pass@2
Pass@4
Pass@8
Prefix advantage assignment
Constant prefix weight
70.84
80.40
87.00
90.70
Positive continuation signal
76.25
85.00
91.07
93.94
Success-gated prefix update
76.54
83.49
88.53
91.84
Family lift over raw query
74.62
81.54
86.59
89.80
Continuation-magnitude anchor (main)
81.63
88.18
92.71
96.00
Table 6: Prefix-side signal design on TextCraft. Continuation-magnitude anchoring gives the best high-budget recovery, while full-demo query reuse is stronger at lower sampling budgets.
Loss combination
λpre
Avg@1
Pass@2
Pass@4
Pass@8
Joint objective
1.0
74.75
82.43
86.33
89.00
Split objective (main)
1.0
81.63
88.18
92.71
96.00
Split objective
0.5
78.61
86.18
90.96
93.88
Table 7: Prefix/continuation loss-combination ablation on TextCraft.
Construction
Prefix KL
Clip Frac.
Loss Share
Token Frac.
Pass@8
Fixed-Ratio 0.1/0.3/0.5
1.067
0.0163
0.0900
0.341
93.75
Fixed-Ratio 0.25/0.5/0.7
1.242
0.0089
0.0618
0.399
94.79
Raw-Entropy Top-3
1.409
0.0084
0.0518
0.401
90.82
Entropy-Change Top-3
1.319
0.0101
0.0696
0.392
96.00
Table 8: Training-side prefix optimization diagnostics on TextCraft. Metrics are averaged over the last 100 optimizer steps.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
TextCraft
Task Family
GRPO
Prefix
Eval
Depth1
94
1368
31
Depth2
232
3169
41
Depth3
46
638
25
Depth4
1
15
3
All
374
5190
100
Appendix
Table 9: Task-family splits for GRPO training, prefix fullflow training, and evaluation across the three retained environments. The prefix fullflow data contain raw queries and entropy-change top-3 prefix rows.
Construction
Candidates
Validated
Trainable
Val. Rate
Train. Rate
Pass@8
Fixed-Ratio 0.1/0.3/0.5
4488
3620
4017
80.66
89.50
93.75
Fixed-Ratio 0.25/0.5/0.7
4488
3745
4131
83.45
92.05
94.79
Raw-Entropy Top-3
4399
3743
3743
85.09
85.09
90.82
Entropy-Change Top-3
4399
3694
3694
83.98
83.98
96.00
Appendix
Table 10: Replay validation quality and downstream performance for TextCraft prefix construction families. Validation quality is computed during data construction; downstream performance is evaluated after RL training.
Field
Value
Task
task_id=100 , goal: warped planks
Prefix strategy
Entropy-Change Top-3 over interaction assistant tokens ( w=11 )
Canonicalized messages
system, goal, assistant, user observation, assistant, cut-state user observation
Assistant prefix span
[782,946) ; span length =164
Prefix mask / old log-probs
∣mi∣=164 , ∑mi=143 , ∣ℓiold∣=164
Training gather rule
gather only positions with mi[u−bi]=1
Appendix
Table 11: Metadata for the prompt-space mask alignment example.
Token region
Example
Mask behavior
First assistant reasoning/action
Think: The user wants... Action: [[ craft 4 warped planks using 1 warped stems ]]
mask =1
Intervening user observation
Could not find enough items...
mask =0
Second assistant reasoning/action
Think: The player is telling... Action: [[ get 1 warped stems ]]
mask =1
Cut-state observation
Got 1 warped stems
outside the prefix span
Appendix
Table 12: Token-region behavior for the same prompt-space mask alignment example.
Parameter
Value
Parameter
Value
Optimization
GPUs
2
Gradient checkpointing
disabled
Train batch size
16
PPO mini-batch size
16
PPO epochs
2
Micro batch/GPU
8
Learning rate
5×10−6
KL loss
disabled
Rollout
Appendix
Table 13: Main TextCraft Prefix-GRPO training configuration.
Parameter
Value
Parameter
Value
Optimization
GPUs
2
Gradient checkpointing
disabled
Train batch size
16
PPO mini-batch size
16
PPO epochs
2
Micro batch/GPU
8
Learning rate
5×10−6
KL loss
disabled
Rollout
Appendix
Table 14: Main BabyAI Prefix-GRPO training configuration.
Parameter
Value
Parameter
Value
Optimization
GPUs
2
Gradient checkpointing
disabled
Train batch size
16
PPO mini-batch size
16
PPO epochs
2
Micro batch/GPU
8
Learning rate
5×10−6
KL loss
disabled
Rollout
Appendix
Table 15: Main ALFWorld Prefix-GRPO training configuration.
Prefix clip setting
Avg@1
Pass@2
Pass@4
Pass@8
0.1/0.1
78.50
84.50
87.83
89.80
0.2/0.2 (main)
81.63
88.18
92.71
96.00
0.3/0.3
69.00
75.11
80.61
87.88
0.5/0.5
74.12
81.75
85.83
89.00
0.2/0.1
73.50
81.29
87.01
90.62
0.28/0.2
79.75
85.61
89.74
93.00
Appendix
Table 16: Prefix clipping ablation on TextCraft. The main Prefix-GRPO setting uses 0.2/0.2 .
Construction
Val. Rate
Replay Failures
Avg@1
Pass@2
Pass@4
Pass@8
Fixed-Ratio 0.1/0.3/0.5
97.98
196
92.50
94.92
96.17
96.67
Fixed-Ratio 0.25/0.5/0.7
98.07
188
92.50
94.44
95.48
96.67
Raw-Entropy Top-3
97.68
204
90.83
93.13
95.48
97.78
Entropy-Change Top-3
97.70
202
94.31
96.98
98.56
100.00
Appendix
Table 17: Construction-family comparison on BabyAI with replay-validation statistics and downstream RL performance. Validation quality is computed during data construction; downstream performance is evaluated after RL training.
Figure 3: Training dynamics of Prefix-GRPO on TextCraft. Curves show that optimization increases reward and group-level positive-reward rate while reducing rollout length and interaction turns, indicating that the learned policy becomes both more successful and more concise.
Figure 4: TextCraft evaluation diagnostics. Prefix-GRPO achieves near-saturated Pass@8 on shallow recipes, while remaining failures concentrate in deeper recipes and are dominated by resource-state errors rather than invalid action formatting.
Figure 5: Representative TextCraft rollouts from Prefix-GRPO. The successful trajectory completes the task with concise state tracking and a correct terminal craft action, while the failed trajectory shows a resource-recovery loop that reaches the turn limit despite partial progress.
We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4× faster per rollout than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.
Reinforcement learning for multi-turn agents suffers from a credit-assignment mismatch: rewards are sparse and trajectory-level, while success often hinges on a few local decisions. Existing online policy distillation (OPD) provides denser token-level supervision, but typically treats heterogeneous agent trajectories as monolithic strings rather than causal interaction units. We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution. StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts and converting token-level log-probability gaps into sign-preserving advantage shaping with a normalized per-step credit budget before the GRPO update. Across ALFWorld and Search-QA with Qwen3-1.7B and Qwen2.5-3B-Instruct, StepOPSD attains best or second-best results on subsets most sensitive to local causal errors, including first-place performance on ALFWorld Heat (79.1%), PickTwo (95.0%), Search-QA TriviaQA (61.6%), and tied-best performance on HotpotQA (40.4%). The results further reveal a consistent two-knob law: smaller α_clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength λ_mix remains task-dependent. These findings suggest that step-aware distillation is most useful when trajectory-level rewards are weakly aligned with the local action that determines downstream success.
A significant hurdle for current LLMs is the execution of complex, multi-stage tasks. Group Relative Policy Optimization (GRPO) has been emerging as a leading choice, but its reliance on sparse outcome rewards severely limits credit assignment across intermediate steps. Existing remedies such as running full rollouts to assign step-level advantages, calling external LLM judges at each step, or computing intrinsic rewards that require ground-truth answers at every evaluation introduce significant costs or practical constraints. We hypothesize that internal correctness probing over LLM hidden states can be repurposed as a step-level reward signal, potentially addressing all of these limitations at once. However, existing probing research assumes clean inputs, and we first show that this assumption breaks down in multi-step settings: hidden-state probes degrade severely under prefix contamination tracking coherence with the (possibly corrupted) prefix rather than grounded correctness, while attention-based features remain robust to contamination but underperform on clean prefixes. Building on this complementary relationship, we propose the Prefix-Aware Internal Reward (PAIR), a two-stage model with a frozen hidden-state probe estimating belief-consistency and a lightweight attention-based head correcting it toward grounded correctness. Experimental results show that PAIR achieves the highest AUROC on contaminated trajectories while operating at negligible inference cost, enabling dense step-level reward signals for GRPO training without external model calls, ground-truth dependencies, or full-trajectory rollouts.