Small language models are attractive backbones for interactive agents, but direct distillation from strong teacher trajectories often turns rich multi-turn behavior into one-shot imitation targets. This is inefficient in long-horizon environments, where early decisions shape later states and rewards. We propose Prefix-GRPO, a reinforcement learning framework that decomposes teacher trajectories into replay-aligned prefix queries and online continuations. Each prefix is replayed in the environment to recover a valid intermediate state, after which the student continues online interaction and receives task reward. Unlike response-only GRPO, Prefix-GRPO also applies clipped policy updates to historical assistant tokens inside the replayed prefix, using a policy-distilled SFT checkpoint to estimate their old log-probabilities. This unifies prefix learning and continuation learning within the same policy-optimization form. Experiments on TextCraft, BabyAI, and ALFWorld show that Prefix-GRPO improves small-model agents over distillation and standard RL baselines, while ablations show that replay alone is insufficient without explicit prefix-token optimization. The implementation and reproduction scripts are available at https://github.com/HappynessI/Prefix_GRPO.
Figures & tables
Figure 1: Overview of Prefix-GRPO. A distilled teacher trajectory is decomposed into prefix queries that can be replayed in the environment; each prefix is replayed to recover a cut state, after which the student continues online interaction. The optimization objective updates both the generated continuation tokens and the historical assistant prefix tokens.
Figure 2: Group-query prefix construction. Prefix-GRPO selects multiple cut points from a teacher trajectory, replays the prefix actions before each cut point, and turns the recovered intermediate states into independent training queries. This construction allows one teacher trajectory to provide several replay-aligned RL starting states rather than a single imitation target.
Method
Depth1
Depth2
Depth3
Depth4
Overall
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Base Model
Base Model
79.03
90.32
21.95
41.46
1.50
8.33
0.00
0.00
33.87
47.47
Distillation
SFT
99.60
100.00
96.65
100.00
53.50
76.00
0.00
0.00
83.87
91.00
RL from Base Model
Table 1: Task-wise Avg@8 and Pass@8 on TextCraft. Bold numbers mark the best Overall Avg@8 and Overall Pass@8 .
Method
GoTo
Pickup
AOD
Find Room
SLoc
Overall
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Base Model
Base Model
65.07
100.00
40.40
73.33
66.74
100.00
35.70
60.00
34.52
80.00
58.75
90.00
Distillation
SFT
91.14
98.18
83.33
100.00
85.00
100.00
56.25
100.00
70.00
80.00
84.44
97.78
RL from Base Model
Table 2: Task-wise Avg@8 and Pass@8 on BabyAI official task families. Bold numbers mark the best Overall Avg@8 and Overall Pass@8 .
Method
Simple Place
Clean Place
Two-Obj Place
Cool Place
Heat Place
Examine
Overall
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Avg@8
Pass@8
Base Model
Base Model
57.61
84.78
19.59
59.46
6.39
28.89
11.16
32.14
13.00
44.00
21.71
63.16
23.56
53.00
Distillation
SFT
63.59
89.13
30.74
72.97
25.00
66.67
33.48
71.43
16.50
56.00
61.84
100.00
38.56
75.50
RL from Base Model
Table 3: Task-wise Avg@8 and Pass@8 on ALFWorld official task families. Bold numbers mark the best Overall Avg@8 and Overall Pass@8 .
Construction
Avg@1
Pass@2
Pass@4
Pass@8
Fixed-Ratio (0.1/0.3/0.5)
75.82
83.02
88.74
93.75
Fixed-Ratio (0.25/0.5/0.7)
80.02
87.32
92.10
94.79
Raw-Entropy Top-3
78.07
85.60
89.20
90.82
Entropy-Change Top-3
81.63
88.18
92.71
96.00
Table 4: Construction-family comparison on TextCraft. All rows use replay-validated prefix data under the current training pipeline.
Method
Avg@1
Pass@2
Pass@4
Pass@8
Replay only, no prefix-token loss
66.25
71.39
75.20
78.79
Prefix-GRPO with SFT old-logprob anchor
81.63
88.18
92.71
96.00
Table 5: Replay-only training versus explicit prefix-token optimization on TextCraft. Both rows use the same data, rollout budget, optimization budget, and evaluation protocol; the replay-only variant excludes prefix tokens from the policy loss.
Variant
Avg@1
Pass@2
Pass@4
Pass@8
Prefix advantage assignment
Constant prefix weight
70.84
80.40
87.00
90.70
Positive continuation signal
76.25
85.00
91.07
93.94
Success-gated prefix update
76.54
83.49
88.53
91.84
Family lift over raw query
74.62
81.54
86.59
89.80
Continuation-magnitude anchor (main)
81.63
88.18
92.71
96.00
Table 6: Prefix-side signal design on TextCraft. Continuation-magnitude anchoring gives the best high-budget recovery, while full-demo query reuse is stronger at lower sampling budgets.
Loss combination
λpre
Avg@1
Pass@2
Pass@4
Pass@8
Joint objective
1.0
74.75
82.43
86.33
89.00
Split objective (main)
1.0
81.63
88.18
92.71
96.00
Split objective
0.5
78.61
86.18
90.96
93.88
Table 7: Prefix/continuation loss-combination ablation on TextCraft.
Construction
Prefix KL
Clip Frac.
Loss Share
Token Frac.
Pass@8
Fixed-Ratio 0.1/0.3/0.5
1.067
0.0163
0.0900
0.341
93.75
Fixed-Ratio 0.25/0.5/0.7
1.242
0.0089
0.0618
0.399
94.79
Raw-Entropy Top-3
1.409
0.0084
0.0518
0.401
90.82
Entropy-Change Top-3
1.319
0.0101
0.0696
0.392
96.00
Table 8: Training-side prefix optimization diagnostics on TextCraft. Metrics are averaged over the last 100 optimizer steps.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
TextCraft
Task Family
GRPO
Prefix
Eval
Depth1
94
1368
31
Depth2
232
3169
41
Depth3
46
638
25
Depth4
1
15
3
All
374
5190
100
Appendix
Table 9: Task-family splits for GRPO training, prefix fullflow training, and evaluation across the three retained environments. The prefix fullflow data contain raw queries and entropy-change top-3 prefix rows.
Construction
Candidates
Validated
Trainable
Val. Rate
Train. Rate
Pass@8
Fixed-Ratio 0.1/0.3/0.5
4488
3620
4017
80.66
89.50
93.75
Fixed-Ratio 0.25/0.5/0.7
4488
3745
4131
83.45
92.05
94.79
Raw-Entropy Top-3
4399
3743
3743
85.09
85.09
90.82
Entropy-Change Top-3
4399
3694
3694
83.98
83.98
96.00
Appendix
Table 10: Replay validation quality and downstream performance for TextCraft prefix construction families. Validation quality is computed during data construction; downstream performance is evaluated after RL training.
Field
Value
Task
task_id=100 , goal: warped planks
Prefix strategy
Entropy-Change Top-3 over interaction assistant tokens ( w=11 )
Canonicalized messages
system, goal, assistant, user observation, assistant, cut-state user observation
Assistant prefix span
[782,946) ; span length =164
Prefix mask / old log-probs
∣mi∣=164 , ∑mi=143 , ∣ℓiold∣=164
Training gather rule
gather only positions with mi[u−bi]=1
Appendix
Table 11: Metadata for the prompt-space mask alignment example.
Token region
Example
Mask behavior
First assistant reasoning/action
Think: The user wants... Action: [[ craft 4 warped planks using 1 warped stems ]]
mask =1
Intervening user observation
Could not find enough items...
mask =0
Second assistant reasoning/action
Think: The player is telling... Action: [[ get 1 warped stems ]]
mask =1
Cut-state observation
Got 1 warped stems
outside the prefix span
Appendix
Table 12: Token-region behavior for the same prompt-space mask alignment example.
Parameter
Value
Parameter
Value
Optimization
GPUs
2
Gradient checkpointing
disabled
Train batch size
16
PPO mini-batch size
16
PPO epochs
2
Micro batch/GPU
8
Learning rate
5×10−6
KL loss
disabled
Rollout
Appendix
Table 13: Main TextCraft Prefix-GRPO training configuration.
Parameter
Value
Parameter
Value
Optimization
GPUs
2
Gradient checkpointing
disabled
Train batch size
16
PPO mini-batch size
16
PPO epochs
2
Micro batch/GPU
8
Learning rate
5×10−6
KL loss
disabled
Rollout
Appendix
Table 14: Main BabyAI Prefix-GRPO training configuration.
Parameter
Value
Parameter
Value
Optimization
GPUs
2
Gradient checkpointing
disabled
Train batch size
16
PPO mini-batch size
16
PPO epochs
2
Micro batch/GPU
8
Learning rate
5×10−6
KL loss
disabled
Rollout
Appendix
Table 15: Main ALFWorld Prefix-GRPO training configuration.
Prefix clip setting
Avg@1
Pass@2
Pass@4
Pass@8
0.1/0.1
78.50
84.50
87.83
89.80
0.2/0.2 (main)
81.63
88.18
92.71
96.00
0.3/0.3
69.00
75.11
80.61
87.88
0.5/0.5
74.12
81.75
85.83
89.00
0.2/0.1
73.50
81.29
87.01
90.62
0.28/0.2
79.75
85.61
89.74
93.00
Appendix
Table 16: Prefix clipping ablation on TextCraft. The main Prefix-GRPO setting uses 0.2/0.2 .
Construction
Val. Rate
Replay Failures
Avg@1
Pass@2
Pass@4
Pass@8
Fixed-Ratio 0.1/0.3/0.5
97.98
196
92.50
94.92
96.17
96.67
Fixed-Ratio 0.25/0.5/0.7
98.07
188
92.50
94.44
95.48
96.67
Raw-Entropy Top-3
97.68
204
90.83
93.13
95.48
97.78
Entropy-Change Top-3
97.70
202
94.31
96.98
98.56
100.00
Appendix
Table 17: Construction-family comparison on BabyAI with replay-validation statistics and downstream RL performance. Validation quality is computed during data construction; downstream performance is evaluated after RL training.
Figure 3: Training dynamics of Prefix-GRPO on TextCraft. Curves show that optimization increases reward and group-level positive-reward rate while reducing rollout length and interaction turns, indicating that the learned policy becomes both more successful and more concise.
Figure 4: TextCraft evaluation diagnostics. Prefix-GRPO achieves near-saturated Pass@8 on shallow recipes, while remaining failures concentrate in deeper recipes and are dominated by resource-state errors rather than invalid action formatting.
Figure 5: Representative TextCraft rollouts from Prefix-GRPO. The successful trajectory completes the task with concise state tracking and a correct terminal craft action, while the failed trajectory shows a resource-recovery loop that reaches the turn limit despite partial progress.