cs.LGJul 3, 2026

From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation

Authors: Yihan Wang, Zhong Guan, Haoran Sun, Jiale Huang, Likang Wu, Hongke Zhao

Organizations: Tianjin University · Peking University · Tianjin University of Technology

Abstract

Small language models are attractive backbones for interactive agents, but direct distillation from strong teacher trajectories often turns rich multi-turn behavior into one-shot imitation targets. This is inefficient in long-horizon environments, where early decisions shape later states and rewards. We propose Prefix-GRPO, a reinforcement learning framework that decomposes teacher trajectories into replay-aligned prefix queries and online continuations. Each prefix is replayed in the environment to recover a valid intermediate state, after which the student continues online interaction and receives task reward. Unlike response-only GRPO, Prefix-GRPO also applies clipped policy updates to historical assistant tokens inside the replayed prefix, using a policy-distilled SFT checkpoint to estimate their old log-probabilities. This unifies prefix learning and continuation learning within the same policy-optimization form. Experiments on TextCraft, BabyAI, and ALFWorld show that Prefix-GRPO improves small-model agents over distillation and standard RL baselines, while ablations show that replay alone is insufficient without explicit prefix-token optimization. The implementation and reproduction scripts are available at https://github.com/HappynessI/Prefix_GRPO.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Multi-Turn On-Policy Distillation with Prefix Replay

    Jul 6, 2026Baohao Liao, Hanze Dong, Christof Monz +3LLM Agent TrainingOn-Policy Distillation

  2. StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning

    May 26, 2026Yanfei Zhang, Xu Lin, Chenglin WuCredit Assignment in RLStep-Level Credit Assignment in RL

  3. PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization

    May 18, 2026Wonjoong Kim, Yeonjun In, Sangwu Park +2Reward ModelingCredit Assignment in RL