cs.LGNov 9, 2026

Which Successes Make Good References? Action-Sequence Alignment for Multi-Turn On-Policy Distillation

Authors: Mufei Li, Nikki Lijing Kuang, Zhuoer Wang, Samson Koelle, Jie Yuan, Manqing Mao, Hong Wang, Yanjun Lin, 

Organizations: Georgia Institute of Technology · Amazon Web Services

Abstract

On-policy distillation (OPD) reduces train-inference distribution mismatch by supervising a student on its own trajectories. For multi-turn tool-use agents, conditioning the teacher on a validated same-task successful trajectory as a reference can provide useful hindsight, but different successes may reach incompatible interaction states and offer misleading supervision. We cast successful-reference selection as a free-target optimal transport problem, which reduces to assigning each failed trajectory its most compatible successful reference. Because latent state paths are hidden, we use action-sequence alignment as an observable proxy for compatibility and instantiate it with longest common subsequence (LCS), which identifies a shared ordered backbone of actions while allowing recoverable detours. This perspective makes trajectory compatibility the central criterion for reference selection in multi-turn OPD. Experiments on ALFWorld and the airline domain of τ3τ^3-bench show that LCS is the only evaluated reference selector to improve over reference-free OPD on both ALFWorld overall scores and all three airline metrics. Without further optimization, the airline-trained models also transfer to the retail domain, where LCS improves all three metrics.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents

    Apr 27, 2026Jiaqi Wang, Wenhao Zhang, Weijie Shi +2Language Model DistillationLLM Agents

  2. UOPD: Uncertainty-Aware Intervention for On-Policy Distillation of Multi-Turn Agents

    Sep 27, 2026Wenbo Zhang, Pengcheng Xu, Weizhi Du +2LLM AgentsUncertainty-Aware Knowledge Distillation

  3. Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation

    Sep 13, 2026Zhiyu Gui, Kexin Huang, Jia Guo +6On-Policy Distillation