Which Successes Make Good References? Action-Sequence Alignment for Multi-Turn On-Policy Distillation
Organizations: Georgia Institute of Technology · Amazon Web Services
Abstract
On-policy distillation (OPD) reduces train-inference distribution mismatch by supervising a student on its own trajectories. For multi-turn tool-use agents, conditioning the teacher on a validated same-task successful trajectory as a reference can provide useful hindsight, but different successes may reach incompatible interaction states and offer misleading supervision. We cast successful-reference selection as a free-target optimal transport problem, which reduces to assigning each failed trajectory its most compatible successful reference. Because latent state paths are hidden, we use action-sequence alignment as an observable proxy for compatibility and instantiate it with longest common subsequence (LCS), which identifies a shared ordered backbone of actions while allowing recoverable detours. This perspective makes trajectory compatibility the central criterion for reference selection in multi-turn OPD. Experiments on ALFWorld and the airline domain of -bench show that LCS is the only evaluated reference selector to improve over reference-free OPD on both ALFWorld overall scores and all three airline metrics. Without further optimization, the airline-trained models also transfer to the retail domain, where LCS improves all three metrics.
Figures & tables
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Value |
|---|---|
| OPD training | |
| Learning rate | |
| Learning-rate schedule | Constant |
| Gradient-norm clipping | 1.0 |
| Configured optimizer updates | 200 |
| Global training batch | 64 turns |
| Setting | Value |
|---|---|
| OPD training | |
| Learning rate | |
| Learning-rate schedule | Constant |
| Gradient-norm clipping | 1.0 |
| Configured optimizer updates | 10 |
| Global training batch | 15 turns |
| Benchmark | Evaluation Split | # Tasks |
|---|---|---|
| ALFWorld | Valid Seen | 140 |
| Valid Unseen | 134 | |
| -bench | Airline (Test) | 20 |
| Retail (Test) | 40 |
On-Policy Distillation
On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.
See all 551 papers