Frontier post-training follows a common recipe: train one expert per domain with reinforcement learning (RL), then consolidate the experts into one model via multi-teacher on-policy distillation (MOPD). Such consolidation is expensive: the student samples in every environment again, an expert scores every sampled token, and the student inherits the drift of the final experts. Yet we argue that the experts' RL runs have already produced what consolidation needs: rollouts with their sampling probabilities and rewards, in the order the experts learned from them. We propose multi-expert replay policy distillation (MRPD), which replays these training logs through the student in the experts' order. At each step, the policy gradient is recomputed on these scored trajectories, and the final experts' distributions are distilled at the logged prefixes. Following the experts' own trajectory reduces the training-inference mismatch and keeps the student within the clip range, so the replay is off-policy in origin but quasi-on-policy throughout consolidation. MRPD samples nothing and calls no environment or reward model: the trainer is the sole process, and consolidating the experts costs only about a seventh of the compute used to train them. On three long-horizon agentic domains (reasoning, search and SQL), MRPD yields the best student model, substantially outperforming online RL baselines, and surpassing MOPD from the same experts at a fraction of MOPD's cost. Our analysis on the same logs shows that the replayed trajectories carry most of the gain, and that both the policy gradient and the distillation term add to it. Most interestingly, the order of the logs matters more than their amount or rewards, suggesting that expert training logs are a curriculum, not a data mixture.
Figures & tables
Figure 1: MRPD replays instead of resampling. MOPD consolidates the experts by resampling and interacting with the environments. MRPD replays the rollouts and rewards from the experts’ own training logs, in the experts’ order, and can be up to 8 × faster.
Figure 2: Overview of expert RL training and off-policy expert consolidation with MRPD. Left: each expert is trained by RL in its own environment; every step’s prompts, rollouts, log-probabilities and rewards are logged (light cells: the model’s tokens; blue cells: tool outputs, excluded from the loss). Right: MRPD replays the logs in the experts’ order, with the policy gradient recomputed on the logged trajectories and the final experts’ distributions distilled at the logged prefixes; no environment, no sampling.
Figure 3: The mean ±1 training rewards of a step’s rollouts: the three experts (dotted) and their average, which is the reward of the batch MRPD replays at that step; online GRPO, online PPO, MOPD and MOPD + GRPO on their own rollouts (576 per step over the three domains). MOPD does not train on the reward.
Figure 4: Cost on eight H100s: the wall-clock time of the full training (left) and the longest single step (right). MOPD and the online RL baselines (PPO, GRPO) sample rollouts at every step; MRPD replays the logs with the full 1536 samples per step, and MRPD ∗ is data-matched to the on-policy methods (576 samples per step). MOPD is timed on the run with task reward (MOPD + GRPO), which samples and runs the experts exactly as MOPD does. PPO’s longest step is a single slow rollout.
Figure 5: Search rollouts cut off at the turn limit in training, as a share of the step’s search trajectories: the search expert, MOPD and MOPD + GRPO (which distill from the final experts), and online GRPO and PPO. The expert reaches the limit at step 150; MOPD’s student is there from step 50 and stays; online RL finds the same behavior at step 200. MRPD samples nothing and has no such curve.
Figure 6: Training-inference mismatch. Left: the mean absolute difference between the student’s and the logged sampling log-probabilities on the training tokens, for the ordered replay, its data-matched variant (576 samples per step, dashed), the same replay with the sampled-token KL, with the PG term alone (offline GRPO), the shuffled replay, and MOPD (whose mismatch stems from the one-step lag in asynchronous training and training-inference engine discrepancies). Right: the clip fraction of the PG term. The ordered replay climbs, then decreases toward MOPD’s level; the shuffled replay stays high on both. The sampled-token term peaks lower but clips more: it passes through the ratio, the dense term does not. The PG-only MRPD stays near MOPD’s level throughout: the KL term is what opens the gap.
Figure 7: Per-domain curves of the replay. Top: the reward of the replayed batch (the expert’s own training reward at that step). Bottom: the student’s KL to the domain’s final expert at the replayed tokens (sampled-token estimate), relative to step 0, under the dense KL (MRPD, solid) and under the sampled-token KL with the PG term (dashed). The SQL expert saturates within about 100 steps. The dense term closes most of the gap within 20 steps on search and SQL, where the sampled-token term is still at a quarter after 100; the gap to the reasoning expert closes last.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Seconds per step on eight H100s over the 300 steps (smoothed); Figure 4 shows their totals and their maxima. The online methods sample at every step and wait for the longest trajectory, which is generated more slowly once the search trajectories exhaust their turns and the reasoning responses lengthen ( Table 7 ); MOPD stands for its run with the task reward and online GRPO for a run with the rewards normalized by the group mean, with the same rollouts. MRPD replays the logs, with their full 1536 samples per step or data-matched to the online runs (576 per step, dashed).
Figure 9: Token entropy of the student during training: MRPD, offline GRPO on the same logs, online GRPO and MOPD. Dashed: the single-domain GRPO experts.
Australian Institute for Machine Learning, Adelaide University, Adelaide, Australia · StepFun, Shanghai, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China +2