Inductive Feedback for Mixed-Policy Distillation

Amir Moeini; Huaijiang Zhu; Daniel Havir; Shangtong Zhang

cs.LGSep 28, 2026

Inductive Feedback for Mixed-Policy Distillation

Authors: Amir Moeini, Huaijiang Zhu, Daniel Havir, Shangtong Zhang

Organizations: Alquist Robotics · University of Virginia

Abstract

Verbal feedback can identify errors and prescribe corrections, providing rich supervision for language-model post-training even when reliable programmatic verifiers are unavailable. Such feedback, often generated by a capable model, can be used to condition the teacher in on-policy distillation, which trains the student to match the teacher's predictions on student-generated rollouts. However, this approach can transfer teacher preferences that the feedback did not motivate, while leaving much of the feedback's guidance unused. We find that both problems come from the standard on-policy distillation objective, specifically the divergence it minimizes and the distribution it uses as its target. Our proposed method addresses both limitations. First, to isolate the information conveyed by the feedback from the teacher's inherent preferences, we treat verbal feedback as evidence for or against the hypothesis that a particular token comes next at a given prefix. We then adopt a probabilistic confirmation framework which uniquely determines an ordering over the vocabulary based on the teacher's predictions before and after it receives feedback. Using a confirmation score consistent with this ordering, we construct a target distribution within a trust region of the student. Second, to learn from guidance that student rollouts can leave unused, we derive a simple shared-rollout estimator of a symmetric divergence between the student and target distributions over rollouts, reusing student and feedback-conditioned teacher rollouts in both directions through importance weighting. Empirical evaluations show that our method outperforms the common on-policy distillation recipe and a recent contrastive variant on knowledge-based and agentic benchmarks.

Figures & tables

Method	Trivia Fantasy	EmbodiedEval	SciKnowEval (L3)				ToolAlpaca
Method	Trivia Fantasy	EmbodiedEval	Chem.	Phys.	Biology	Materials	ToolAlpaca
Base model	$\phantom{0}2.8$ $\pm 1.9$	$57.3$ $\pm 2.0$	$31.9$ $\pm 0.3$	$53.8$ $\pm 0.1$	$22.1$ $\pm 4.1$	$68.4$ $\pm 0.8$	$47.6$ $\pm 1.7$
SDPO	$61.0$ $\pm 7.1$	$73.7$ $\pm 1.0$	$35.9$ $\pm 3.4$	$54.9$ $\pm 1.9$	$27.5$ $\pm 0.7$	$73.2$ $\pm 0.3$	$59.5$ $\pm 0.5$
W2S-OPD	$41.7$ $\pm 7.2$	$60.8$ $\pm 4.2$	$35.0$ $\pm 1.1$	$54.8$ $\pm 0.4$	$27.3$ $\pm 1.9$	$72.1$ $\pm 0.4$	$59.7$ $\pm 2.0$
Ours	$\mathbf{98.3}$ $\pm 2.9$	$\mathbf{82.1}$ $\pm 3.0$	$\mathbf{50.2}$ $\pm 2.9$	$\mathbf{66.9}$ $\pm 1.6$	$\mathbf{28.2}$ $\pm 1.8$	$\mathbf{73.7}$ $\pm 1.5$	$\mathbf{61.9}$ $\pm 2.4$

Table 1: Our method achieves the highest mean score across all benchmarks and SciKnowEval subjects. Scores (%; higher is better) are means with 95% confidence intervals; the highest mean in each column is bold. Trained-model scores are averaged over three checkpoint evaluations; base-model scores are averaged over independent evaluation passes (Section 5.1 ).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Policy pair	Vocabulary size, horizon	Relative gap
Neural, KL calibration $0.02$ nats	$32{,}768,\ 512$	$2.0\%$
Neural, KL calibration $0.5$ nats	$32{,}768,\ 512$	$3.3\%$

Appendix

Table 2: Relative reduction from unskewed Jeffreys for perturbed neural policies. Entries are Monte Carlo estimates using

4{,}096

rollouts per policy, shared prefixes for both objectives, and exact vocabulary sums.

	Trivia Fantasy	EmbodiedEval	SciKnowEval (L3)	ToolAlpaca
Student	Qwen3.5-2B	Qwen3.5-4B	Qwen3.5-4B	Qwen3.5-4B
Training set	20 questions	3,765 scenarios	450–1,890 per subject	4,046 requests
Evaluation set	20 questions	12 suites $\times$ 20 tasks	50 / 210 / 80 / 94 (bio. / chem. / phys. / mat.)	68 requests
Learning rate	$10^{-6}$	$10^{-6}$	$10^{-7}$	$5\times 10^{-7}$
Warmup steps	5	50	50	50
Sequences per update	512	128	128	128

Appendix

Table 3: Settings shared by all methods on each benchmark. All methods use Adam with weight decay

0.01

, gradient-norm clipping at

1

, a constant learning rate after linear warmup, training sampling at temperature

1

and top-

p

0.95

, and Gemini 3.7 Flash at temperature

1

as the feedback judge. A rollout becomes stale once the policy that generated it is more than four weight synchronizations old.

	Ours	SDPO	W2S-OPD
Teacher	frozen initial student	frozen initial student	frozen initial student
Teacher contexts	with and without feedback	with feedback	with and without feedback
Training rollouts	student and teacher, $\lambda=1/2$	student only	student only
Token divergence	$\alpha$ -skew Jeffreys, $\alpha=0.01$	reverse KL	reverse KL
Target	corrected target equation 6 , $\beta=10$	feedback-conditioned teacher	$\widetilde{p}_{\mathrm{W2S}}$ , contrast strength $0.75$ , anchor $\pi_{0}$
Importance weights	prefix weights equation 10 , forward weight clipped at $c_{\mathrm{IS}}=2$	per token, $\min\{2,\pi_{\theta}/\pi_{\mathrm{beh}}\}$	per token, $\min\{2,\pi_{\theta}/\pi_{\mathrm{beh}}\}$

Appendix

Table 4: Method-specific settings.

Explore similar work

Multi-Rollout On-Policy Distillation via Peer Successes and Failures
May 12, 2026Weichen Yu, Xiaomin Li, Yizhou Zhao +8Token-Level Supervision Parallel Rollouts
1Carnegie Mellon University · 2Microsoft · 3Purdue University
Learning from Language Feedback via Variational Policy Distillation
May 14, 2026Yang Li, Erik Nijkamp, Semih Yavuz +1Unsupervised On-Policy Self-Distillation Online-Policy Distillation
Salesforce AI Research
OVD: On-policy Verbal Distillation
Jan 29, 2026Jing Xiong, Hui Shen, Shansan Gong +7Online-Policy Distillation Student-Generated Trajectories
The University of Hong Kong, Hong Kong, China · Nanjing University, Nanjing, China · Huawei Technologies, China

May 12, 2026, cs.LGJ/K move · Enter open · S save

Multi-Rollout On-Policy Distillation via Peer Successes and Failures

Weichen Yu, Xiaomin Li, Yizhou Zhao, Xiaoze Liu+7

1Carnegie Mellon University · 2Microsoft · 3Purdue University

Token-Level Supervision Parallel Rollouts Student-Generated Trajectories

Open paper AskPDF arXiv

Large language models are often post-trained with sparse verifier rewards, which indicate whether a sampled trajectory succeeds but provide limited guidance about where reasoning succeeds or fails. On-policy distillation (OPD) offers denser token-level supervision by training on student-generated trajectories, yet existing methods typically distill each rollout independently and ignore the other attempts sampled for the same prompt. We introduce Multi-Rollout On-Policy Distillation (MOPD), a peer-conditioned distillation framework that uses the student's local rollout group to construct more informative teacher signals. MOPD conditions the teacher on both successful and failed peer rollouts: successes provide positive evidence for valid reasoning patterns, while failures provide structured negative evidence about plausible mistakes to avoid. We study two peer-context constructions: positive peer imitation and contrastive success-failure conditioning. Experiments on competitive programming, mathematical reasoning, scientific question answering, and tool-use benchmarks show that MOPD consistently improves over standard on-policy baselines. Further teacher-signal analysis shows that mixed success-failure contexts better align teacher scores with verifier rewards, indicating that the gains arise from more faithful, instance-adaptive supervision. These results indicate that effective on-policy distillation should exploit the student's multi-rollout trial-and-error behavior rather than treating rollouts as isolated samples.

Abstract

Explore similar work

Multi-Rollout On-Policy Distillation via Peer Successes and Failures

Learning from Language Feedback via Variational Policy Distillation

OVD: On-policy Verbal Distillation