Truncate Bad, Upweight Good: BoN-Style Distillation via Rank-Based Classification

Yarin Bar; Yaniv Romano

cs.LGAug 20, 2026

Truncate Bad, Upweight Good: BoN-Style Distillation via Rank-Based Classification

Authors: Yarin Bar, Yaniv Romano

Organizations: Department of Electrical and Computer Engineering Technion–Israel Institute of Technology · Department of Computer Science Technion–Israel Institute of Technology

Abstract

Inference-time selection methods, such as Best-of-N, improve generation by sampling a pool of candidates and selecting the top-ranked completion according to a reward model. Distillation seeks to amortize this procedure into a single policy by replacing raw rewards with in-pool ranks and learning a policy that upweights higher-ranked completions. However, existing rank-based policies typically use smooth full-support reweighting, so low-ranked completions receive less mass but remain in the target support. Although a sharper reweighting reduces lower-tail mass, it also increases reliance on brittle ranking at the top made by a single reward model. We propose TUP: a Truncate-bad, Upweight-good Policy that removes low-ranked completions from the support and reweights only the retained upper tail with a tunable sharpness. TUP admits a closed-form, prompt-independent normalization and can be trained fully offline via binary cross-entropy, using shifted-truncated win-rates as soft labels and distilled-to-reference log-likelihood ratios as logits. Theoretically, under certain assumptions, we show that for any unknown oracle reward, the best monotone rank-reweighting can be matched by a lower-tail truncation rule, providing formal support for removing the lower tail rather than merely downweighting it. Empirically, we show that TUP is competitive with strong offline alignment baselines.

Figures & tables

Panel of Figure 2 — Figure 2: Reward model agreement (left) and the cost–benefit tradeoff of $\lambda$ -truncation (right) on UltraFeedback reference completions. We compare ArmoRM, used as the proxy reward model, against the independent Skywork-v2-Llama reward model. Left: Fraction of completions placed by ArmoRM in the top or bottom $p$ region that Skywork also places in the same region, as a function of $p$ . Right: Probability that $\lambda$ -truncation discards a Skywork-top- $25\%$ completion or retains a Skywork-bottom- $25\%$ completion. The crossover marks where these error probabilities are equal.

Figure 3: Fixed-threshold illustration for the quadratic un-normalized oracle-proxy profile

m_{s}(w)=1-(w-s)^{2}

\lambda=1/2

. The x axis is the proxy-rank location

s

where oracle utility is maximized, and the y axis shows the unnormalized oracle utility

\mathcal{U}_{s}

. The curves compare the best QRPO policy, pure truncation, and the best TUP reweighting at the fixed threshold

\sup_{\beta\in(0,\infty]}\mathcal{U}_{s}(1/2,\beta)

. The vertical dashed line marks

s_{\beta}

, above which finite

\beta

improves over pure truncation, and the dotted line marks

s_{T}

, where pure truncation no longer outperforms the best QRPO curve.

\hat{w}_{r}(x_{i},y_{i,j})=\frac{1+\sum_{\ell\neq j}\mathds{1}[r(x_{i},y_{i,j})\geq r(x_{i},y_{i,\ell})]}{K}\in\left\{\frac{1}{K},\dots,1\right\}

Algorithm 1 TUP offline BoN distillation with BCE

	UltraFeedback eval split			Magpie Air eval split
Method	ArmoRM	Skywork-Llama	Skywork-Qwen	ArmoRM	Skywork-Llama	Skywork-Qwen
Initial	$0.1300{\scriptstyle\pm 0.0008}$	$9.18{\scriptstyle\pm 0.24}$	$3.80{\scriptstyle\pm 0.19}$	$0.1531{\scriptstyle\pm 0.0012}$	$10.14{\scriptstyle\pm 0.30}$	$5.08{\scriptstyle\pm 0.25}$
DPO	$0.1499{\scriptstyle\pm 0.0001}$	$16.16{\scriptstyle\pm 0.06}$	$7.97{\scriptstyle\pm 0.03}$	$0.1727{\scriptstyle\pm 0.0005}$	$16.33{\scriptstyle\pm 0.15}$	$9.89{\scriptstyle\pm 0.05}$
REBEL	$0.1494{\scriptstyle\pm 0.0003}$	$16.37{\scriptstyle\pm 0.14}$	$8.01{\scriptstyle\pm 0.07}$	$0.1709{\scriptstyle\pm 0.0007}$	$17.32{\scriptstyle\pm 0.34}$	$9.94{\scriptstyle\pm 0.15}$
REBEL (random)	$0.1496{\scriptstyle\pm 0.0005}$	$16.84{\scriptstyle\pm 0.14}$	$8.14{\scriptstyle\pm 0.08}$	$0.1718{\scriptstyle\pm 0.0001}$	$15.08{\scriptstyle\pm 0.19}$	$9.27{\scriptstyle\pm 0.06}$
QRPO	$0.1521{\scriptstyle\pm 0.0003}$	$17.20{\scriptstyle\pm 0.29}$	$8.22{\scriptstyle\pm 0.08}$	$0.1748{\scriptstyle\pm 0.0003}$	$16.32{\scriptstyle\pm 0.29}$	$9.07{\scriptstyle\pm 0.18}$

Table 1: In-dataset evaluation for Llama 8B Tülu 3 SFT trained separately on UltraFeedback and Magpie Air. We report the LC reward metric for each reward model.

		Reward model			GPT Judge
Train set	Method	ArmoRM	Skywork-Llama	Skywork-Qwen	LC Win	Win	Len
UltraFeedback	Initial	$0.1448{\scriptstyle\pm 0.0030}$	$13.44{\scriptstyle\pm 1.60}$	$6.65{\scriptstyle\pm 0.89}$	$15.18{\scriptstyle\pm 0.36}$	$8.57{\scriptstyle\pm 0.98}$	1011
	DPO	$0.1619{\scriptstyle\pm 0.0001}$	$21.59{\scriptstyle\pm 0.16}$	$10.91{\scriptstyle\pm 0.08}$	$\mathbf{42.30{\scriptstyle\pm 0.16}}$	$37.08{\scriptstyle\pm 1.70}$	1807
	REBEL	$0.1620{\scriptstyle\pm 0.0002}$	$22.15{\scriptstyle\pm 0.12}$	$11.19{\scriptstyle\pm 0.08}$	$41.81{\scriptstyle\pm 0.12}$	$39.81{\scriptstyle\pm 1.73}$	1938
	REBEL (random)	$0.1625{\scriptstyle\pm 0.0003}$	$22.50{\scriptstyle\pm 0.18}$	$11.31{\scriptstyle\pm 0.05}$	$\mathbf{42.44{\scriptstyle\pm 0.15}}$	$40.93{\scriptstyle\pm 1.73}$	1959
	QRPO	$\mathbf{0.1668{\scriptstyle\pm 0.0000}}$	$20.69{\scriptstyle\pm 0.19}$	$10.23{\scriptstyle\pm 0.13}$	$39.64{\scriptstyle\pm 0.16}$	$\mathbf{53.17{\scriptstyle\pm 1.76}}$	3100

Table 2: AlpacaEval performance for Llama 8B Tülu 3 SFT trained on UltraFeedback (top) and Magpie Air (bottom). We report the LC reward metric for each reward model, and win-rate using gpt-4o judge results; length is measured in characters. Methods marked “random” are trained on a random pair of completions from each pool, whereas unmarked pairwise methods use the best and worst completions. TUP is trained on a single random completion per pool.

	ArmoRM		Skywork-v2-Llama		Skywork-v2-Qwen		GPT Judge
Method	MA	AE	MA	AE	MA	AE	LC Win	Win	Len
Initial	$0.1646{\scriptstyle\pm 0.0006}$	$0.1477{\scriptstyle\pm 0.0002}$	$17.36{\scriptstyle\pm 0.11}$	$16.53{\scriptstyle\pm 0.19}$	$11.57{\scriptstyle\pm 0.07}$	$8.76{\scriptstyle\pm 0.09}$	$21.94{\scriptstyle\pm 0.27}$	$18.76{\scriptstyle\pm 1.38}$	$1736$
DPO	$0.1792{\scriptstyle\pm 0.0008}$	$0.1583{\scriptstyle\pm 0.0010}$	$22.62{\scriptstyle\pm 0.22}$	$21.50{\scriptstyle\pm 0.11}$	$16.94{\scriptstyle\pm 0.19}$	$12.43{\scriptstyle\pm 0.05}$	$31.80{\scriptstyle\pm 0.05}$	$37.14{\scriptstyle\pm 1.70}$	$2860$
REBEL	$0.1732{\scriptstyle\pm 0.0021}$	$0.1462{\scriptstyle\pm 0.0016}$	$20.83{\scriptstyle\pm 0.16}$	$13.58{\scriptstyle\pm 0.19}$	$14.08{\scriptstyle\pm 0.15}$	$7.51{\scriptstyle\pm 0.15}$	$22.11{\scriptstyle\pm 0.02}$	$19.25{\scriptstyle\pm 1.39}$	$8470$
REBEL (random)	$0.1822{\scriptstyle\pm 0.0026}$	$0.1588{\scriptstyle\pm 0.0004}$	$\mathbf{24.64{\scriptstyle\pm 0.23}}$	$\mathbf{21.91{\scriptstyle\pm 0.46}}$	$\mathbf{18.32{\scriptstyle\pm 0.26}}$	$\mathbf{12.67{\scriptstyle\pm 0.18}}$	$34.45{\scriptstyle\pm 0.12}$	$40.12{\scriptstyle\pm 1.73}$	$3013$
QRPO	$0.1720{\scriptstyle\pm 0.0005}$	$0.1565{\scriptstyle\pm 0.0002}$	$21.09{\scriptstyle\pm 0.12}$	$20.29{\scriptstyle\pm 0.41}$	$14.23{\scriptstyle\pm 0.03}$	$11.02{\scriptstyle\pm 0.14}$	$31.97{\scriptstyle\pm 0.19}$	$34.78{\scriptstyle\pm 1.68}$	$2261$

Table 3: LC rewards for Mistral 7B SFT, trained on Magpie Air. Tested on in-dataset test split (MA) and AlpacaEval (AE) evaluated by ArmoRM, Skywork-Llama and Skywork-Qwen reward models, with AlpacaEval also judged by gpt-4o .

Figure 4: Ablation study of TUP’s truncation threshold $\lambda$ and sharpness $\beta$ for Llama 8B Tülu 3 SFT trained on Magpie Air and evaluated on AlpacaEval with Skywork-Llama reward model. TUP uses $K=6$ reference completions per prompt. Cells report the mean and sample standard deviation across three generation seeds. Left: LC rewards; higher is better. Right: Mean response length in tokens.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Optimal finite sharpness in the fixed-threshold quadratic illustration. For

\lambda=1/2

and

m_{s}(w)=1-(w-s)^{2}

, we numerically maximize

\mathcal{U}_{s}(1/2,\beta)

over finite

\beta

for each oracle peak location

s>s_{\beta}

. The dashed vertical line marks

s_{\beta}\approx 0.7284

, below which pure truncation is locally optimal and the optimizer is

\beta=\infty

. Just above

s_{\beta}

, the optimal finite tilt is arbitrarily weak, so

\beta^{\star}(s)

is large; as

s

approaches one, the optimal policy concentrates more strongly near the top proxy ranks, and

\beta^{\star}(s)

decreases.

	ArmoRM		Skywork-v2-Llama		Skywork-v2-Qwen		UF
Method	UF	AE	UF	AE	UF	AE	Len
Initial	$0.1300{\scriptstyle\pm 0.0008}$	$0.1370{\scriptstyle\pm 0.0003}$	$9.15{\scriptstyle\pm 0.18}$	$10.78{\scriptstyle\pm 0.05}$	$3.78{\scriptstyle\pm 0.15}$	$4.97{\scriptstyle\pm 0.03}$	$389.1{\scriptstyle\pm 8.0}$
DPO	$0.1497{\scriptstyle\pm 0.0002}$	$0.1620{\scriptstyle\pm 0.0003}$	$16.27{\scriptstyle\pm 0.10}$	$21.36{\scriptstyle\pm 0.18}$	$8.00{\scriptstyle\pm 0.07}$	$10.87{\scriptstyle\pm 0.07}$	$501.5{\scriptstyle\pm 2.3}$
REBEL	$0.1489{\scriptstyle\pm 0.0002}$	$0.1620{\scriptstyle\pm 0.0002}$	$16.35{\scriptstyle\pm 0.07}$	$22.15{\scriptstyle\pm 0.16}$	$8.00{\scriptstyle\pm 0.06}$	$11.20{\scriptstyle\pm 0.04}$	$516.1{\scriptstyle\pm 3.6}$
REBEL (rand)	$0.1436{\scriptstyle\pm 0.0002}$	$0.1568{\scriptstyle\pm 0.0003}$	$14.28{\scriptstyle\pm 0.10}$	$20.05{\scriptstyle\pm 0.09}$	$6.81{\scriptstyle\pm 0.02}$	$10.18{\scriptstyle\pm 0.03}$	$473.9{\scriptstyle\pm 5.2}$
QRPO	$0.1517{\scriptstyle\pm 0.0007}$	$0.1666{\scriptstyle\pm 0.0001}$	$17.41{\scriptstyle\pm 0.07}$	$21.08{\scriptstyle\pm 0.14}$	$8.24{\scriptstyle\pm 0.01}$	$10.31{\scriptstyle\pm 0.17}$	$730.1{\scriptstyle\pm 4.8}$

Appendix

Table 4: Raw reward for Llama 8B Tülu 3 SFT on UltraFeedback eval split (UF) and AlpacaEval (AE), with reporting token length.

	ArmoRM		Skywork-v2-Llama		Skywork-v2-Qwen		MA
Method	MA	AE	MA	AE	MA	AE	Len
Initial	$0.1530{\scriptstyle\pm 0.0010}$	$0.1370{\scriptstyle\pm 0.0003}$	$10.01{\scriptstyle\pm 0.36}$	$10.78{\scriptstyle\pm 0.05}$	$4.99{\scriptstyle\pm 0.29}$	$4.97{\scriptstyle\pm 0.03}$	$331.6{\scriptstyle\pm 2.4}$
DPO	$0.1800{\scriptstyle\pm 0.0003}$	$0.1581{\scriptstyle\pm 0.0006}$	$19.23{\scriptstyle\pm 0.12}$	$18.82{\scriptstyle\pm 0.19}$	$11.93{\scriptstyle\pm 0.09}$	$9.69{\scriptstyle\pm 0.08}$	$484.5{\scriptstyle\pm 3.2}$
REBEL	$0.1795{\scriptstyle\pm 0.0003}$	$0.1591{\scriptstyle\pm 0.0007}$	$20.69{\scriptstyle\pm 0.31}$	$20.65{\scriptstyle\pm 0.19}$	$12.31{\scriptstyle\pm 0.14}$	$10.37{\scriptstyle\pm 0.12}$	$543.2{\scriptstyle\pm 4.5}$
REBEL (random)	$0.1778{\scriptstyle\pm 0.0002}$	$0.1565{\scriptstyle\pm 0.0003}$	$17.60{\scriptstyle\pm 0.20}$	$17.81{\scriptstyle\pm 0.14}$	$10.88{\scriptstyle\pm 0.14}$	$9.14{\scriptstyle\pm 0.08}$	$479.8{\scriptstyle\pm 4.4}$
QRPO	$0.1772{\scriptstyle\pm 0.0003}$	$0.1578{\scriptstyle\pm 0.0000}$	$18.23{\scriptstyle\pm 0.23}$	$18.99{\scriptstyle\pm 0.09}$	$10.25{\scriptstyle\pm 0.13}$	$9.30{\scriptstyle\pm 0.01}$	$554.0{\scriptstyle\pm 2.2}$

Appendix

Table 5: Raw reward for Llama 8B Tülu 3 SFT on Magpie Air eval split (MA) and AlpacaEval (AE), with reporting token length.

	ArmoRM		Skywork-v2-Llama		Skywork-v2-Qwen		MA
Method	MA	AE	MA	AE	MA	AE	Len
Initial	$0.1628{\scriptstyle\pm 0.0002}$	$0.1478{\scriptstyle\pm 0.0002}$	$15.99{\scriptstyle\pm 0.15}$	$16.55{\scriptstyle\pm 0.20}$	$9.86{\scriptstyle\pm 0.08}$	$8.77{\scriptstyle\pm 0.09}$	$493.1{\scriptstyle\pm 4.1}$
DPO	$0.1743{\scriptstyle\pm 0.0003}$	$0.1530{\scriptstyle\pm 0.0005}$	$22.71{\scriptstyle\pm 0.28}$	$20.64{\scriptstyle\pm 0.23}$	$16.75{\scriptstyle\pm 0.19}$	$11.84{\scriptstyle\pm 0.13}$	$750.1{\scriptstyle\pm 2.7}$
REBEL	$0.1593{\scriptstyle\pm 0.0004}$	$0.1424{\scriptstyle\pm 0.0014}$	$14.27{\scriptstyle\pm 0.20}$	$11.68{\scriptstyle\pm 0.20}$	$10.18{\scriptstyle\pm 0.10}$	$6.50{\scriptstyle\pm 0.13}$	$1375.7{\scriptstyle\pm 6.8}$
REBEL (random)	$0.1717{\scriptstyle\pm 0.0001}$	$0.1539{\scriptstyle\pm 0.0002}$	$\mathbf{23.29{\scriptstyle\pm 0.05}}$	$\mathbf{20.97{\scriptstyle\pm 0.04}}$	$\mathbf{17.19{\scriptstyle\pm 0.10}}$	$\mathbf{12.08{\scriptstyle\pm 0.01}}$	$861.0{\scriptstyle\pm 6.5}$
QRPO	$0.1710{\scriptstyle\pm 0.0006}$	$0.1532{\scriptstyle\pm 0.0002}$	$21.14{\scriptstyle\pm 0.13}$	$19.80{\scriptstyle\pm 0.06}$	$14.28{\scriptstyle\pm 0.03}$	$10.86{\scriptstyle\pm 0.11}$	$629.1{\scriptstyle\pm 2.2}$

Appendix

Table 6: Raw reward for Mistral 7B SFT on Magpie Air eval split (MA) and AlpacaEval (AE), with reporting token length.

Panel of Figure 6 — Figure 6: Additional cross-model agreement rates between ArmoRM and independent Skywork reward models. For each selected fraction $p$ , the plots show the fraction of completions placed by ArmoRM in the top or bottom $p$ region that the Skywork evaluator also places in the corresponding region. Left: UltraFeedback with Skywork-v2-Qwen. Center: Magpie Air with Skywork-v2-Llama. Right: Magpie Air with Skywork-v2-Qwen.

Panel of Figure 7 — Figure 7: Additional cost–benefit curves for $\lambda$ -truncation. Each plot compares the probability that truncation discards a completion ranked in the top $25\%$ by the independent Skywork evaluator with the probability that it retains a completion ranked in the bottom $25\%$ by that evaluator. Left: UltraFeedback with Skywork-v2-Qwen. Center: Magpie Air with Skywork-v2-Llama. Right: Magpie Air with Skywork-v2-Qwen.

Method	ArmoRM	Skywork-Llama	Skywork-Qwen	Prompts retained (of 2415)
DPO	53.1%	55.3%	49.1%	276
REBEL	50.9%	50.2%	45.8%	463
REBEL (random)	64.9%	63.2%	61.8%	238
QRPO	51.4%	56.6%	56.6%	415
QRPO (random)	54.2%	53.1%	46.7%	240
BoNBoN	71.0%	60.7%	56.5%	169

Appendix

Table 7: TUP (with

\lambda=0.5

) win rates against each benchmark method on similar-length AlpacaEval responses for Llama 8B Tülu 3 SFT trained on Magpie Air. For each comparison, we retain response pairs whose lengths differ by at most

10\%

. Values above

50\%

favor TUP and are shown in bold. The final column gives the number of retained pairs out of

2{,}415

Model	Role in this work	Model type	Global Rank	Type Rank
ArmoRM-Llama3-8B-v0.1	Training proxy reward model	Classifier	#34	#26
Skywork-Reward-V2-Llama-3.1-8B	Evaluation reward model	Classifier	#1	#1
Skywork-Reward-V2-Qwen3-8B	Evaluation reward model	Classifier	#6	#3
$\texttt{gpt-4o}^{*}$	GPT judge	Generative	#36	#10

Appendix

Table 8: RewardBench standings (regardless of public availability) for the reward and judge models used in our experiments, retrieved in April 2026. Global Rank denotes the model’s overall position on the RewardBench leaderboard, and Type Rank denotes its position within the corresponding model category. The asterisk marks gpt-4o ; because AlpacaEval uses an API-hosted judge, the exact served model version may not match the RewardBench entry.

Method	Dataset variant	$\beta$	$lr$	Num ref	$\lambda$	Checkpoint
DPO	offpolicy2best	0.01	3e-07	6	-	300
REBEL	offpolicy2best	1e-06	3e-07	6	-	200
REBEL (rand.)	offpolicy2random	1e-06	1e-06	6	-	100
QRPO	offpolicy2best	3e-04	3e-07	3	-	100
QRPO (rand.)	offpolicy2random	3e-04	3e-07	3	-	100
BoNBoN	offpolicy2best	1e-03	1e-06	6	-	476

Appendix

Table 9: Training hyperparameters Llama 8B Tülu 3 SFT on UltraFeedback reported in Table 1 , Table 2 and Table 4 .

Method	Dataset variant	$\beta$	$lr$	Num ref	$\lambda$	Checkpoint
DPO	offpolicy2random	0.01	1e-06	6	-	480
REBEL	offpolicy2best	1e-06	1e-07	6	-	640
REBEL (rand.)	offpolicy2random	1e-04	1e-06	6	-	320
QRPO	offpolicy2best	1e-03	1e-07	1	-	320
QRPO (rand.)	offpolicy2random	1e-03	1e-07	1	-	320
BoNBoN	offpolicy2best	1e-03	1e-06	6	-	764

Appendix

Table 10: Training hyperparameters Llama 8B Tülu 3 SFT on Magpie Air reported in Table 1 , Table 2 and Table 5 .

Method	Dataset variant	$\beta$	$lr$	Num ref	$\lambda$	Checkpoint
DPO	offpolicy2random	0.03	3e-07	6	-	480
REBEL	offpolicy2best	1e-04	1e-07	6	-	480
REBEL (rand.)	offpolicy2random	1e-04	3e-07	6	-	320
QRPO	offpolicy2best	3e-03	1e-07	1	-	764
QRPO (rand.)	offpolicy2random	3e-03	1e-06	1	-	764
BoNBoN	offpolicy2best	1e-03	1e-07	6	-	480

Appendix

Table 11: Training hyperparameters Mistral 7B SFT on Magpie Air reported in Table 3 and Table 6 .

Asset	Role in this work	Reported license or terms
QRPO reference codebase	Experimental framework and baseline implementation.	MIT license.
QRPO UltraFeedback reference datasets offpolicy2best , offpolicy2random	Preprocessed UltraFeedback reference-completion pools and ArmoRM scores used for training and evaluation.	MIT license.
QRPO Magpie Air reference datasets offpolicy2best , offpolicy2random	Preprocessed Magpie Air reference-completion pools and ArmoRM scores used for training and evaluation.	MIT license for the QRPO preprocessed releases; the upstream Magpie Air dataset card does not declare a separate license field.
openbmb/UltraFeedback	Source instruction-tuning dataset underlying the UltraFeedback QRPO setting.	MIT license.
tatsu-lab/alpaca_eval	AlpacaEval prompt set used for GPT-judge evaluation.	CC-BY-NC-4.0 license.
allenai/reward-bench	RewardBench leaderboard context for the reward and judge models in Table 8 .	ODC-BY-1.0 license.

Appendix

Table 12: External assets used in the experiments and their reported licenses or terms. For the QRPO reference data, the table lists the released preprocessed datasets actually loaded by our experiments; these are derived reference-completion pools and rewards rather than newly collected data in this work.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 514 papers

Aug 12, 2026

Tail-Aware Top- $k$ On-Policy Distillation

On-policy distillation (OPD) has emerged as an effective paradigm for transferring knowledge between language models, where a student is trained to align its next-token distribution with the teacher's along its own trajectories. To provide dense supervision at tractable cost, many works minimize the reverse Kullback-Leibler (KL) divergence between the student and teacher's normalized distributions over the teacher's top-

k

tokens. However, this normalized objective discards the information about tail probability: the total probability outside the teacher's top-

k

tokens. As a result, the optimization can steadily increase the student's tail probability and entropy, empirically degrading downstream accuracy. To address this issue, we propose Tail-Aware Top-

k

OPD (TA-OPD), a novel distillation method that restores the missing tail probability signal. In particular, TA-OPD minimizes the reverse KL divergence over the top-

k

tokens plus a tail token that carries the tail probability. In effect, TA-OPD better aligns the student's next-token distribution with the teacher's, preventing the increase in tail probability and entropy caused by top-

k

normalization. Extensive experiments demonstrate the superiority of TA-OPD, improving Avg@8 by up to 8.05 points on common benchmarks. Our code is available at https://github.com/HuipengHuang/TA-OPD.

Huipeng Huang, Hongxin Wei

Department of Statistics and Data Science, Southern University of Science and Technology

Language Model Distillation On-Policy Distillation Knowledge Distillation

Ask PDF

Sep 28, 2026

Reward-Aligned Reweighting for On-Policy Distillation

On-policy distillation (OPD) trains a student language model with dense feedback from a stronger teacher on student-generated trajectories. Yet standard OPD weights token-level distillation terms uniformly, implicitly treating local teacher preference as a proxy for correction utility. A decision's task value, however, depends on how the student completes the subsequent reasoning. This mismatch can cause imitation to suppress viable student strategies or reinforce paths the student cannot reliably execute. Verified trajectory outcomes provide complementary evidence about continuation quality, but do not directly identify the utility of individual decisions. We introduce Reward-Aligned Reweighting for On-Policy Distillation (R

^{2}

-OPD), which uses outcome agreement and the magnitude of teacher--student disagreement to continuously reallocate teacher supervision. It gives reward-aligned corrections greater relative influence while retaining dense feedback, moving beyond uniform imitation and hard filtering. Our analysis formalizes the mismatch between local teacher preference and student continuation value and establishes sufficient conditions for reallocation to improve first-order task progress over uniform OPD. Across seven mathematical reasoning benchmarks, R

^{2}

-OPD achieves the highest average accuracy among the compared training methods in both cross-size and same-size distillation. It outperforms standard OPD on all seven benchmarks, with average gains of 3.5 and 2.4 percentage points for 1.7B and 4B students, respectively. An extension to code generation yields an average gain of 1.6 percentage points over standard OPD. These results highlight outcome-guided supervision allocation as an effective way to translate dense teacher feedback into stronger student performance across model scales and task domains.

Haofeng Xu, Junwei Su, Lansong Diao +2

The University of Hong Kong · Alibaba Group · University of Science and Technology of China

RL for Knowledge Distillation Language Model Distillation On-Policy Distillation Knowledge Distillation

Ask PDF

Sep 1, 2026

On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers

Compact instruction-following rerankers are attractive for deployment, but conventional distillation pipelines typically train students by offline imitation of teacher outputs on a fixed set of examples, constraining supervision to the teacher's observed ranking space. We revisit reranker distillation through the lens of reinforcement learning. We propose a two-stage framework combining off-policy teacher optimization with on-policy student distillation. In Stage 1, a 4B teacher reranker is strengthened with off-policy GRPO using LLM-judge feedback on 88K instruction-following examples. In Stage 2, a compact 1B student samples rankings from its own policy and receives soft teacher-derived rewards on those rankings, coupling student exploration with knowledge transfer. Our strongest gains appear under distribution shift. On MAIR-11, the original 11-subset, 869-query evaluation, the proposed student reaches 0.7670 nDCG@6, outperforming offline listwise KD by +4.6 points. Controlled comparisons against offline pairwise RankNet KD and on-policy GKD show that neither changing the offline distillation objective nor moving teacher-distribution matching on-policy reproduces the performance of reward-based on-policy distillation over student-sampled rankings. The advantage persists on MAIR-Full: across all 126 tasks and 9,356 queries, the proposed method obtains the highest task-macro point estimates among the evaluated distillation variants, reaching 0.6808 nDCG@6 and 0.7865 MRR@6. It also exceeds two released 7B RL-trained rerankers on the comparable MAIR-11 evaluation, while the same Stage 2 training procedure consistently improves three architecturally distinct alternative student backbones. On the 9,861-query validation benchmark, the resulting 1B reranker achieves 0.7624 nDCG@6 while providing a favorable quality-efficiency tradeoff relative to larger alternatives.

Vignesh Prabhakar, Jialing Pan, Anil Babu Ankisettipalli

SAP Labs, Palo Alto, CA, USA

Reinforcement Learning Learning to Rank LLM Reranking On-Policy Distillation

Ask PDF

Abstract

Language Model Distillation

Explore similar work

Tail-Aware Top-kkk On-Policy Distillation

Reward-Aligned Reweighting for On-Policy Distillation

On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers

Tail-Aware Top- $k$ On-Policy Distillation