Lexicographic Multi-Objective On-Policy Distillation

Doseok Jang; Jon Ander Campos; Youran Qi

cs.LGOct 1, 2026

Lexicographic Multi-Objective On-Policy Distillation

Authors: Doseok Jang, Jon Ander Campos, Youran Qi

Organizations: Cohere · Mila, Universit´e de Montr´eal

Abstract

Reinforcement learning from verifiable rewards (RLVR) usually optimizes answer correctness, yet useful language-model behavior also requires high-quality reasoning and concise responses. Existing multi-reward post-training methods typically scalarize rewards or combine specialists without explicitly protecting a reward priority order. This is problematic when trade-offs are asymmetric: conciseness, for example, should not improve at the cost of correctness. We introduce Lexicographic Multi-Objective On-Policy Distillation (LMOPD), a multi-teacher method for integrating reward-specialized policies under explicit priorities. For each student rollout, LMOPD selects the specialist for the first objective whose gate detects a deficiency, then locally projects its centered log-policy correction to remove components that oppose higher-priority specialists. We evaluate 30B-A3B mixture-of-experts transformer models in two- and four-expert settings on three math benchmarks, measuring retained specialist gains. With two experts, LMOPD's point estimates fully retain the accuracy and reasoning-quality gains while acquiring $46.9\%$ of the conciseness gain. With four experts, it retains $\approx90\%$ of both the accuracy gain and reasoning-correctness gain, compared to only $\approx57\%$ by the next best evaluated baseline. Matched four-expertablations show that lexicographic routing outperforms random routing and that projection further strengthens both top-priority capabilities. Across both scales, LMOPD preserves the highest-priority capabilities more effectively than the existing baselines we evaluate, demonstrating the value of explicit priorities for specialist integration.

Figures & tables

Panel of Figure 1 — Figure 1: LMOPD overview. (a) We train reward-specialized experts and assign each reward to an expert; one expert may serve multiple rewards. (b) Each student rollout is checked in that order and routed to the first objective whose gate detects a deficiency; a rollout that passes every gate is self-distilled. (c) At each token, LMOPD removes only components of the routed specialist’s correction that oppose earlier specialists, producing a projected virtual-teacher target.

Weight-space merging
Model	pass@1	RQ	Conc	Average
Rewarded Soup (uniform; 1:1)	$61.2\pm 13.7$ %	$61.6\pm 5.0$ %	$67.2\pm 2.3$ %	63.3%
Rewarded Soup (correctness-emphasized; 2:1)	$\underline{72.7\pm 14.0}$ %	$\underline{84.3\pm 5.3}$ %	$44.4\pm 2.1$ %	67.1%
Multi-reward reinforcement learning
Correctness-Gated RLVR (1,1,1)	$70.0\pm 13.8$ %	$40.2\pm 5.0$ %	$49.1\pm 2.2$ %	53.1%
GDPO (1,1,1)	$-457.9\pm 85.2$ %	$-154.4\pm 15.3$ %	$\mathbf{355.8\pm 9.8}$ %	-85.5%

Table 1: Two-expert retained gain. LMOPD has the highest average among unified methods while retaining gains in accuracy and aggregate reasoning quality.

Weight-space merging
Model	Composite pass@1	RQ-corr	RQ-verif	RQ-eff	Composite Conc	Average
Rewarded Soup (uniform; 1:1:1:1)	$31.4\pm 14.3$ %	$51.6\pm 7.5$ %	$-2.3\pm 2.5$ %	$28.7\pm 2.1$ %	$40.2\pm 1.4$ %	29.9%
Rewarded Soup (correctness-emphasized; 2:1:1:1)	$30.5\pm 11.6$ %	$\underline{63.5\pm 10.4}$ %	$\mathbf{4.2\pm 2.9}$ %	$19.2\pm 2.0$ %	$28.5\pm 0.8$ %	29.2%
Multi-reward reinforcement learning
Correctness-Gated RLVR (1,1,1)	$\underline{57.1\pm 9.5}$ %	$55.7\pm 7.7$ %	$\underline{3.2\pm 2.4}$ %	$30.8\pm 4.4$ %	$25.2\pm 1.8$ %	34.4%
GDPO (1,1,1)	$-373.4\pm 55.7$ %	$-265.0\pm 35.7$ %	$-161.5\pm 6.1$ %	$\mathbf{95.1\pm 2.0}$ %	$\mathbf{182.1\pm 3.0}$ %	-104.6%

Table 2: Four-expert retained gain, with objectives ordered by priority. LMOPD leads unified methods on composite pass@1, RQ-corr, and average.

Variant	pass@1	RQ	Conc	Average
LMOPD	$102.9\pm 15.5$ %	$103.3\pm 5.7$ %	$46.9\pm 2.2$ %	84.4%
LMOPD (routing only)	$95.3\pm 12.5$ %	$97.3\pm 4.8$ %	$43.5\pm 1.8$ %	78.7%
LMOPD (random routing)	$30.4\pm 14.0$ %	$46.8\pm 4.2$ %	$72.8\pm 2.0$ %	50.0%

Table 3: Routing and projection ablations. Lexicographic routing has higher performance than random routing; matched four-expert projection shifts retention toward the top priorities.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	pass@1	RQ	Conc
Rewarded Soup (uniform; 1:1)	$0.7311\pm 0.0009$	$0.7501\pm 0.0010$	$0.8038\pm 0.0002$
Rewarded Soup (correctness-emphasized; 2:1)	$0.7349\pm 0.0013$	$0.7618\pm 0.0008$	$0.7973\pm 0.0003$
Correctness-Gated RLVR (1,1,1)	$0.7340\pm 0.0025$	$0.7392\pm 0.0032$	$0.7987\pm 0.0009$
GDPO (1,1,1)	$0.5592\pm 0.0049$	$0.6397\pm 0.0016$	$0.8858\pm 0.0004$
GDPO (5,1,1)	$0.6074\pm 0.0069$	$0.6734\pm 0.0028$	$0.8623\pm 0.0007$
GDPO (25,1,1)	$0.7004\pm 0.0044$	$0.7174\pm 0.0010$	$0.8104\pm 0.0009$

Appendix

Table 4: Raw objective scores for the evaluated configurations and reference policies associated with Tables 1 and 2 , including the intermediate GDPO

(5,1,1)

tuning runs. All objectives are composites over AIME 2024, AIME 2025, and HMMT February 2025. Entries are mean

\pm

SEM over five evaluation-suite replicates. The RQ-Correctness Expert’s RQ subaxes use four replicates. Every Conc column reports raw

S_{\mathrm{conc}}

. Correctness-Gated RLVR and GDPO reuse the same policy checkpoints in panels (a) and (b); panel (b) decomposes aggregate reasoning quality into subaxes only for evaluation.

Training seed	Composite pass@1	Composite RQ	Composite Conc	pass@1 RG	RQ RG	Conc RG	Average RG
Seed 1 (13207799; median, main-table run)	$0.7450\pm 0.0042$	$0.7715\pm 0.0014$	$0.7980\pm 0.0006$	$102.9\%$	$103.3\%$	$46.9\%$	$84.4\%$
Seed 2 (24681357)	$0.7405\pm 0.0032$	$0.7720\pm 0.0016$	$0.7989\pm 0.0002$	$89.4\%$	$104.3\%$	$50.0\%$	$81.2\%$
Seed 3 (98765431)	$0.7507\pm 0.0039$	$0.7760\pm 0.0033$	$0.7997\pm 0.0007$	$120.2\%$	$112.1\%$	$52.8\%$	$95.1\%$
Three-seed mean	0.7454	0.7732	0.7989	104.2%	106.6%	49.9%	86.9%
Between-seed SD	0.0051	0.0025	0.0009	15.4 pt	4.8 pt	3.0 pt	–

Appendix

Table 5: Three-seed replication of full two-expert LMOPD at step 40. Raw objective entries are mean

\pm

SEM; RG denotes retained gain. Seed 1 is the run reported in the main tables and is the median seed by average retained gain. Between-seed SD measures training-run variation across the three trained models.

Figure 2: Reverse-KL distillation loss for the three two-expert LMOPD training runs. Thin lines show per-step loss and thick lines show a five-step moving average. All three runs converge to similar loss values by the fixed step-40 evaluation endpoint.

Figure 3: Contrastive trajectories on Problem 62. We deliberately select completed trajectories that expose distinct run-level outcomes: the Reasoning Quality + Correctness Expert performs an extended search and encounters evidence that its candidate answer is inconsistent, but does not revise it, leading to the wrong answer. The Conciseness Expert reaches an answer more directly, but an early setup error remains unchecked, leading to the same wrong answer. LMOPD abandons an unproductive intermediate path, recovers the correct answer, and validates it through an independent check. In the original ten-rollout trace sets, exact-answer counts are 4/10, 6/10, and 8/10, respectively. Ellipses and bracketed text denote omissions and light editing, respectively.

Explore similar work

Aug 25, 2026cs.LG

On-policy Distillation with Verifiable Reward

Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at https://github.com/LeapLabTHU/OPDVR.

Wenze Lin, Jiale Zhao, Xitai Jiang +5

LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +2

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards On-Policy Distillation RL Post-Training

Ask PDF

Sep 28, 2026cs.LG

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

Reinforcement learning can turn one language model into several specialists, each excellent at a single skill such as mathematics, coding or following instructions, but users need one model with all of these skills. Multi-teacher on-policy distillation (MOPD) merges them by letting the specialists teach one student: the student answers each prompt, and the specialist for that prompt's domain gives feedback on every token. This routing decides which specialist teaches, but not how strongly its feedback moves the shared student. In Qwen3.5 models at three sizes, we find that MOPD's student does not beat one taught by the best single specialist and gains little of the mathematics specialist's advantage. The feedback is unbalanced: instruction-following feedback is several times more spread out than mathematics feedback and dominates the student's updates. We propose Domain-Normalized MOPD (DN-MOPD), which keeps the routing and rescales each domain's feedback by its measured spread. On six public benchmarks, DN-MOPD improves the average score over MOPD at every size, across three random seeds and under two answer-length limits, and recovers most of the lost mathematics gain. Controls with fixed domain weights show that the gain comes mainly from turning down instruction-following feedback rather than turning up mathematics alone, and that fixed weights close to those DN-MOPD measures perform comparably. Combining specialists therefore requires deciding not only which one teaches, but also how strongly its feedback counts.

Xin Li, Hao Jiang, Xin Gao +6

Nanyang Technological University · Yale University · University of Manchester

Language Model Distillation Multi-Teacher Knowledge Distillation On-Policy Distillation Policy Learning Knowledge Distillation

Ask PDF

Sep 3, 2026cs.CL

Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR

Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have emerged as two dominant methods for post-training reasoning LLMs. Prior work uses OPD's dense token-level supervision to complement the sparse RL reward, fusing the two signals within a single step: either as a \emph{weighted-additive combination} or a \emph{teacher-modulated rescaling} of the RL advantage. In this paper, we show that a simple two-stage scheme, OPD-then-RL, consistently outperforms pure OPD, pure RLVR, and all such joint baselines across logic and math reasoning benchmarks. Beyond the empirical results, we further provide a systematic understanding of this through pass@

k

behavior, learning dynamics, and parameter updates, yielding a consistent explanation: OPD expands the student's coverage of teacher-supported solutions and RL sharpens within that support, while jointly optimizing the two signals causes them to interfere. To provide a practical recipe, we find that the OPD validation score is the key signal for when to switch to RL, and that OPD is a better cold start for RL than SFT. Together, our results establish OPD-then-RL as a simple yet strong way to combine the two methods, turning two entangled signals into complementary stages.

Boyan Li, Bingsen Chen, Chenghao Yang +3

University of Alberta · New York University · NYU Shanghai +3

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards On-Policy Distillation RL Post-Training

Ask PDF