Lexicographic Multi-Objective On-Policy Distillation

Doseok Jang; Jon Ander Campos; Youran Qi

cs.LGOct 1, 2026

Lexicographic Multi-Objective On-Policy Distillation

Authors: Doseok Jang, Jon Ander Campos, Youran Qi

Organizations: Cohere · Mila, Universit´e de Montr´eal

Abstract

Reinforcement learning from verifiable rewards (RLVR) usually optimizes answer correctness, yet useful language-model behavior also requires high-quality reasoning and concise responses. Existing multi-reward post-training methods typically scalarize rewards or combine specialists without explicitly protecting a reward priority order. This is problematic when trade-offs are asymmetric: conciseness, for example, should not improve at the cost of correctness. We introduce Lexicographic Multi-Objective On-Policy Distillation (LMOPD), a multi-teacher method for integrating reward-specialized policies under explicit priorities. For each student rollout, LMOPD selects the specialist for the first objective whose gate detects a deficiency, then locally projects its centered log-policy correction to remove components that oppose higher-priority specialists. We evaluate 30B-A3B mixture-of-experts transformer models in two- and four-expert settings on three math benchmarks, measuring retained specialist gains. With two experts, LMOPD's point estimates fully retain the accuracy and reasoning-quality gains while acquiring $46.9\%$ of the conciseness gain. With four experts, it retains $\approx90\%$ of both the accuracy gain and reasoning-correctness gain, compared to only $\approx57\%$ by the next best evaluated baseline. Matched four-expertablations show that lexicographic routing outperforms random routing and that projection further strengthens both top-priority capabilities. Across both scales, LMOPD preserves the highest-priority capabilities more effectively than the existing baselines we evaluate, demonstrating the value of explicit priorities for specialist integration.

Figures & tables

Panel of Figure 1 — Figure 1: LMOPD overview. (a) We train reward-specialized experts and assign each reward to an expert; one expert may serve multiple rewards. (b) Each student rollout is checked in that order and routed to the first objective whose gate detects a deficiency; a rollout that passes every gate is self-distilled. (c) At each token, LMOPD removes only components of the routed specialist’s correction that oppose earlier specialists, producing a projected virtual-teacher target.

Weight-space merging
Model	pass@1	RQ	Conc	Average
Rewarded Soup (uniform; 1:1)	$61.2\pm 13.7$ %	$61.6\pm 5.0$ %	$67.2\pm 2.3$ %	63.3%
Rewarded Soup (correctness-emphasized; 2:1)	$\underline{72.7\pm 14.0}$ %	$\underline{84.3\pm 5.3}$ %	$44.4\pm 2.1$ %	67.1%
Multi-reward reinforcement learning
Correctness-Gated RLVR (1,1,1)	$70.0\pm 13.8$ %	$40.2\pm 5.0$ %	$49.1\pm 2.2$ %	53.1%
GDPO (1,1,1)	$-457.9\pm 85.2$ %	$-154.4\pm 15.3$ %	$\mathbf{355.8\pm 9.8}$ %	-85.5%

Table 1: Two-expert retained gain. LMOPD has the highest average among unified methods while retaining gains in accuracy and aggregate reasoning quality.

Weight-space merging
Model	Composite pass@1	RQ-corr	RQ-verif	RQ-eff	Composite Conc	Average
Rewarded Soup (uniform; 1:1:1:1)	$31.4\pm 14.3$ %	$51.6\pm 7.5$ %	$-2.3\pm 2.5$ %	$28.7\pm 2.1$ %	$40.2\pm 1.4$ %	29.9%
Rewarded Soup (correctness-emphasized; 2:1:1:1)	$30.5\pm 11.6$ %	$\underline{63.5\pm 10.4}$ %	$\mathbf{4.2\pm 2.9}$ %	$19.2\pm 2.0$ %	$28.5\pm 0.8$ %	29.2%
Multi-reward reinforcement learning
Correctness-Gated RLVR (1,1,1)	$\underline{57.1\pm 9.5}$ %	$55.7\pm 7.7$ %	$\underline{3.2\pm 2.4}$ %	$30.8\pm 4.4$ %	$25.2\pm 1.8$ %	34.4%
GDPO (1,1,1)	$-373.4\pm 55.7$ %	$-265.0\pm 35.7$ %	$-161.5\pm 6.1$ %	$\mathbf{95.1\pm 2.0}$ %	$\mathbf{182.1\pm 3.0}$ %	-104.6%

Table 2: Four-expert retained gain, with objectives ordered by priority. LMOPD leads unified methods on composite pass@1, RQ-corr, and average.

Variant	pass@1	RQ	Conc	Average
LMOPD	$102.9\pm 15.5$ %	$103.3\pm 5.7$ %	$46.9\pm 2.2$ %	84.4%
LMOPD (routing only)	$95.3\pm 12.5$ %	$97.3\pm 4.8$ %	$43.5\pm 1.8$ %	78.7%
LMOPD (random routing)	$30.4\pm 14.0$ %	$46.8\pm 4.2$ %	$72.8\pm 2.0$ %	50.0%

Table 3: Routing and projection ablations. Lexicographic routing has higher performance than random routing; matched four-expert projection shifts retention toward the top priorities.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	pass@1	RQ	Conc
Rewarded Soup (uniform; 1:1)	$0.7311\pm 0.0009$	$0.7501\pm 0.0010$	$0.8038\pm 0.0002$
Rewarded Soup (correctness-emphasized; 2:1)	$0.7349\pm 0.0013$	$0.7618\pm 0.0008$	$0.7973\pm 0.0003$
Correctness-Gated RLVR (1,1,1)	$0.7340\pm 0.0025$	$0.7392\pm 0.0032$	$0.7987\pm 0.0009$
GDPO (1,1,1)	$0.5592\pm 0.0049$	$0.6397\pm 0.0016$	$0.8858\pm 0.0004$
GDPO (5,1,1)	$0.6074\pm 0.0069$	$0.6734\pm 0.0028$	$0.8623\pm 0.0007$
GDPO (25,1,1)	$0.7004\pm 0.0044$	$0.7174\pm 0.0010$	$0.8104\pm 0.0009$

Appendix

Table 4: Raw objective scores for the evaluated configurations and reference policies associated with Tables 1 and 2 , including the intermediate GDPO

(5,1,1)

tuning runs. All objectives are composites over AIME 2024, AIME 2025, and HMMT February 2025. Entries are mean

\pm

SEM over five evaluation-suite replicates. The RQ-Correctness Expert’s RQ subaxes use four replicates. Every Conc column reports raw

S_{\mathrm{conc}}

. Correctness-Gated RLVR and GDPO reuse the same policy checkpoints in panels (a) and (b); panel (b) decomposes aggregate reasoning quality into subaxes only for evaluation.

Training seed	Composite pass@1	Composite RQ	Composite Conc	pass@1 RG	RQ RG	Conc RG	Average RG
Seed 1 (13207799; median, main-table run)	$0.7450\pm 0.0042$	$0.7715\pm 0.0014$	$0.7980\pm 0.0006$	$102.9\%$	$103.3\%$	$46.9\%$	$84.4\%$
Seed 2 (24681357)	$0.7405\pm 0.0032$	$0.7720\pm 0.0016$	$0.7989\pm 0.0002$	$89.4\%$	$104.3\%$	$50.0\%$	$81.2\%$
Seed 3 (98765431)	$0.7507\pm 0.0039$	$0.7760\pm 0.0033$	$0.7997\pm 0.0007$	$120.2\%$	$112.1\%$	$52.8\%$	$95.1\%$
Three-seed mean	0.7454	0.7732	0.7989	104.2%	106.6%	49.9%	86.9%
Between-seed SD	0.0051	0.0025	0.0009	15.4 pt	4.8 pt	3.0 pt	–

Appendix

Table 5: Three-seed replication of full two-expert LMOPD at step 40. Raw objective entries are mean

\pm

SEM; RG denotes retained gain. Seed 1 is the run reported in the main tables and is the median seed by average retained gain. Between-seed SD measures training-run variation across the three trained models.

Figure 2: Reverse-KL distillation loss for the three two-expert LMOPD training runs. Thin lines show per-step loss and thick lines show a five-step moving average. All three runs converge to similar loss values by the fixed step-40 evaluation endpoint.

Figure 3: Contrastive trajectories on Problem 62. We deliberately select completed trajectories that expose distinct run-level outcomes: the Reasoning Quality + Correctness Expert performs an extended search and encounters evidence that its candidate answer is inconsistent, but does not revise it, leading to the wrong answer. The Conciseness Expert reaches an answer more directly, but an early setup error remains unchecked, leading to the same wrong answer. LMOPD abandons an unproductive intermediate path, recovers the correct answer, and validates it through an independent check. In the original ten-rollout trace sets, exact-answer counts are 4/10, 6/10, and 8/10, respectively. Ellipses and bracketed text denote omissions and light editing, respectively.

Explore similar work

On-policy Distillation with Verifiable Reward
Aug 25, 2026Wenze Lin, Jiale Zhao, Xitai Jiang +5RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +2
Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation
Sep 28, 2026Xin Li, Hao Jiang, Xin Gao +6Language Model Distillation Multi-Teacher Knowledge Distillation
Nanyang Technological University · Yale University · University of Manchester
Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
Sep 3, 2026Boyan Li, Bingsen Chen, Chenghao Yang +3RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
University of Alberta · New York University · NYU Shanghai +3

Aug 25, 2026, cs.LGJ/K move · Enter open · S save

On-policy Distillation with Verifiable Reward

Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao+4

LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University+2

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards On-Policy Distillation RL Post-Training

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at https://github.com/LeapLabTHU/OPDVR.

Abstract

Explore similar work

On-policy Distillation with Verifiable Reward

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR