RM-OPD: On-Policy Distillation for Generative Reward Models

Qinglin Su; Licheng Pan; Zhen Fang; Jiakang Wang; Qingnan Ren; Yunsheng Lu; Yan Gao; Zhixuan Chu

cs.LGOct 14, 2026

RM-OPD: On-Policy Distillation for Generative Reward Models

Authors: Qinglin Su, Licheng Pan, Zhen Fang, Jiakang Wang, Qingnan Ren, Yunsheng Lu, Yan Gao, Zhixuan Chu

Organizations: Zhejiang University · Xiaohongshu Inc. · University of Science and Technology of China

Abstract

Reinforcement Learning on Mixed-domain preference data (Mix-RL) has become the mainstream approach for training general-purpose Generative Reward Models (GRMs). However, Mix-RL suffers from two critical bottlenecks: the reward sparsity induced by outcome rewards, and the cross-domain interference arising from jointly optimizing heterogeneous capabilities, which together give rise to imbalanced capability acquisition and pervasive reward hacking. Inspired by the success of On-Policy Distillation (OPD) in multi-domain capability integration, we introduce RM-OPD (Reward Modeling via On-Policy Distillation), a two-stage framework that separates specialist acquisition from capability integration. First, we cultivate domain-specialized teacher models via single-domain GRPO fine-tuning, allowing each expert to reach its performance ceiling in isolation; then we consolidate these teachers into one student model through (i) routed on-policy distillation, which provides dense signed token-level supervision on student-generated critiques; (ii) domain contribution reweighting, which reweights token guidance toward target domain shares, preventing specific domains from dominating optimization; and (iii) optional criteria context distillation, which conditions teachers on domain-specific evaluation criteria to sharpen token-level supervision. Across multiple RM benchmarks and two base models, RM-OPD consistently outperforms vanilla Mix-RL and other strong baselines, achieving overall superior performance and exhibiting an emergent "teacher-surpassing" effect. These results establish RM-OPD as a promising paradigm for building generalist GRMs.

Figures & tables

Figure 1: Performance Comparison in Multi-domain Training. (a) On RM-Bench Math, RM-OPD converges faster than both the RL teacher and Mix-RL. In contrast, Mix-RL converges more slowly and saturates at a noticeably lower performance level. (b) Across Safety, Math, and Code, RM-OPD consistently outperforms Mix-RL while closely matching or even exceeding the corresponding specialist teachers, demonstrating that RM-OPD effectively consolidates domain-specific capabilities into a single model without the performance trade-offs of Mix-RL.

Model	Safety		Math		Code
Base	71.89		86.58		86.79
$+$ Math	68.92	( $\downarrow$ 2.97)	96.42		91.57
$+$ Code	67.97	( $\downarrow$ 0.95)	93.96	( $\downarrow$ 2.46)	93.39
$+$ Safety	82.16		92.62	( $\downarrow$ 1.34)	93.80

Table 1: Mix-RL trade-offs. RewardBench accuracy (%). Red values denote drops. Base model is DeepSeek-R1-Distill-Qwen-7B.

Figure 2: Overview of RM-OPD . Independent single-domain GRPO produces frozen math, code, and safety reward teachers. Each student-generated judgment receives token-level supervision from its domain teacher, conditioned on evaluation criteria. Domain contribution reweighting assigns target coefficients to domain token-mean losses within each micro-batch. The teachers jointly supervise one student, which performs multi-domain judgment at deployment.

Base Models (LLM-as-a-Judge)
Model	RM-Bench $\uparrow$				RewardBench $\uparrow$				RewardBench 2 $\uparrow$			JudgeBench $\uparrow$
	Safety	Math	Code	Overall	Safety	Math	Code	Overall	Safety	Math	Overall	Math	Code	Overall
Qwen2.5-7B-Instruct	85.13	59.95	50.24	65.11	82.57	62.64	86.38	77.20	69.78	39.89	54.84	55.56	60.27	57.92
DeepSeek-R1-Distill-Qwen-7B	78.28	82.50	56.73	72.50	71.89	86.58	86.79	81.75	32.67	57.92	45.30	67.78	49.32	58.55
Existing Generative Reward Models
JudgeLRM-7B	87.23	60.45	51.75	66.48	80.41	64.43	80.08	74.97	68.67	44.26	56.47	67.78	60.27	64.03

Table 2: Evaluation results on RM-Bench, RewardBench, RewardBench 2, and JudgeBench (%). Arrows (

\uparrow

) indicate that higher is better. Overall averages the displayed domains. Among model rows, bold denotes the best result and the second best is underlined . Additional results and the aggregate score across all benchmark subsets are reported in Appendix Table 7 .

Figure 3: Multi-domain scaling. Domain panels average four fixed subsets each; Overall averages all 12 across every configuration. Training data expands from Math to Math+Code to Math+Code+Safety. Horizontal lines mark the performance of Base and domain specialists.

RM-OPD
Model	Math	Code	Safety	Avg.
Base	73.70	62.93	59.91	65.51
RL Teacher	86.91	74.05	76.38	79.11
w/ DCR	86.02	73.85	78.60	79.49
w/o DCR	86.11	70.42	76.13	77.55
$\Delta$ (vs. w/ DCR)	+0.09	-3.43	-2.48	-1.94

Table 3: DCR ablation (%). Domain means over 12 benchmark subsets.

Figure 4: Ablation of criteria context distillation and out of domain evaluation. (a) Accuracy and response length without CCD and with CCD. Each domain averages four subsets, and Overall averages all 12 benchmark subsets. (b) For OOD generalization, we evaluate Base, Mix-RL, and RM-OPD on RM-Bench Chat/Normal and two RMB-BoN generation subsets.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Data and sampling
Hyperparameter	Value
Maximum prompt length	6,144 tokens
Maximum response length	8,192 tokens
Training batch size	128 prompts
Rollouts per prompt	8
Sampling temperature	1.0

Appendix

Table 4: Student training hyperparameters. Configuration for RM-OPD with DeepSeek-R1-Distill-Qwen-7B and Qwen2.5-7B-Instruct.

Mathematics (25,000)
Source	Size
math_rlvr mixture	24,277
Skywork Magpie Math	619
HelpSteer3 STEM (math-filtered)	104
Code (24,201)
Themis-CodePreference	9,501

Appendix

Table 5: Training data overview. Counts are preference pairs in the student prompt pool. Source names denote the provenance stored with each example; “Skywork mined” comprises domain-filtered subsets of the Skywork preference mixture ( Liu et al., 2026 ) .

Benchmark	Math	Code	Safety	Total
RewardBench	447	984	740	2,171
RewardBench 2	183	–	450	633
RM-Bench	4,761	2,052	3,969	10,782
RMB	–	1,224	7,064	8,288
RMB-BoN	–	280	1,677	1,957
JudgeBench	90	73	–	163

Appendix

Table 6: Evaluation benchmark overview. Domain columns count the official sample groups used in the final evaluation.

Base Models (LLM-as-a-Judge)
Model	RMB $\uparrow$			RMB-BoN $\uparrow$			PKU-SafeRLHF $\uparrow$	Themis $\uparrow$	Average $\uparrow$
	Safety	Code	Overall	Safety	Code	Overall	Safety	Code	All 16
Qwen2.5-7B-Instruct	78.38	77.04	77.71	61.54	57.86	59.70	65.96	67.72	66.31
DeepSeek-R1-Distill-Qwen-7B	63.46	70.42	66.94	45.32	43.93	44.63	56.80	58.89	63.08
Existing Generative Reward Models
JudgeLRM-7B	78.43	78.27	78.35	63.27	63.57	63.42	65.62	64.40	67.43

Appendix

Table 7: Additional evaluation results (%). Overall is the unweighted mean of the displayed domains within each benchmark, while Average equally weights all 16 benchmark subsets across Table 2 and this table. Among model rows, bold and underlined values mark the best and second-best distinct results. Delta rows report RM-OPD minus MixRL-GRPO, with positive values in green and negative values in red .

Training domains	Model	Math	Code	Safety
–	Base	73.70	62.93	59.91
Separate specialists	RL Teacher	86.91	74.05	76.38
Math	Mix-RL	83.87	68.57	56.25
Math	RM-OPD	84.10	68.91	56.14
Math+Code	Mix-RL	84.71	70.50	55.73
Math+Code	RM-OPD	86.99	71.53	58.00

Appendix

Table 8: Domain means under multi-domain scaling (%). Each domain averages four fixed evaluation subsets. All models use DeepSeek-R1-Distill-Qwen-7B. Bold marks the higher score between Mix-RL and RM-OPD within each configuration.

Benchmark	Domain	w/ DCR	w/o DCR	$\Delta$ (vs. w/ DCR)
RM-Bench	Safety	92.01	90.35	-1.66
	Math	91.30	91.89	+0.59
	Code	60.62	57.85	-2.77
RewardBench	Safety	84.05	83.11	-0.94
	Math	95.75	95.53	-0.22
	Code	95.02	93.50	-1.52

Appendix

Table 9: Detailed DCR ablation (%). Results cover the fixed 12 subsets. Green and red values denote increases and decreases relative to w/ DCR.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation
Aug 19, 2026Huan-ang Gao, Haohan Chi, Yong Yan +7Multi-Teacher Knowledge Distillation On-Policy Distillation
SIA-Lab of Tsinghua AIR and ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University
Reward-Gated On-Policy Distillation
Jul 4, 2026Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi +3Language Model Distillation On-Policy Distillation
University of Cologne · AWS AI · UT Austin +1
On-policy Distillation with Verifiable Reward
Aug 25, 2026Wenze Lin, Jiale Zhao, Xitai Jiang +5RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
LeapLab, Tsinghua University · Qiuzhen College, Tsinghua University · Beihang University +2

Aug 19, 2026J/K move · Enter open · S save

Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation

Huan-ang Gao, Haohan Chi, Yong Yan, Shiyuan Feng+6

SIA-Lab of Tsinghua AIR and ByteDance Seed · Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University

Multi-Teacher Knowledge Distillation On-Policy Distillation Multi-Task Optimization Multi-Task RL

Open paper AskPDF arXiv

Multi-teacher on-policy distillation (M-OPD) has emerged as a promising paradigm for consolidating domain-specialized reinforcement learning (RL) experts into a single generalist student via dense, token-level reward supervision. Despite its practical success, the optimization dynamics governing multi-teacher capability integration remain poorly understood, and open, rigorously reproducible recipes are conspicuously lacking. In this work, we establish a controlled M-OPD benchmark on SmolLM3-3B-Base with oracle routing, isolating capability integration from routing ambiguity. Our investigation reveals a pronounced capability integration gap: standard M-OPD captures only 35.6% of the available headroom relative to a domain-routed oracle ensemble, with concise tasks such as instruction following suffering severe degradation and premature stagnation. Crucially, we show that this failure stems not from gradient conflict, but from a severe misallocation of the token-level optimization budget. This pathology is driven by three orthogonal factors: structural sequence-length disparities across domains, dynamic convergence drift due to non-uniform learning rates, and multi-step reward staleness from asynchronous policy updates. To resolve these imbalances, we introduce Open-MOPD, a principled framework incorporating token-share balancing, gap-aware dynamic budget allocation, and student reward refresh. Together, these mechanisms systematically restore cross-domain balance, elevating headroom recovery from 35.6% to 83.4% in a single deployable student. We fully open-source our end-to-end post-training recipe, training trajectories, and evaluation suites on an academically accessible hardware budget.

Abstract

Reward Modeling

Explore similar work

Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation

Reward-Gated On-Policy Distillation

On-policy Distillation with Verifiable Reward