IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models
Authors: Haonan Song, Qingchen Xie, Huan Zhu, Feng Xiao, Luxi Xing, Liu Kang, Fuzhen Li, Zhiyong Zheng, Feng Jiang, Ziheng Li, Kun Yan, Qingyi Si, Yanghua Xiao, Hongcheng Guo, Fan Yang
Organizations: HUJING Digital Media & Entertainment Group (XingYun Lab), Beijing, China · Department of Automation, Tsinghua University, Beijing, China · Beihang University, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Fudan University, Shanghai, China
Generative Reward Models (GRMs) have demonstrated strong performance in reward modeling, due to their interpretability and potential for refinement through reinforcement learning (RL). However, widely used pairwise GRMs create a computational bottleneck in reinforcement learning from human feedback (RLHF), when calibrating or aggregating preference signals over n candidates, often incurring O(n^2) pairwise judgments. To address this issue, we propose Intergroup Relative Preference Modeling (IRPM), an RL-based method that extends the Bradley-Terry preference-learning paradigm via intergroup comparisons to train pointwise GRMs from pairwise preference data. IRPM derives a pointwise reward for each response by contrasting groups of chosen vs. rejected samples, enabling pointwise scores comparable across candidate sets and O(n) reward evaluation for a variable number of candidates during RL training, while preserving interpretability and scalability. Experiments show that IRPM achieves state-of-the-art performance among pointwise GRMs on JudgeBench and RewardBench, and approaches the performance of leading pairwise GRMs. In addition, IRPM achieves substantial gains in post-training evaluations, demonstrating its effectiveness. Our models and code are available at https://github.com/XINGYUN-AI-LAB/IRPM.
Figures & tables
Figure 1: Different reward generation paradigms and scoring patterns for reward modeling, including the Bradley–Terry model, pairwise GRMs, listwise GRMs and pointwise GRMs. We compare these modeling approaches based on the following criteria: (1) flexibility: whether the method accommodates different protocols with desirable performance; (2) interpretability; (3) fine-grained reward capability: producing calibrated absolute scores that are comparable across candidate sets; and (4) inference-time scalability: the ability to improve performance by allocating more computation at inference time.
Figure 2: Overview of the RL pipeline of IRPM. Given a pairwise preference dataset D={(x(i),yc(i),yr(i))}i=1N , IRPM takes a single prompt–response pair (x,y) as input and produces a critique c followed by a scalar score s . For each chosen/rejected pair, IRPM samples G rollouts with the GRPO rollout mechanism to form two completion groups, and computes intergroup rewards through intergroup comparisons: including IRPM-Preference, IRPM-AUC and rule-based variants, yielding rollout-level rewards. Finally, the policy is optimized using the same objective and update procedure as GRPO.
Models
Train Data
RM-Bench
JGBench
RWBench
PPE
Overall
Pref.
Corr.
Avg.
SOTA GRMs (Pairwise)
DeepSeek-GRM-27B
237k
—
—
86.0
64.7
59.8
62.3
—
EvalPlanner-Llama-8B
22k
68.1
30.2
83.0
—
—
54.3
—
EvalPlanner-Llama-70B
22k
82.1
56.6
93.8
65.6
70.2
67.9
75.1
RRM-7B
420k
70.4
67.0
82.2
—
65.4
—
—
Table 1: Results on four benchmarks (RM-Bench, JGBench, RWBench and PPE). Overall denotes the average performance across the four benchmarks. The number after “+” indicates the absolute improvement over the corresponding backbone model of the same size. Bold marks the best overall results, and underlining indicates the best results among pointwise GRMs. We additionally report results with the Llama-3.1-8B-Instruct backbone and report results for the latest available version of PaTaRM.
Figure 3: Left: Accuracy (%) of different reward design methods on RM-Bench, JudgeBench, RewardBench, and PPE (Preference and Correctness). Right: Per-batch variance of the training scores over training steps for each reward design method.
Models
RM-Bench
JGBench
Overall
BT-70B †
78.5
68.9
73.7
GRM-70B †
82.7
75.1
78.9
MSE-32B
70.7
67.1
68.9
BT-32B
74.3
68.9
71.6
GRM-32B (Pairwise)
84.3
71.1
77.7
IRPM-Mean-32B
79.6
74.6
77.1
Table 2: Performance comparison of models on RM-Bench and JGBench under the same training data. † indicates results reported from the original papers, corresponding to models trained on the full HelpSteer3 dataset, while all other models are trained on the Qwen3-32B backbone using the filtered 21k HelpSteer3 dataset.
Methods
RM-Bench
JGBench
Overall
Full Model
77.9
70.0
74.0
CoT Reasoning
Score Only
76.8
66.4
71.6
Adaptive Preference Strength
Full Dataset
75.3
66.0
70.7
w/o Adaptive Strength
77.5
68.3
72.9
Table 3: Ablation study of design choices for Qwen3-8B on RM-Bench and JudgeBench. All ablations are compared against our full model.
Figure 4: Post-training MMLU-Pro results for Qwen2.5-7B using feedback from Skywork-Reward-Gemma-2-27B, RRM-32B, RM-R1-32B, and IRPM-32B.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameters
Value
Epoch
2
Batch Size
96
Mini Batch Size
96
Learning Rate
5×10−6
Warmup Steps
50
Rollout
4
Appendix
Table 4: Hyperparameters for training the IRPM model.
Hyperparameters
Value
Epoch
1
Batch Size
64
Mini Batch Size
64
Learning Rate
5×10−6
Rollout
4
KL Coefficient
10−3
Appendix
Table 5: Hyperparameters for post-training.
Figure 5: Model performance on different benchmarks throughout IRPM-Mean-8B training.
Benchmarks
Tie Rate
Tie Rate (voting@2)
Tie Rate (voting@8)
RM-Bench
5.1%
1.5%
0.28%
JudgeBench
9.0%
2.8%
0.3%
RewardBench
5.0%
1.2%
0.13%
PPE Pref.
11.3%
3.2%
0.89%
PPE Corr.
10.4%
2.6%
0.73%
Avg.
8.2%
2.3%
0.47%
Appendix
Table 6: Tie rate of IRPM-Mean-32B on four benchmarks
Figure 6: Comparison of key training metrics for IRPM-baseline and IRPM-adaptive (KL Loss, entropy, mean reward, and mean response length).
Figure 7: Comparison of Average Training Rewards under Different Sampling Temperatures.
Figure 8: Accuracy Improvements under Different Voting Sizes across Benchmarks.
Figure 9: Average training reward under different reward designs.
G
Pairwise GRM (2G)
IRPM(G)
2
73.5
71.6
4
74.9
74.0
8
75.1
73.7
Appendix
Table 7: Comparison of Pairwise GRM and IRPM under different G values.
Figure 10: Performance comparison of Qwen2.5-7B before and after post-training with IRPM-Mean-32B as the reward model across the 14 MMLU-Pro subsets.
Figure 11: Performance comparison of Qwen2.5-14B before and after post-training with IRPM-Mean-32B as the reward model across the 14 MMLU-Pro subsets.
Models
Training Time(h)
RL Time(h)
RM Inference Time(s/step)
B–T
21
38
133
RM-R1
60
108
560
IRPM
82
47
210
Appendix
Table 8: Comparison of training time, RL time, and RM inference time.
Reward Models (RMs) are critical components in the Reinforcement Learning from Human Feedback (RLHF) pipeline, directly determining the alignment quality of Large Language Models (LLMs). Recently, Generative Reward Models (GRMs) have emerged as a superior paradigm, offering higher interpretability and stronger generalization than traditional scalar RMs. However, existing methods for GRMs focus primarily on outcome-level supervision, neglecting analytical process quality, which constrains their potential. To address this, we propose ReflectRM, a novel GRM that leverages self-reflection to assess analytical quality and enhance preference modeling. ReflectRM is trained under a unified generative framework for joint modeling of response preference and analysis preference. During inference, we use its self-reflection capability to identify the most reliable analysis, from which the final preference prediction is derived. Experiments across four benchmarks show that ReflectRM consistently improves performance, achieving an average accuracy gain of +3.7 on Qwen3-4B. Further experiments confirm that response preference and analysis preference are mutually reinforcing. Notably, ReflectRM substantially mitigates positional bias, yielding +10.2 improvement compared with leading GRMs and establishing itself as a more stable evaluator. Our code is available at https://github.com/yuliangCarmelo/ReflectRM.
Kai Qin, Liangxin Liu, Yu Liang +7
Shenzhen International Graduate School, Tsinghua University · Baidu Inc., Beijing, China
Generative reward models (GRMs) are important for LLM optimization. Unlike scalar reward models, GRMs generate natural-language critiques alongside preference judgments, providing finer-grained evaluation signals. Their effectiveness depends heavily on critique reliability. However, existing GRM training typically uses final preference correctness as outcome supervision. Because the preference outcome space is highly constrained, unreliable critiques can still yield correct outcomes and thus be reinforced. Recent work leverages human critiques for process supervision, but such critiques are scarce and are often reduced to scalar rewards, leaving their fine-grained evaluative information underutilized. We argue that evaluative criteria learned from human critiques can be generalized to broader outcome-only preference data. To this end, we propose EnGRICH, a GRM training framework that pairs the GRM with a training-time MetaCritic learned from a small set of human critiques. MetaCritic constructs response-specific rubrics and uses them to evaluate the evidence coverage and correctness of generated critiques. The resulting signals provide both process rewards for fine-grained credit assignment and structured guidance for exploring better critiques. During GRM training, MetaCritic is further optimized to generalize human-grounded evaluative criteria to outcome-only data. At inference, the trained GRM operates independently. Experiments across seven reward-model benchmarks show that EnGRICH consistently improves over competitive baselines, while further analyses validate the effectiveness of its core mechanisms.
Xuancheng Li, Beining Wang, Haitao Li +7
Department of Computer Science and Technology, Tsinghua University, Beijing, China · Tencent, Beijing, China
Recent advances in reward modeling show a paradigm shift from discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm adopted by existing RL algorithms. To bridge this gap, we propose a Ranking-based Reward Construction (RRC) approach, which enables generative reward models to provide more effective RL learning signals by deriving rewards from relative preference rankings. RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches. Our code can be found at https://github.com/wangclnlp/RRC.
Chenglong Wang, Ziming Zhu, Yifu Huo +9
School of Computer Science and Engineering, Northeastern University, Shenyang, China · 2NiuTrans Research, Shenyang, China · 3Independent Researcher, Beijing, China +1