cs.LGJan 2, 2026

IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models

Authors: Haonan Song, Qingchen Xie, Huan Zhu, Feng Xiao, Luxi Xing, Liu Kang, Fuzhen Li, Zhiyong Zheng, 

Organizations: HUJING Digital Media & Entertainment Group (XingYun Lab), Beijing, China · Department of Automation, Tsinghua University, Beijing, China · Beihang University, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Generative Reward Models (GRMs) have demonstrated strong performance in reward modeling, due to their interpretability and potential for refinement through reinforcement learning (RL). However, widely used pairwise GRMs create a computational bottleneck in reinforcement learning from human feedback (RLHF), when calibrating or aggregating preference signals over n candidates, often incurring O(n^2) pairwise judgments. To address this issue, we propose Intergroup Relative Preference Modeling (IRPM), an RL-based method that extends the Bradley-Terry preference-learning paradigm via intergroup comparisons to train pointwise GRMs from pairwise preference data. IRPM derives a pointwise reward for each response by contrasting groups of chosen vs. rejected samples, enabling pointwise scores comparable across candidate sets and O(n) reward evaluation for a variable number of candidates during RL training, while preserving interpretability and scalability. Experiments show that IRPM achieves state-of-the-art performance among pointwise GRMs on JudgeBench and RewardBench, and approaches the performance of leading pairwise GRMs. In addition, IRPM achieves substantial gains in post-training evaluations, demonstrating its effectiveness. Our models and code are available at https://github.com/XINGYUN-AI-LAB/IRPM.

Figures & tables

Figure 1: Different reward generation paradigms and scoring patterns for reward modeling, including the Bradley–Terry model, pairwise GRMs, listwise GRMs and pointwise GRMs. We compare these modeling approaches based on the following criteria: (1) flexibility: whether the method accommodates different protocols with desirable performance; (2) interpretability; (3) fine-grained reward capability: producing calibrated absolute scores that are comparable across candidate sets; and (4) inference-time scalability: the ability to improve performance by allocating more computation at inference time.
Figure 2: Overview of the RL pipeline of IRPM. Given a pairwise preference dataset D={(x(i),yc(i),yr(i))}i=1N\mathcal{D}=\{(x^{(i)},y_{c}^{(i)},y_{r}^{(i)})\}_{i=1}^{N} , IRPM takes a single prompt–response pair (x,y)(x,y) as input and produces a critique cc followed by a scalar score ss . For each chosen/rejected pair, IRPM samples GG rollouts with the GRPO rollout mechanism to form two completion groups, and computes intergroup rewards through intergroup comparisons: including IRPM-Preference, IRPM-AUC and rule-based variants, yielding rollout-level rewards. Finally, the policy is optimized using the same objective and update procedure as GRPO.
ModelsTrain DataRM-BenchJGBenchRWBenchPPEOverall
     Pref.Corr.Avg. 
SOTA GRMs (Pairwise)
DeepSeek-GRM-27B237k——86.064.759.862.3—
EvalPlanner-Llama-8B22k68.130.283.0——54.3—
EvalPlanner-Llama-70B22k82.156.693.865.670.267.975.1
RRM-7B420k70.467.082.2—65.4——
Table 1: Results on four benchmarks (RM-Bench, JGBench, RWBench and PPE). Overall denotes the average performance across the four benchmarks. The number after “+” indicates the absolute improvement over the corresponding backbone model of the same size. Bold marks the best overall results, and underlining indicates the best results among pointwise GRMs. We additionally report results with the Llama-3.1-8B-Instruct backbone and report results for the latest available version of PaTaRM.
Figure 3: Left: Accuracy (%) of different reward design methods on RM-Bench, JudgeBench, RewardBench, and PPE (Preference and Correctness). Right: Per-batch variance of the training scores over training steps for each reward design method.
ModelsRM-BenchJGBenchOverall
BT-70B †78.568.973.7
GRM-70B †82.775.178.9
MSE-32B70.767.168.9
BT-32B74.368.971.6
GRM-32B (Pairwise)84.371.177.7
IRPM-Mean-32B79.674.677.1
Table 2: Performance comparison of models on RM-Bench and JGBench under the same training data. †\dagger indicates results reported from the original papers, corresponding to models trained on the full HelpSteer3 dataset, while all other models are trained on the Qwen3-32B backbone using the filtered 21k HelpSteer3 dataset.
MethodsRM-BenchJGBenchOverall
Full Model77.970.074.0
CoT Reasoning
Score Only76.866.471.6
Adaptive Preference Strength
Full Dataset75.366.070.7
w/o Adaptive Strength77.568.372.9
Table 3: Ablation study of design choices for Qwen3-8B on RM-Bench and JudgeBench. All ablations are compared against our full model.
Figure 4: Post-training MMLU-Pro results for Qwen2.5-7B using feedback from Skywork-Reward-Gemma-2-27B, RRM-32B, RM-R1-32B, and IRPM-32B.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

HyperparametersValue
Epoch2
Batch Size96
Mini Batch Size96
Learning Rate5×10−65\times 10^{-6}
Warmup Steps50
Rollout4
Appendix
Table 4: Hyperparameters for training the IRPM model.
HyperparametersValue
Epoch1
Batch Size64
Mini Batch Size64
Learning Rate5×10−65\times 10^{-6}
Rollout4
KL Coefficient10−310^{-3}
Appendix
Table 5: Hyperparameters for post-training.
Figure 5: Model performance on different benchmarks throughout IRPM-Mean-8B training.
BenchmarksTie RateTie Rate (voting@2)Tie Rate (voting@8)
RM-Bench5.1%1.5%0.28%
JudgeBench9.0%2.8%0.3%
RewardBench5.0%1.2%0.13%
PPE Pref.11.3%3.2%0.89%
PPE Corr.10.4%2.6%0.73%
Avg.8.2%2.3%0.47%
Appendix
Table 6: Tie rate of IRPM-Mean-32B on four benchmarks
Figure 6: Comparison of key training metrics for IRPM-baseline and IRPM-adaptive (KL Loss, entropy, mean reward, and mean response length).
Figure 7: Comparison of Average Training Rewards under Different Sampling Temperatures.
Figure 8: Accuracy Improvements under Different Voting Sizes across Benchmarks.
Figure 9: Average training reward under different reward designs.
GGPairwise GRM (2G)IRPM(G)
273.571.6
474.974.0
875.173.7
Appendix
Table 7: Comparison of Pairwise GRM and IRPM under different GG values.
Figure 10: Performance comparison of Qwen2.5-7B before and after post-training with IRPM-Mean-32B as the reward model across the 14 MMLU-Pro subsets.
Figure 11: Performance comparison of Qwen2.5-14B before and after post-training with IRPM-Mean-32B as the reward model across the 14 MMLU-Pro subsets.
ModelsTraining Time(h)RL Time(h)RM Inference Time(s/step)
B–T2138133
RM-R160108560
IRPM8247210
Appendix
Table 8: Comparison of training time, RL time, and RM inference time.
More in this topic

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 313 papers

Explore similar work

CardsList

  1. ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework

    Apr 8, 2026Kai Qin, Liangxin Liu, Yu Liang +7Preference ModelingSelf-Reflection in Language Model Agents

    Shenzhen International Graduate School, Tsinghua University · Baidu Inc., Beijing, China

  2. EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans

    Oct 4, 2026Xuancheng Li, Beining Wang, Haitao Li +7Reward ModelingProcess Reward Models

    Department of Computer Science and Technology, Tsinghua University, Beijing, China · Tencent, Beijing, China

  3. RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

    Aug 6, 2026Chenglong Wang, Ziming Zhu, Yifu Huo +9Reward ModelingRL for Language Models

    School of Computer Science and Engineering, Northeastern University, Shenyang, China · 2NiuTrans Research, Shenyang, China · 3Independent Researcher, Beijing, China +1