ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training

Yu Liang; Liangxin Liu; Longzheng Wang; Yan Wang; Yueyang Zhang; Long Xia; Zhiyuan Sun; Daiting Shi

cs.AIApr 8, 2026

ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training

Authors: Yu Liang, Liangxin Liu, Longzheng Wang, Yan Wang, Yueyang Zhang, Long Xia, Zhiyuan Sun, Daiting Shi

Organizations: Baidu Inc., Beijing, China

Abstract

Generative reward models (GRMs) have emerged as a promising approach for aligning Large Language Models (LLMs) with human preferences by offering greater representational capacity and flexibility than traditional scalar reward models. However, GRMs face two major challenges: reliance on costly human-annotated data restricts scalability, and self-training approaches often suffer from instability and vulnerability to reward hacking. To address these issues, we propose ConsistRM, a self-training framework that enables effective and stable GRM training without human annotations. ConsistRM incorporates the Consistency-Aware Answer Reward, which produces reliable pseudo-labels with temporal consistency, thereby providing more stable model optimization. Moreover, the Consistency-Aware Critique Reward is introduced to assess semantic consistency across multiple critiques and allocates fine-grained and differentiated rewards. Experiments on five benchmark datasets across four base models demonstrate that ConsistRM outperforms vanilla Reinforcement Fine-Tuning (RFT) by an average of 1.5%. Further analysis shows that ConsistRM enhances output consistency and mitigates position bias caused by input order, highlighting the effectiveness of consistency-aware rewards in improving GRMs. Our implementation is available at https://github.com/yuliangCarmelo/ConsistRM.

Figures & tables

Figure 1: An overview of ConsistRM in comparison with vanilla GMRs, emphasizing its self-training framework based on consistency-aware learning to enhance model pairwise preference performance.

Figure 2: Overall Framework of ConsistRM. The framework aims to produce an output $o=(c,y)$ , which comprises a textual critique $c$ and a binary preference label $y$ . Here, $y=-1$ indicates that $a_{1}$ is preferred over $a_{2}$ ( $a_{1}\succ a_{2}$ )., and $y=1$ indicates the opposite preference ( $a_{2}\succ a_{1}$ ). Consistency-Aware Answer Reward module integrates online-state preference and memory-driven preference to generate pseudo-labels for training. The Consistency-Aware Critique Reward module provides fine-grained rewards by measuring the similarity of multiple criteria. By combining these two rewards, ConsistRM performs self-training to progressively enhance the GRMs’ ability.

ID System	GT	RewardBench	PPE Pref	RM-Bench	RMB	JudgeBench	Overall
ID System	GT	RewardBench	PPE Pref	RM-Bench	RMB	JudgeBench	AVG	$\Delta$ ( $\uparrow$ )
	Open Generative Reward Models
1 EvalPlanner-Llama-8B	✓	83.0	54.3	68.1	-	30.2	-	-
2 J1-Llama-8B	✓	85.7	59.8	73.4	-	42.0	-	-
3 DeepSeek-GRM-27B	✓	88.5	67.2	-	70.3	-	-	-
	Implemented Existing Method

Table 1: Overall performance of different GRMs on five benchmarks. ConsistRM achieved the best results without requiring any human-annotated training data, demonstrating our effectiveness of the self-training paradigm. Bold indicating the best performance and underline indicating the second-best performance.

System	RewardBench	PPE Pref	RM-Bench	RMB	JudgeBench	Avg	$\Delta$ ( $\uparrow$ )
ConsistRM	85.6	67.7	78.3	79.1	56.9	73.5	-
w/o CACR	84.9	64.8	77.3	78.1	56.0	72.2	-1.3
w/o Online-State Consistency Preference	85.5	64.1	78.6	76.7	56.7	72.3	-1.2
w/o Memory-Driven Consistency Preference	84.3	63.1	75.4	74.2	54.8	70.4	-3.2

Table 2: Ablation study results of ConsistRM on five benchmarks.

Foundation Model	Method	RewardBench	PPE Pref	RM-Bench	RMB	JudgeBench	Overall
Foundation Model	Method	RewardBench	PPE Pref	RM-Bench	RMB	JudgeBench	AVG	$\Delta$ ( $\uparrow$ )
Qwen3-4B	Vanilla RFT	80.0	64.7	76.6	76.7	51.1	69.8	-
Qwen3-4B	ConsistRM	80.3	65.1	76.8	76.5	56.1	71.0	+1.2
Qwen3-8B	Vanilla RFT	85.4	65.4	78.2	78.2	55.4	72.5	-
Qwen3-8B	ConsistRM	85.6	67.7	79.2	79.2	55.9	73.5	+1.0
Qwen3-14B	Vanilla RFT	87.4	66.3	79.3	76.3	68.6	75.6	-

Table 3: Results of ConsistRM with various foundation models. ConsistRM can be stably applied to LLMs of different sizes and architectures.

System	RewardBench	JudgeBench	Overall
System	RewardBench	JudgeBench	AVG	$\Delta$ ( $\uparrow$ )
Various Critique Rweard
ConsistRM	80.3	56.1	68.2	-
w/ CACR-Low	81.2	50.6	65.8	-2.4
w/ DeepConf-Tail	80.4	50.3	65.4	-2.8
w/ DeepConf-Low	79.8	51.2	65.5	-2.7

Table 4: Comparison results with different setups on GRMs. ConsistRM achieves the better Results.

Figure 3: Comparison of the stability of different GRMs with different rollout sizes.

Figure 4: Changing trends of the average response length with different GRMs. ConsistRM achieves the balance between performance and efficiency.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Training Epochs	4
Global Batch Size	64
Learning Rate	1e-6
Maximum Prompt Length	4096 tokens
Maximum Generation Length	1024 tokens
PPO Mini-Batch Size	32

Appendix

Table 5: Implementation Details of Our Method

Model	Number of Examples
Qwen3-4B	13,236
Qwen3-8B	12,098
Qwen3-14B	10,340

Appendix

Table 6: Statistics of the final training dataset

	0 steps	80 steps	160 steps	240 steps	320 steps
Acc (%)	80.25	83.45	87.31	88.25	91.50

Appendix

Table 7: Accuracy of consistency-aware pseudo-labels over training steps.

Model	CodePrefBench	$\Delta$
Qwen3-4B + RFT	62.1	–
Qwen3-4B + ConsistRM	66.6	+4.5
Qwen3-8B + RFT	69.7	–
Qwen3-8B + ConsistRM	72.1	+2.4

Appendix

Table 8: Performance on CodePrefBench.

System	Reward Bench	PPE Pref	RM- Bench	RMB	Judge Bench	Avg
ConsistRM	85.6	67.7	78.3	79.1	56.9	73.5
w/ removal	86.1	67.9	76.2	79.2	57.2	73.3

Appendix

Table 9: Effect of eliminating incorrect pseudo-labels on model performance.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 313 papers

ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
Apr 8, 2026Kai Qin, Liangxin Liu, Yu Liang +7Preference Modeling Self-Reflection in Language Model Agents
Shenzhen International Graduate School, Tsinghua University · Baidu Inc., Beijing, China
EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
Oct 4, 2026Xuancheng Li, Beining Wang, Haitao Li +7Reward Modeling Process Reward Models
Department of Computer Science and Technology, Tsinghua University, Beijing, China · Tencent, Beijing, China
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
May 15, 2025Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12Reward Modeling LLM Alignment
NLP Group, Fudan University · Hong Kong Baptist University · LLM Team, Meituan Inc

Apr 8, 2026J/K move · Enter open · S save

ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework

Kai Qin, Liangxin Liu, Yu Liang, Longzheng Wang+6

Shenzhen International Graduate School, Tsinghua University · Baidu Inc., Beijing, China

Preference Modeling Self-Reflection in Language Model Agents Reward Modeling LLM Alignment Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

Reward Models (RMs) are critical components in the Reinforcement Learning from Human Feedback (RLHF) pipeline, directly determining the alignment quality of Large Language Models (LLMs). Recently, Generative Reward Models (GRMs) have emerged as a superior paradigm, offering higher interpretability and stronger generalization than traditional scalar RMs. However, existing methods for GRMs focus primarily on outcome-level supervision, neglecting analytical process quality, which constrains their potential. To address this, we propose ReflectRM, a novel GRM that leverages self-reflection to assess analytical quality and enhance preference modeling. ReflectRM is trained under a unified generative framework for joint modeling of response preference and analysis preference. During inference, we use its self-reflection capability to identify the most reliable analysis, from which the final preference prediction is derived. Experiments across four benchmarks show that ReflectRM consistently improves performance, achieving an average accuracy gain of +3.7 on Qwen3-4B. Further experiments confirm that response preference and analysis preference are mutually reinforcing. Notably, ReflectRM substantially mitigates positional bias, yielding +10.2 improvement compared with leading GRMs and establishing itself as a more stable evaluator. Our code is available at https://github.com/yuliangCarmelo/ReflectRM.

Abstract

Reward Modeling

Explore similar work

ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework

EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment