Reward Models (RMs) are critical components in the Reinforcement Learning from Human Feedback (RLHF) pipeline, directly determining the alignment quality of Large Language Models (LLMs). Recently, Generative Reward Models (GRMs) have emerged as a superior paradigm, offering higher interpretability and stronger generalization than traditional scalar RMs. However, existing methods for GRMs focus primarily on outcome-level supervision, neglecting analytical process quality, which constrains their potential. To address this, we propose ReflectRM, a novel GRM that leverages self-reflection to assess analytical quality and enhance preference modeling. ReflectRM is trained under a unified generative framework for joint modeling of response preference and analysis preference. During inference, we use its self-reflection capability to identify the most reliable analysis, from which the final preference prediction is derived. Experiments across four benchmarks show that ReflectRM consistently improves performance, achieving an average accuracy gain of +3.7 on Qwen3-4B. Further experiments confirm that response preference and analysis preference are mutually reinforcing. Notably, ReflectRM substantially mitigates positional bias, yielding +10.2 improvement compared with leading GRMs and establishing itself as a more stable evaluator. Our code is available at https://github.com/yuliangCarmelo/ReflectRM.
Figures & tables
Figure 1: Mutual reinforcement in ReflectRM. By unifying response preference and analysis preference into a single conditional generative process, ReflectRM internalizes a more robust and reliable evaluative logic.
Figure 2: Overview of the ReflectRM method. (Top) Our Unified Judgment Framework , which models response preference and analysis preference as a single conditional generative process. (Bottom) The Two-Stage Inference Strategy that leverages the model’s self-reflection capability to identify and aggregate reliable analytical traces, yielding more robust and reliable final judgments.
ID
System
Source
RewardBench
RM-Bench
RMB
PPE-Preference
Overall
Pref.
Refl.
AVG
Δ ( ↑ )
Open Generative Reward Models
1
Llama-3-OffsetBias-RM-8B
-
-
89.0
71.3
57.8
59.2
69.3
-
2
ArmoRM-Llama3-8B-v0.1
-
-
90.4
69.3
64.6
60.6
71.2
-
3
Skywork-Reward-Llama-3.1-8B-v0.2
-
-
93.1
72.1
66.6
62.2
73.5
-
4
GRAM-R 2 -3B
-
-
-
84.6
-
-
-
-
Table 1: Main results on four RM benchmarks. Qwen3-4B and Qwen3-8B are chosen as the backbone model. The best results under each backbone are labeled using bold font .
Figure 3: Effect of preference-to-reflection ratio. A 4:1 ratio provides the optimal balance for training.
System
R.B.
RMB
AVG
Δ
ReflectRM
86.7
77.1
81.9
-
w/ Random Anchor
86.0
76.8
81.4
-0.5
w/ Random Winners
85.7
76.0
80.9
-1.0
Table 2: Performance comparison on inference strategies. Both a reliable anchor and self-reflection capability are essential for ReflectRM.
System
Pref.
Refl.
Sum
R.B.
RMB
AVG
RFT
13.7k
-
13.7k
84.0
71.2
77.6
w/ Scaled Pref.
17.1k
-
17.1k
85.0
73.2
79.1
w/ Refl. (Ours)
13.7k
3.4k
17.1k
85.3
75.3
80.3
Table 3: Impact of reflection data on response preference. Reflection data offers richer learning signals compared to simply scaling preference data volume.
Figure 4: Impact of preference data on self-reflection. Preference data provides a robust foundation for the evaluative logic used to judge analytical processes.
System
Qwen3-4B
Qwen3-8B
Qwen3-14B
Base Model
72.7
75.1
75.7
ReflectRM (Ours)
76.4
77.3
79.5
Table 4: Performance scaling across model sizes. ReflectRM yields consistent performance gains, demonstrating the framework’s high scalability.
ID
System
JudgeBench
RewardBench
RM-Bench
RMB
PPE-Preference
Overall
(1k)
(1k)
(1k)
(1k)
AVG
Δ ( ↑ )
7
Qwen3-4B
49.3
74.8
60.5
52.3
34.9
54.3
-
8
4 + RFT
53.1
75.1
67.1
56.5
46.6
59.7
5.4
9
4 + Unified Training
53.1
78.5
64.3
64.3
48.1
61.7
7.4
10
6 + Two-Stage Inference (ReflectRM)
54.9
81.1
66.6
68.3
51.7
64.5
10.2
11
Qwen3-8B
52.7
78.4
64.9
60.9
42.1
59.8
-
Table 5: Evaluation of positional consistency across five benchmarks. Qwen3-4B and Qwen3-8B are chosen as the backbone model. The best results under each backbone are labeled using bold font . ReflectRM achieves a substantial improvement of up to +10.2 points, nearly doubling the consistency gains of the standard RFT baseline, showing that supervising the analytical trace leads to a more stable and reliable judgment process.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
Pref.
Refl.
Sum
Qwen3-4B
13,692
3,420
17,112
Qwen3-8B
13,560
3,380
16,940
Qwen3-14B
13,534
3,384
16,918
Appendix
Table 6: Detailed statistics of the training dataset.
System
R.B.
RMB
AVG
Δ
ReflectRM
86.7
77.1
81.9
-
w/ Anchor-only
81.1
74.5
77.8
-4.1
Review & Revise
84.5
71.5
78.0
-3.9
Appendix
Table 7: Comparison with simpler inference strategies.
System
RewardBench
RM-Bench
RMB
PPE-Preference
AVG
Δ
GRM
84.0
76.5
71.3
62.7
73.6
-
ReflectRM (Ours)
86.7
77.7
77.1
64.2
76.4
+2.8
Appendix
Table 8: Performance comparison under matched computational budgets.
System
RewardBench
RM-Bench
RMB
PPE-Preference
AVG
Δ
Qwen3-4B
84.8
75.4
70.4
60.1
72.7
-
+ RFT
83.1
75.1
74.1
62.2
73.6
+0.9
+ ReflectRM (Ours)
86.7
76.1
72.5
64.1
74.9
+2.2
Appendix
Table 9: Main Performance on Skywork-Reward-V2 with Qwen3-4B.
System
JudgeBench
RewardBench (1k)
RM-Bench (1k)
RMB (1k)
PPE-Preference (1k)
AVG
Δ
Qwen3-4B
49.3
74.8
60.5
52.3
34.9
54.3
-
+ RFT
54.6
77.6
67.5
57.5
36.2
58.7
+4.4
+ ReflectRM (Ours)
57.6
79.6
69.1
59.3
40.8
61.3
+7.0
Appendix
Table 10: Positional Consistency on Skywork-Reward-V2 with Qwen3-4B.
System
R.B.
RMB
AVG
Qwen3-4B
84.8
70.4
77.6
+ Pref.
84.0
71.2
77.6
+ Pref. & Refl.
85.3
75.3
80.3
Qwen3-8B
85.8
72.9
79.4
+ Pref.
85.0
73.9
79.5
+ Pref. & Refl.
87.9
73.0
80.5
Appendix
Table 11: Performance on RewardBench (R.B.) and RMB with mixed training data across different scales and architectures.
Query and Responses
Query
In this conversation, please keep your responses succint the to the point. Please avoid lengthy explanations or examples unless instructed.
Response 1
Understood! I’ll keep my responses brief and to the point.
Response 2
Absolutely, I’ll keep my responses concise and to the point. How can I assist you today?
Output 1
Analysis
The user requested that responses be “succint the to the point” and to avoid lengthy explanations or examples unless instructed. Response 1 simply says “Understood! I’ll keep my responses brief and to the point.” while Response 2 adds “Absolutely, I’ll keep my responses concise and to the point. How can I assist you today?” Response 2 is more comprehensive as it not only confirms understanding of the user’s instruction but also actively asks how the user wants to proceed, which shows engagement and willingness to assist. However, the user’s query was simply to confirm that the responses would be brief and to the point, and the initial response from the assistant was already meeting that requirement. Response 1 is more in line with the user’s explicit instruction to be “succint the to the point” as it directly confirms understanding without adding extra information. Response 2, while polite, includes an additional question that may not be necessary if the user’s query was solely about confirming the response style. Critique 1 Anchor
Predict
Response 1 is better than Response 2 ×
Appendix
Table 12: A case study of ReflectRM’s two-stage inference strategy. By reflecting on the analytical processes, ReflectRM finally derives the correct verdict.