Reward modeling is essential for aligning Large Language Models(LLMs) with human preferences, yet conventional reward models suffer from poor interpretability and heavy reliance on costly expert annotations. While recent rubric-based approaches enhance evaluation transparency, they lack systematic quality control, yielding noisy and redundant criteria, failing to mitigate persistent biases (e.g., verbosity, position) in LLM evaluators, and creating a scalability-reliability trade-off. To address these limitations, we propose CDRRM (Contrast-Driven Rubric Reward Model), a framework built on a novel Contrast-then-Synthesis paradigm for high-quality rubric generation and guided preference judgment. CDRRM first conducts multi-dimensional contrastive profiling on preference pairs to identify causal discriminative factors, then synthesizes these insights into compact, context-aware rubrics to guide preference judg- ments. Extensive experiments on three authoritative benchmarks (RewardBench, RMBench, RMB) demonstrate that CDRRM achieves state-of-the-art performance across diverse domains and effectively mitigates aforementioned evaluation biases. Notably, our approach delivers exceptional data efficiency: training the rubric generator on only 3k high-quality samples empowers a frozen pre-trained judge model to outperform fully fine-tuned baselines. This work offers a scalable, interpretable, and data-efficient path for reward modeling.
Figures & tables
Figure 1. An illustrative example of rubric generation for a Greatest Common Divisor (GCD) task, contrasting rubrics from direct prompting (right, redundant and potentially misleading) with those from our Contrast-then-Synthesis paradigm (left, concise and effective). The bottom-left panel shows statistics on the number of generated rubrics with respect to single preferences. A short description for accessibility.
Figure 2. The CDRRM framework. (Top) The Contrast-then-Synthesis paradigm synthesizes evidence-based rubrics via contrastive analysis of preference pairs. (Bottom) These rubrics, paired with synthesized rubric-grounded justifications, supervise the training of a Rubric Generator (to automate context-aware criterion synthesis) and a Judge Model (to generate rubric-aligned justifications for precise preference predictions).
Del
Baseline
Del 0
Del 1
Del 2
Acc. (%)
95.65
95.70
95.23
95.37
Δ (%)
0.00
+0.05
-0.42
-0.28
Table 1. Impact of Rubric Reduction on Accuracy. A short description of the table for accessibility.
Models
RewardBench
RMBench
RMB
Avg.
Overall
Easy
Medium
Hard
Overall
Help
Harm
Overall
Scalar RMs
SteerLM-RM-70B
88.8
48.3
54.9
54.3
52.5
57.4
67.3
62.4
67.9
InternLM2-20B-Reward
90.2
82.6
71.6
50.7
68.3
76.3
67.0
71.7
76.7
ArmoRM-Llama3-8B-v0.1
90.4
82.2
71.0
49.8
67.7
78.7
66.3
72.5
76.9
Skywork-Reward-Llama-3.1-8B
92.5
89.0
74.7
46.6
70.1
78.1
75.9
77.0
79.9
Table 2. Main results on RewardBench, RMBench, and RMB. We report the accuracy scores for each category. The best results in each column are highlighted in bold . “Help” and “Harm” denote Helpfulness and Harmlessness, respectively. The “Average” column represents the mean of the overall scores across the three benchmarks.
Method
RewardBench
RMBench
RMB
Average
Direct Judge
79.7
74.7
74.9
76.4
One-step Rubric Judge
86.0
75.0
75.9
79.0
CDRRM-8B(Base)
90.4
86.1
80.8
85.8
Table 3. Ablation study on different judging strategies. We compare CDRRM against Direct Judge and One-step Rubric Judge. All models are based on the Qwen3-8B backbone.
Figure 3. Impact of training data size on model performance. Subplots (a) and (b) illustrate the scaling trends for the Rubric Generator and the Judge Model, respectively, demonstrating that performance stabilizes with minimal training data.
Instruction: GIVE ME AN EXECUTIVE SUMMARY OF LOTTADIGITAL.COM
Response A (Chosen):
LottaDigital.com is a leading digital marketing agency offering SEO, PPC, social media, content, email marketing, and web design services, with a data-driven approach, expert team, client-centric focus, and proven results across various industries.
Response B (Rejected):
Executive Summary of LottaDigital.com LottaDigital.com stands as a pioneering digital marketing agency that excels in delivering comprehensive digital solutions… Core Services Offered: 1. SEO (Search Engine Optimization): We employ advanced SEO strategies… 2. PPC (Pay-Per-Click) Advertising: Leveraging platforms like Google Ads… 3. Social Media Marketing: We manage and optimize social media profiles… […] (Content Omitted for Brevity) 6. Software Development: We offer custom software development services… Key Strengths: - Data-Driven Approach: We rely on data analytics… - Client [Truncated mid-sentence]
Baseline: Direct Judge Analysis
Critique: “Response B is comprehensive, well-structured, and detailed . It provides a thorough overview… and highlights key strengths.” / “Response A lacks depth and detail…” Flaw: The model exhibits Verbosity Bias , favoring the lengthy formatting of Response B while completely overlooking the critical truncation error at the end. Prediction: ✗ Winner: Response B (Incorrect)
Table 4. Case Study on Robustness. The Direct Judge falls into the “verbosity trap,” preferring the lengthy Response B despite it being cut off. In contrast, CDRRM generates specific hard rules targeting completeness and conciseness, correctly penalizing the truncation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Parameter
Value
Rubric Generator
Epochs
1
Max Length
25000
Batch Size
128
Optimizer
AdamW
Learning Rate
5e-5
Warmup Ratio
0.05
Appendix
Table 5. Hyperparameter Settings for CDRRM Components
Models
Chat
Chat-Hard
Safety
Reasoning
Overall
Scalar RMs
SteerLM-RM-70B
91.3
80.3
92.8
90.6
88.8
InternLM2-20B-Reward
98.9
76.5
89.5
95.8
90.2
ArmoRM-Llama3-8B-v0.1
96.9
76.8
90.5
97.3
90.4
Skywork-Reward-Llama-3.1-8B
95.8
87.3
90.8
96.2
92.5
INF-ORM-Llama3.1-70B
96.6
91.0
93.6
99.1
95.1
Appendix
Table 6. Performance comparison on RewardBench. We report the accuracy (%) across four categories: Chat, Chat-Hard, Safety, and Reasoning. The best results in each column are highlighted in bold.
Instruction: Is JavaScript a back end or front end language?
Response A (Chosen/Ground Truth):
JavaScript is primarily known as a front-end language, as it is widely used for client-side scripting in web browsers to create dynamic and interactive web pages. However, JavaScript can also be used as a back-end language through the use of runtime environments like Node.js. Node.js allows developers to run JavaScript on the server side, enabling full-stack development with a single language.
Response B (Rejected):
JavaScript is primarily a front-end language but can also be used as a back-end language with Node.js for full-stack development.
### Front-End Capabilities
- DOM Manipulation / Event Handling / Animations and Effects with libs like jQuery/React
Appendix
Table 7. Case Study on Verbosity Bias Mitigation. Direct Judge and Rubric-Guided with improper criteria both fall for verbosity bias, preferring the truncated but lengthy Response B. Our CDRRM generates optimized hard rules targeting completeness and structural integrity , correctly penalizing Response B’s critical flaws and selecting the concise, complete Response A.
Instruction: Write a C++ function string int_to_mini_romank(int number) to convert a positive integer ( 1≤num≤1000 ) to lowercase Roman numerals.
Response A (Chosen/Ground Truth):
Core Implementation (Key Part):
string int_to_mini_romank(int number) // Correct function name per instruction
Table 8. Case Study on Subtle Naming Error Identification (Code Scenario). Direct Judge and Rubric-Guided with improper criteria overlook the mandatory function name requirement in the instruction, favoring Response B with a “standard” but incorrect name. Our CDRRM generates rubrics targeting instruction-aligned naming rules , correctly identifying Response A’s compliance and Response B’s critical naming flaw.
Instruction: A circle of radius 1 is surrounded by 4 circles of radius r (centers form a square). Derive the exact value of r .
Response A (Chosen/Ground Truth):
Core Geometric Modeling & Derivation:
1. Centers of 4 surrounding circles form a square; distance between adjacent centers: 2r ;
2. Distance from central circle center to surrounding circle center: 1+r (key correct distance);
3. Form right triangle (central center, surrounding center, square side midpoint) with legs r,r , hypotenuse 1+r ;
4. Pythagorean theorem: (1+r)2=2r2⟹r2−2r−1=0 ;
Appendix
Table 9. Case Study on Subtle Geometric Modeling Error Identification (Math Reasoning Scenario). Direct Judge and Rubric-Guided with improper criteria fail to capture the core geometric flaw of Response B, while our CDRRM generates task-aligned rubrics targeting valid geometric modeling and answer-configuration consistency , accurately identifying the correct solution in Response A.
Reward models are essential for aligning language model outputs with human preferences, yet existing approaches often lack both controllability and interpretability. These models are typically optimized for narrow objectives, limiting their generalizability to broader downstream tasks. Moreover, their scalar outputs are difficult to interpret without contextual reasoning. To address these limitations, we introduce R3, a novel reward modeling framework that is rubric-agnostic, generalizable across evaluation dimensions, and provides interpretable, reasoned score assignments. R3 enables more transparent and flexible evaluation of language models, supporting robust alignment with diverse human values and use cases. Our models, data, and code are available as open source at https://github.com/rubricreward/r3.
David Anugraha, Zilu Tang, Lester James V. Miranda +5
Department of Computer Science, Stanford University · Department of Computer Science, Boston University · Allen Institute for AI +4
Reliable reward and preference signals are critical for evaluating and optimizing large language models on open-ended tasks. Rubric-based judges offer a transparent way to decompose such judgments into explicit evaluation criteria, but existing annotation-free rubric generators typically rely on a single generic evaluator. As a result, they may overlook important dimensions of human preference, a failure mode we term dimensional blind spots. To address this limitation, we propose Multi-Role Rubric Generation (MRRG), a training-free and reference-free framework that elicits evaluation criteria from multiple complementary roles and consolidates them into an auditable rubric-based scorer. This scorer can be used both to validate pairwise preferences and to provide rewards for GRPO-style Reinforcement Learning with Verifiable Rewards (RLVR). Experiments on preference validation benchmarks show that MRRG consistently outperforms single-role rubric generation baselines across multiple backbone models. Further RLVR experiments demonstrate that MRRG yields a stronger reward signal for improving open-ended generation.
Dazhi Fu, Jiuding Yang, Yiwen Guo +1
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher
Pointwise reward modeling offers critical signals for LLM post-training, yet struggles with absolute scoring in subjective, non-verifiable settings. Rubric-based methods address this by decomposing evaluation into explicit criteria, but existing approaches typically depend on frontier LLMs and suffer from ties caused by hard Boolean aggregation. We present RUBRIC-ARROW, an alternating framework that jointly trains a rubric generator and a rubric-conditioned judge, with its RL stage using only pairwise preference data. Our method couples a probability-based scoring rule that reduces ties with phase-specific preference-based rewards and an alternating GRPO scheme that together train the pointwise evaluator. Extensive experiments show that RUBRIC-ARROW achieves competitive reward-modeling accuracy and yields consistent gains for downstream policy post-training.
Haoxiang Jiang, Zihan Dong, Tianci Liu +5
University at Albany · 2Rutgers University · 3Purdue University +3