cs.AIMar 9, 2026
SaveCDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling
Organizations: University of Science and Technology of China · Peking University · Meituan · BeiHang University
Abstract
Reward modeling is essential for aligning Large Language Models(LLMs) with human preferences, yet conventional reward models suffer from poor interpretability and heavy reliance on costly expert annotations. While recent rubric-based approaches enhance evaluation transparency, they lack systematic quality control, yielding noisy and redundant criteria, failing to mitigate persistent biases (e.g., verbosity, position) in LLM evaluators, and creating a scalability-reliability trade-off. To address these limitations, we propose CDRRM (Contrast-Driven Rubric Reward Model), a framework built on a novel Contrast-then-Synthesis paradigm for high-quality rubric generation and guided preference judgment. CDRRM first conducts multi-dimensional contrastive profiling on preference pairs to identify causal discriminative factors, then synthesizes these insights into compact, context-aware rubrics to guide preference judg- ments. Extensive experiments on three authoritative benchmarks (RewardBench, RMBench, RMB) demonstrate that CDRRM achieves state-of-the-art performance across diverse domains and effectively mitigates aforementioned evaluation biases. Notably, our approach delivers exceptional data efficiency: training the rubric generator on only 3k high-quality samples empowers a frozen pre-trained judge model to outperform fully fine-tuned baselines. This work offers a scalable, interpretable, and data-efficient path for reward modeling.
Figures & tables
Figure 1. An illustrative example of rubric generation for a Greatest Common Divisor (GCD) task, contrasting rubrics from direct prompting (right, redundant and potentially misleading) with those from our Contrast-then-Synthesis paradigm (left, concise and effective). The bottom-left panel shows statistics on the number of generated rubrics with respect to single preferences. A short description for accessibility.
Figure 2. The CDRRM framework. (Top) The Contrast-then-Synthesis paradigm synthesizes evidence-based rubrics via contrastive analysis of preference pairs. (Bottom) These rubrics, paired with synthesized rubric-grounded justifications, supervise the training of a Rubric Generator (to automate context-aware criterion synthesis) and a Judge Model (to generate rubric-aligned justifications for precise preference predictions).
| Del | Baseline | Del 0 | Del 1 | Del 2 |
|---|---|---|---|---|
| Acc. (%) | 95.65 | 95.70 | 95.23 | 95.37 |
| (%) | 0.00 | +0.05 | -0.42 | -0.28 |
Table 1. Impact of Rubric Reduction on Accuracy. A short description of the table for accessibility.
| Models | RewardBench | RMBench | RMB | Avg. | |||||
| Overall | Easy | Medium | Hard | Overall | Help | Harm | Overall | ||
| Scalar RMs | |||||||||
| SteerLM-RM-70B | 88.8 | 48.3 | 54.9 | 54.3 | 52.5 | 57.4 | 67.3 | 62.4 | 67.9 |
| InternLM2-20B-Reward | 90.2 | 82.6 | 71.6 | 50.7 | 68.3 | 76.3 | 67.0 | 71.7 | 76.7 |
| ArmoRM-Llama3-8B-v0.1 | 90.4 | 82.2 | 71.0 | 49.8 | 67.7 | 78.7 | 66.3 | 72.5 | 76.9 |
| Skywork-Reward-Llama-3.1-8B | 92.5 | 89.0 | 74.7 | 46.6 | 70.1 | 78.1 | 75.9 | 77.0 | 79.9 |
Table 2. Main results on RewardBench, RMBench, and RMB. We report the accuracy scores for each category. The best results in each column are highlighted in bold . “Help” and “Harm” denote Helpfulness and Harmlessness, respectively. The “Average” column represents the mean of the overall scores across the three benchmarks.
| Method | RewardBench | RMBench | RMB | Average |
|---|---|---|---|---|
| Direct Judge | 79.7 | 74.7 | 74.9 | 76.4 |
| One-step Rubric Judge | 86.0 | 75.0 | 75.9 | 79.0 |
| CDRRM-8B(Base) | 90.4 | 86.1 | 80.8 | 85.8 |
Table 3. Ablation study on different judging strategies. We compare CDRRM against Direct Judge and One-step Rubric Judge. All models are based on the Qwen3-8B backbone.
Figure 3. Impact of training data size on model performance. Subplots (a) and (b) illustrate the scaling trends for the Rubric Generator and the Judge Model, respectively, demonstrating that performance stabilizes with minimal training data.
| Instruction: GIVE ME AN EXECUTIVE SUMMARY OF LOTTADIGITAL.COM |
|---|
| Response A (Chosen): |
| LottaDigital.com is a leading digital marketing agency offering SEO, PPC, social media, content, email marketing, and web design services, with a data-driven approach, expert team, client-centric focus, and proven results across various industries. |
| Response B (Rejected): |
| Executive Summary of LottaDigital.com LottaDigital.com stands as a pioneering digital marketing agency that excels in delivering comprehensive digital solutions… Core Services Offered: 1. SEO (Search Engine Optimization): We employ advanced SEO strategies… 2. PPC (Pay-Per-Click) Advertising: Leveraging platforms like Google Ads… 3. Social Media Marketing: We manage and optimize social media profiles… […] (Content Omitted for Brevity) 6. Software Development: We offer custom software development services… Key Strengths: - Data-Driven Approach: We rely on data analytics… - Client [Truncated mid-sentence] |
| Baseline: Direct Judge Analysis |
| Critique: “Response B is comprehensive, well-structured, and detailed . It provides a thorough overview… and highlights key strengths.” / “Response A lacks depth and detail…” Flaw: The model exhibits Verbosity Bias , favoring the lengthy formatting of Response B while completely overlooking the critical truncation error at the end. Prediction: ✗ Winner: Response B (Incorrect) |
Table 4. Case Study on Robustness. The Direct Judge falls into the “verbosity trap,” preferring the lengthy Response B despite it being cut off. In contrast, CDRRM generates specific hard rules targeting completeness and conciseness, correctly penalizing the truncation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Parameter | Value |
|---|---|---|
| Rubric Generator | Epochs | 1 |
| Max Length | 25000 | |
| Batch Size | 128 | |
| Optimizer | AdamW | |
| Learning Rate | 5e-5 | |
| Warmup Ratio | 0.05 |
Table 5. Hyperparameter Settings for CDRRM Components
| Models | Chat | Chat-Hard | Safety | Reasoning | Overall |
|---|---|---|---|---|---|
| Scalar RMs | |||||
| SteerLM-RM-70B | 91.3 | 80.3 | 92.8 | 90.6 | 88.8 |
| InternLM2-20B-Reward | 98.9 | 76.5 | 89.5 | 95.8 | 90.2 |
| ArmoRM-Llama3-8B-v0.1 | 96.9 | 76.8 | 90.5 | 97.3 | 90.4 |
| Skywork-Reward-Llama-3.1-8B | 95.8 | 87.3 | 90.8 | 96.2 | 92.5 |
| INF-ORM-Llama3.1-70B | 96.6 | 91.0 | 93.6 | 99.1 | 95.1 |
Table 6. Performance comparison on RewardBench. We report the accuracy (%) across four categories: Chat, Chat-Hard, Safety, and Reasoning. The best results in each column are highlighted in bold.
| Instruction: Is JavaScript a back end or front end language? |
|---|
| Response A (Chosen/Ground Truth): |
| JavaScript is primarily known as a front-end language, as it is widely used for client-side scripting in web browsers to create dynamic and interactive web pages. However, JavaScript can also be used as a back-end language through the use of runtime environments like Node.js. Node.js allows developers to run JavaScript on the server side, enabling full-stack development with a single language. |
| Response B (Rejected): |
| JavaScript is primarily a front-end language but can also be used as a back-end language with Node.js for full-stack development. |
| ### Front-End Capabilities |
| - DOM Manipulation / Event Handling / Animations and Effects with libs like jQuery/React |
Table 7. Case Study on Verbosity Bias Mitigation. Direct Judge and Rubric-Guided with improper criteria both fall for verbosity bias, preferring the truncated but lengthy Response B. Our CDRRM generates optimized hard rules targeting completeness and structural integrity , correctly penalizing Response B’s critical flaws and selecting the concise, complete Response A.
| Instruction: Write a C++ function string int_to_mini_romank(int number) to convert a positive integer ( ) to lowercase Roman numerals. |
|---|
| Response A (Chosen/Ground Truth): |
| Core Implementation (Key Part): |
| string int_to_mini_romank(int number) // Correct function name per instruction |
| { |
| vector<pair<int, string>> roman = {{1000, "m"}, {900, "cm"}, ..., {1, "i"}}; |
| string result = ""; |
Table 8. Case Study on Subtle Naming Error Identification (Code Scenario). Direct Judge and Rubric-Guided with improper criteria overlook the mandatory function name requirement in the instruction, favoring Response B with a “standard” but incorrect name. Our CDRRM generates rubrics targeting instruction-aligned naming rules , correctly identifying Response A’s compliance and Response B’s critical naming flaw.
| Instruction: A circle of radius 1 is surrounded by 4 circles of radius (centers form a square). Derive the exact value of . |
|---|
| Response A (Chosen/Ground Truth): |
| Core Geometric Modeling & Derivation: |
| 1. Centers of 4 surrounding circles form a square; distance between adjacent centers: ; |
| 2. Distance from central circle center to surrounding circle center: (key correct distance); |
| 3. Form right triangle (central center, surrounding center, square side midpoint) with legs , hypotenuse ; |
| 4. Pythagorean theorem: ; |
Table 9. Case Study on Subtle Geometric Modeling Error Identification (Math Reasoning Scenario). Direct Judge and Rubric-Guided with improper criteria fail to capture the core geometric flaw of Response B, while our CDRRM generates task-aligned rubrics targeting valid geometric modeling and answer-configuration consistency , accurately identifying the correct solution in Response A.