C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences

Akira Kawabata; Saku Sugawara

cs.CLApr 15, 2026

C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences

Authors: Akira Kawabata, Saku Sugawara

Organizations: The Graduate University for Advanced Studies (SOKENDAI) · National Institute of Informatics · The Asahi Shimbun Company · The University of Tokyo

Abstract

Rubric-augmented verification guides reward models with explicit evaluation criteria, yielding more reliable judgments than single-model verification. However, most existing methods require costly rubric annotations, limiting scalability. Moreover, we find that rubric generation is vulnerable to a failure of cooperation; low-quality rubrics actively mislead reward models rather than help. Inspired by the principle of cooperative communication, we propose Cooperative yet Critical reward modeling (C2), a framework that significantly improves reward model judgments by having the reward model critically collaborate with a rubric generator trained solely from binary preferences. In C2, we synthesize helpful and misleading rubric pairs by measuring how each rubric shifts the reward model toward or away from the correct preference. Using these contrastive pairs, we train a cooperative rubric generator to propose helpful rubrics, and a critical verifier to assess rubric validity before making its judgment, following only rubrics it deems helpful at inference time. C2 outperforms reasoning reward models trained on the same binary preferences, with gains of up to 6.5 points on RM-Bench and 6.0 points length-controlled win rate on AlpacaEval 2.0. Without external rubric annotations, C2 enables an 8B reward model to match performance achieved with rubrics from a 4 $\times$ larger model. Overall, our work demonstrates that eliciting deliberate cooperation in rubric-augmented verification makes reward models more trustworthy in a scalable way.

Figures & tables

Figure 1: We frame rubric generation and rubric-grounded verification as cooperative yet critical communication: the generator cooperatively explores rubrics to guide the verifier toward correct judgments, and the verifier critically assesses which rubrics to follow based on their outcomes.

Panel of Figure 2 — Figure 2: Impact of self-generated rubrics on RM-Bench hard subset. (a) Most rubrics produce near-zero confidence shift (distribution concentrated around $\Delta=0$ ). (b) High-quality rubrics boost accuracy while low-quality ones degrade performance below the rubric-free baseline.

Figure 3: Overview of our C2 framework. (Step 1) Helpful and misleading rubrics are synthesized by measuring their effect on verifier confidence. (Step 2) The generator is trained via DPO to produce helpful rubrics, and the verifier is trained via GRPO to judge preferences while assessing rubric quality. (Step 3) At inference, the verifier selectively follows rubrics it deems helpful and falls back to rubric-free evaluation otherwise.

Tulu3-8B-SFT
Method	RewardBench	RM-Bench	RewardBench2	JudgeBench	Avg.
Base Model	67.2 $\pm$ 0.8	56.1 $\pm$ 0.6	35.2 $\pm$ 1.3	22.7 $\pm$ 1.3	45.3 $\pm$ 0.5
Reasoning RM	73.7 $\pm$ 0.9	64.9 $\pm$ 0.5	45.6 $\pm$ 1.1	35.8 $\pm$ 0.9	55.0 $\pm$ 0.4
+ Self-Rubric	70.8 $\pm$ 0.7	64.2 $\pm$ 0.9	40.8 $\pm$ 1.0	35.2 $\pm$ 1.3	52.8 $\pm$ 0.5
+ External-Rubric (32B)	84.9 $\pm$ 0.6	77.7 $\pm$ 0.5	59.6 $\pm$ 1.3	59.2 $\pm$ 0.8	70.4 $\pm$ 0.4
C2 (Ours)	77.2 $\pm$ 0.8	65.6 $\pm$ 0.5	50.7 $\pm$ 1.2	39.8 $\pm$ 0.8	58.3 $\pm$ 0.4

Table 1: Accuracy (%) on preference prediction benchmarks. For JudgeBench, we report positional consistent accuracy. We report mean and standard deviation over 3 training seeds. Gray rows indicate the external-rubric setting using rubrics from a significantly larger model (Qwen3-32B). Best results excluding this setting are in bold.

Tulu3-8B-SFT
	AlpacaEval 2.0		Arena-Hard
Method	WR	LC	WR
Base Model	0 7.5	12.8	13.3
+ DPO w/ Reasoning RM	13.1	19.0	21.3
+ DPO w/ C2 (Ours)	18.3	25.0	26.8
Qwen3-8B

Table 2: Downstream alignment performance of policies trained with DPO. WR and LC denote raw and length-controlled win rates (%).

Panel of Figure 4 — Figure 4: Comparison of C2 and compute-matched Reasoning RM with majority voting on RewardBench. We report mean and standard deviation over 3 runs.

Panel of Figure 5 — Figure 5: Accuracy under varying proportions of high-quality vs. low-quality rubrics. Gray regions indicate gains from selective inference.

Panel of Figure 6 — Figure 6: Distribution of rubric quality scores.

Tulu3-8B-SFT
Variant	RB	RMB	RB2	Avg.
C2 (Full)	77.2	65.6	50.7	64.5
w/o Cooperative Gen.	76.8	64.8	48.3	63.3
w/o Critical Verifier	76.1	64.2	47.8	62.7
w/o Negative Rubrics	72.1	63.5	47.2	60.9
Qwen3-8B

Table 3: Ablation results (%) on RewardBench (RB), RM-Bench (RMB), and RewardBench2 (RB2).

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Rubric Generation Prompt
You are an expert AI evaluator.
Your task is to analyze a specific User Question and two Assistant Answers (Assistant A and Assistant B) to determine the most effective way to distinguish their quality.

Task Instructions:

1. Analyze:

Appendix

Figure 7: Prompt template for rubric generation used by

M_{g}

and

G_{\phi}

Rubric-Free Verification Prompt
You are given a user question and two responses from two AI assistants.
Your task is to act as an impartial judge and decide which response better follows the user’s instructions.

First, present your reasoning inside <analyze> and </analyze> tags. This should include:

The evaluation criteria for a high-quality response,

Appendix

Figure 8: Prompt template for rubric-free verification used by

M_{v}

and

V_{\theta}

Rubric-Augmented Verification Prompt
You are given a user question, two responses from two AI assistants, and a specific evaluation rubric.
Your task is to act as an impartial judge and decide which response better follows the user’s instructions.
You are provided with an evaluation rubric, but you must exercise vigilance . The provided rubric may be flawed, incomplete, or fundamentally incorrect (e.g., it might ignore the user’s core intent or guide you toward a wrong conclusion).
You must not treat the provided rubric as absolute truth if it contradicts the user’s instructions or logical reasoning.

First, conduct a deep reasoning process inside <analyze> and </analyze> tags. Your reasoning must follow these steps:

Appendix

Figure 9: Prompt template for rubric-augmented verification used by

V_{\theta}

Rubric Quality Evaluation Prompt
You are an expert evaluator assessing the quality of auto-generated rubrics for comparing two AI assistant responses.

Your Task:
Evaluate the generated rubric (including its reasoning in <analyze> tags and the criteria/rubric questions) on a 1-5 scale based on how effectively it enables accurate quality comparison between two responses.

Input:

Appendix

Figure 10: Prompt template for rubric quality evaluation using GPT-5.

Hyperparameter	Tulu3-8B	Qwen3-8B
Learning rate	5e-7	5e-7
Batch size	64	64
Mini-batch size	32	32
Number of epochs	3 (Reasoning RM) / 1 (C2)
KL loss coefficient	0.01	0.01
Number of rollouts	8	8

Appendix

Table 4: Hyperparameters for GRPO training (Verifier).

Hyperparameter	Tulu3-8B / Qwen3-8B
Learning rate	5e-7
Batch size	64
Number of epochs	3
$\beta$	0.1
Max sequence length	4096
Optimizer	AdamW

Appendix

Table 5: Hyperparameters for DPO training (Rubric Generator). The same hyperparameters are used for both Tulu3-8B and Qwen3-8B.

Benchmark	Method	$N{=}4$	$N{=}8$	$N{=}16$
GPQA-Diamond	RRM	41.4	43.4	44.4
GPQA-Diamond	C2	43.9	46.5	47.5
MATH500	RRM	88.2	89.8	90.2
MATH500	C2	90.6	92.0	92.6
LogiQA	RRM	73.0	73.3	73.3
LogiQA	C2	73.4	73.6	73.7

Appendix

Table 6: Best-of-N selection results (%) with Qwen3-8B. C2 consistently outperforms Reasoning RM (RRM) across all benchmarks and values of

N

, with gains particularly pronounced on GPQA-Diamond (+3.1 at

N{=}16

) and MATH500 (+2.4 at

N{=}16

	AlpacaEval 2.0		Arena-Hard
Method	WR	LC	WR
Reasoning RM	39.9	39.3	70.4
C2 (Ours)	42.1	41.3	72.6

Appendix

Table 7: Rejection sampling results (%) with Qwen3-8B. C2 yields +2.0 LC win rate on AlpacaEval 2.0 and +2.2 on Arena-Hard.

Model	Reasoning RM	C2
Tulu3-8B-SFT	1,551 $\pm$ 766 ms	3,628 $\pm$ 1,801 ms
Qwen3-8B	1,989 $\pm$ 981 ms	4,834 $\pm$ 2,393 ms

Appendix

Table 8: Per-example inference latency (mean

\pm

std) on RewardBench using a single A100 GPU. C2 is approximately 2.3–2.4

\times

slower than Reasoning RM due to rubric generation and the potential retry mechanism.

Error Type	Base Model	C2
Missing key constraints	17 (21.3%)	8 (10.0%)
Irrelevant criteria	21 (26.3%)	10 (12.5%)
Contradictory	3 (3.8%)	3 (3.8%)
Ambiguous	27 (33.8%)	4 (5.0%)
Over-constrained	10 (12.5%)	15 (18.8%)
No error	28 (35.0%)	42 (52.5%)

Appendix

Table 9: Error analysis of rubrics generated by the base model vs. C2 generator (Qwen3-8B). Counts and percentages are shown for each error type over 80 rubrics per model. Multiple error labels can apply to a single rubric.

Prompt: Implement a program to find the common elements in two arrays without using any extra data structures.

High-Quality Rubric (Score: 4)

Low-Quality Rubric (Score: 2)

<analyze> Intent: The user wants a program to find common elements in two arrays without using any extra data structures . The core intent is to solve the problem efficiently while adhering to the constraint of no additional memory allocation. Ideal Answer: A model answer should use a method that does not require extra memory (e.g., nested loops for unsorted arrays or two-pointer technique for sorted arrays), avoid using lists, sets, or dictionaries to store intermediate results, and provide a working code example. </analyze> <criteria_1> Instruction-following </criteria_1> <rubric_1> Does the response avoid using any extra data structures (e.g., lists, sets, dictionaries) to store intermediate or final results, strictly adhering to the “no extra data structures” constraint? </rubric_1> <criteria_2> Helpfulness </criteria_2> <rubric_2> Does the response provide a working code example that correctly identifies common elements while following the constraint? </rubric_2> <criteria_3> Completeness </criteria_3> <rubric_3> Does the response include all necessary details (e.g., algorithm logic, code, explanation) to solve the problem? </rubric_3>

<analyze> The user’s intent is to implement a program that finds common elements without extra data structures. Assistant B offers a nested loop implementation but uses a list to store results, which might violate the constraint. However, the result list is necessary for output, so it may be acceptable. </analyze> <criteria_1> Helpfulness </criteria_1> <rubric_1> Does the response provide a working implementation that correctly identifies common elements? </rubric_1> <criteria_2> Completeness </criteria_2> <rubric_2> Does the response include all necessary code, including the use of a result list to store common elements? </rubric_2> <criteria_3> Instruction-following </criteria_3> <rubric_3> Does the response avoid using structures like lists for storing results, even if they are required for output? </rubric_3>

Appendix

Figure 11: High-quality vs. low-quality rubrics for implementing common element detection without extra data structures (Qwen3-8B). The high-quality rubric clearly defines the “no extra data structures” constraint and derives consistent evaluation criteria from it. The low-quality rubric shows ambiguous interpretation of the constraint, stating that violating it “may be acceptable,” leading to contradictory criteria that simultaneously require and forbid using a result list.

Prompt: I’ve put a plate on top of an apple, and take the plate from kitchen to dinning room. Where is the apple now

High-Quality Rubric (Score: 4)

Low-Quality Rubric (Score: 1)

<analyze> Intent: The user wants to know the location of the apple after it has been moved along with a plate from the kitchen to the dining room. Ideal Answer: A model answer should clearly state that the apple remains in the kitchen while the plate is moved to the dining room. The apple’s location is dictated by the plate’s position when it was initially placed on top of it, rather than by subsequent movements of the plate. </analyze> <criteria_1> Completeness </criteria_1> <rubric_1> Does the response accurately describe the apple’s final location and mention that the apple did not move with the plate? </rubric_1> <criteria_2> Safety </criteria_2> <rubric_2> Does the response avoid any content that may imply the apple is in the dining room or mislead the user? </rubric_2> <criteria_3> Helpfulness </criteria_3> <rubric_3> Does the response provide a clear explanation that directly addresses the user’s question about the apple’s location? </rubric_3>

<analyze> Intent: The user is asking about the location of the apple in relation to the plate after moving the plate. Ideal Answer: A model answer should specify that the apple remains in the kitchen, directly under the plate, since the apple was concealed underneath the plate and did not move. </analyze> <criteria_1> Helpfulness </criteria_1> <rubric_1> Does the response accurately explain the final location of the apple? </rubric_1> <criteria_2> Completeness </criteria_2> <rubric_2> Does the response provide a full explanation including initial and final positions of both objects? </rubric_2> <criteria_3> Safety </criteria_3> <rubric_3> Does the response avoid introducing any dangerous, harmful, or false information about relative positions? </rubric_3>

Appendix

Figure 12: High-quality vs. low-quality rubrics for a spatial reasoning question (Tulu3-8B-SFT). The high-quality rubric correctly interprets “plate on top of apple” and reasons that moving the plate leaves the apple stationary in the kitchen. The low-quality rubric misunderstands the physical configuration (claiming the apple is “concealed underneath the plate”), and inappropriately applies the Safety criterion to object positions rather than harmful content.

Prompt: Let $k$ and $m$ be real numbers, and suppose that the roots of the equation $x^{3}-7x^{2}+kx-m=0$ are three distinct positive integers. Compute $k+m.$
Assistant A (Chosen Response):
Given the polynomial $x^{3}-7x^{2}+kx-m=0$ with roots $a$ , $b$ , $c$ as distinct positive integers, by Vieta’s formulas: $a+b+c=7$ , $ab+bc+ca=k$ , $abc=m$ . Since $a$ , $b$ , $c$ are distinct positive integers summing to 7, the only solution is $\{1,2,4\}$ . Thus: $k=1\cdot 2+2\cdot 4+4\cdot 1=14$ , $m=1\cdot 2\cdot 4=8$ . Therefore, $k+m=14+8=\mathbf{22}$ .
Assistant B (Rejected Response):
Given the polynomial with roots as distinct positive integers, by Vieta’s formulas: $a+b+c=7$ . The smallest distinct positive integers are $1,2,3$ . Using these: $k=1\cdot 2+2\cdot 3+3\cdot 1=11$ , $m=1\cdot 2\cdot 3=6$ . Therefore, $k+m=11+6=\mathbf{17}$ .
Base Model (Tulu3-8B-SFT, Score: 2)
<analyze> Intent: The User Question seeks to find the values of $k$ and $m$ in the polynomial equation given that the roots are three distinct positive integers. The goal is to compute $k+m$ . Comparison: Assistant A correctly identifies the roots as 1, 2, and 4 and computes $k+m$ as 22. Assistant B correctly identifies the roots as 1, 2, and 3 and computes $k+m$ as 17. Assistant A’s answer is more accurate because it correctly matches the polynomial’s roots with the given constraints. </analyze> <criteria_1> Completeness </criteria_1> <rubric_1> Does the answer include all necessary steps to derive the values of $k$ and $m$ ? </rubric_1> <criteria_2> Helpfulness </criteria_2> <rubric_2> Does the answer directly address the user’s intent and provide a correct and relevant solution? </rubric_2>

Appendix

Figure 13: Rubric comparison across Tulu3 family models. The base model produces generic rubrics (“Completeness” and “Helpfulness”) that fail to target the critical constraint: roots must sum to 7. The C2 generator explicitly identifies this constraint in its criteria, enabling accurate discrimination between correct (roots 1, 2, 4) and incorrect (roots 1, 2, 3) solutions.

Prompt: The greatest common divisor of positive integers $m$ and $n$ is 8. The least common multiple of $m$ and $n$ is 112. What is the least possible value of $m+n$ ?
Assistant A (Chosen Response):
Using $\gcd(m,n)\cdot\text{lcm}(m,n)=m\cdot n$ , we get $8\cdot 112=896$ , so $m\cdot n=896$ . Let $m=8a$ and $n=8b$ where $\gcd(a,b)=1$ . Then $64ab=896$ , giving $ab=14$ . The coprime pairs $(a,b)$ with product 14 are: $(1,14)$ and $(2,7)$ . For $(1,14)$ : $m+n=8+112=120$ . For $(2,7)$ : $m+n=16+56=72$ . The minimum is $\mathbf{72}$ .
Assistant B (Rejected Response):
Using $\gcd(m,n)\cdot\text{lcm}(m,n)=m\cdot n$ , we get $m\cdot n=896$ . Let $m=8a$ and $n=8b$ where $\gcd(a,b)=1$ . Then $ab=14$ . The pairs $(a,b)$ satisfying this are $(1,14)$ and $(14,1)$ . For both cases: $m+n=8+112=120$ . Therefore, the minimum is $\mathbf{120}$ .
Base Model (Qwen3-8B, Score: 3)
<analyze> Intent: The user wants the minimal sum of two numbers with specific GCD and LCM. The key here is to use the relationship between GCD and LCM. Ideal Answer: Must enumerate all coprime pairs of $a$ and $b$ that multiply to 14, compute $m+n$ for each pair and select the minimum. </analyze> <criteria_1> Completeness </criteria_1> <rubric_1> Does the answer include all valid coprime pairs $(a,b)$ such that $ab=14$ , including $(1,14),(2,7),(7,2),(14,1)$ ? </rubric_1> <criteria_2> Helpfulness </criteria_2> <rubric_2> Does the answer correctly identify the minimal possible value of $m+n$ as 72, or does it provide an incorrect value (e.g., 120)? </rubric_2> <criteria_3> Instruction-following </criteria_3> <rubric_3> Does the answer adhere to the mathematical principles and logical steps required to solve the problem? </rubric_3>

Appendix

Figure 14: Rubric comparison across Qwen3 family models. All three models correctly identify the key issue—the missing

(2,7)

coprime pair—but differ in rubric specificity. The C2 generator produces the most focused criteria by directly asking whether all coprime pairs are enumerated and whether the correct minimum (72) is identified, without redundant criteria.

Case 1: Correctly Following a Helpful Rubric (Qwen3-8B)
Prompt: How can I make bubble solution?
Response A:
Create bubble solution by mixing water, vinegar, and optional glycerin or corn syrup, then use a homemade wand to blow bubbles for a fun activity.
Response B:
Making your own bubble solution at home is a fun and easy activity for kids and adults alike. Here are a few simple recipes to get you started. Basic Bubble Solution Recipe. Ingredients: 6 cups of water, 1/2 cup of dish soap (like Dawn or Joy), 2 tablespoons of glycerin (optional, but helps create longer-lasting bubbles). Instructions: Mix the water and dish soap in a large container. If using glycerin, add it to the mixture and stir well. Let the solution sit for a few hours or overnight to allow the soap to dissolve fully and the bubbles to become more stable. Use a bubble wand or a homemade wand (like a wire hanger bent into a circle) to blow bubbles. Advanced Bubble Solution Recipe. Ingredients: 4 cups of water, 1/2 cup of dish soap, 1/4 cup of corn syrup (helps create larger bubbles), 1 tablespoon of baking powder (optional, for extra strength). Instructions: Combine the water, dish soap, and corn syrup in a large container. If using baking powder, add it to the mixture and stir until dissolved. Let the solution sit for a few hours or overnight. Use a bubble wand to blow bubbles. Homemade Bubble Wands. If you don’t have a bubble wand, you can easily make one. Wire Hanger Wand: Straighten a wire hanger and bend one end into a circle. Twist the ends together to secure the circle. Yarn Wand: Wrap yarn around a wire hanger or a stick to create a fluffy wand that can catch more solution. Plastic Ring Wand: Use the ring from a milk jug or any other plastic ring. Dip it into the solution and blow through the center. Tips for Blowing Bubbles. Warm Solution: Warm water can help create better bubbles. Gentle Blowing: Blow gently to create larger, longer-lasting bubbles. Outdoor Play: Bubbles are best enjoyed outdoors where there is less chance of them popping quickly due to dust or drafts. Enjoy your bubble-blowing adventure!
Generated Rubric:

Appendix

Figure 15: The verifier correctly identifies a helpful rubric that highlights the accuracy criterion (avoiding vinegar as a misleading ingredient). By following this rubric, the verifier selects Response B, which provides correct ingredients and detailed instructions, over Response A, which incorrectly includes vinegar.

Case 2: Correctly Rejecting a Misleading Rubric (Tulu3-8B-SFT)
Prompt: Write a C++ function vector<string> words_string(string s) to solve the following problem: You will be given a string of words separated by commas or spaces. Your task is to split the string into words and return a vector of the words. For example: words_string("Hi, my name is John") == {"Hi", "my", "name", "is", "John"}
Response A:
To solve the problem of splitting a string into words separated by commas or spaces, you can use the following C++ code: #include <iostream> #include <vector> #include <string> #include <sstream> using namespace std; vector<string> words_string(string s) { vector<string> words; stringstream ss(s); string word; while (getline(ss, word, ’,’)) { stringstream word_ss(word); string token; while (word_ss >> token) { words.push_back(token); } } return words; } This function uses a two-step process to split the string: first by commas, then by spaces within the substrings.
Response B:
To solve the problem of splitting a string into words separated by commas or spaces, we need to implement a function that can handle both delimiters. Here’s a complete implementation in C++: #include <vector> #include <string> #include <sstream> #include <algorithm> using namespace std; vector<string> words_string(string s) { vector<string> words; stringstream ss(s); string word; while (ss >> word) { word.erase(remove(word.begin(), word.end(), ’,’), word.end()); words.push_back(word); } return words; } This function extracts words from the stream one by one and removes any commas from the extracted word before adding it to the vector.
Rubric:

Appendix

Figure 16: The verifier correctly identifies that the rubric misinterprets the problem requirements by treating commas as possible word characters rather than delimiters. By rejecting the misleading criterion and falling back to evaluating delimiter handling, the verifier makes the correct judgment.

Case 3: Incorrectly Rejecting a Helpful Rubric (Tulu3-8B-SFT)
Prompt: I live in Scotland and would like to know the most cost effective way to heat my house considering the rise in both gas and electric prices. I have a gas powered boiler which can provide central heating to the whole house, but I could also use electric heaters to heat individual rooms. Would it be cheaper to heat an individual room using electricity, or the whole using gas? The radiators are hidden behind shrouds so I am unable to turn off the heating in individual rooms.
Response A:
Heating individual rooms with electric heaters may be more cost-effective and environmentally friendly than using a gas boiler to heat the entire house, especially given the rise in energy prices and the inability to control individual radiators.
Response B:
Given the current rise in both gas and electric prices, it’s important to consider the efficiency and cost of both heating methods. Here are some factors to consider: First, Energy Efficiency: Modern gas boilers are generally very efficient, often operating at around 90-94% efficiency. This means that for every unit of gas used, you get close to 1 unit of heat energy. Electric heaters are 100% efficient because all the electricity consumed is converted directly into heat. However, electricity is typically more expensive per unit than gas. Second, Cost Comparison: The cost of gas is generally lower per unit of energy compared to electricity. However, the exact cost will depend on your energy supplier and the current market prices. Electricity is more expensive per unit of energy, but if you only need to heat a single room, the cost might be lower than heating the entire house with gas. Third, Heating Individual Rooms vs. Whole House: If you use your gas boiler to heat the entire house, you will be paying for the gas used to heat all the radiators, even if some rooms are not occupied. This can be inefficient if you only need to heat a few rooms. Using electric heaters to heat individual rooms can be more cost-effective if you only need to heat a specific area. You can turn off the gas central heating and use electric heaters in the rooms you are using. Fourth, Practical Considerations: Since your radiators are hidden behind shrouds, you can’t easily control the heating in individual rooms. This means you might end up heating rooms you don’t need to, which is inefficient. If possible, consider installing Thermostatic Radiator Valves (TRVs) on your radiators. These valves allow you to control the temperature in each room individually, making it easier to manage heating costs. Fifth, Alternative Solutions: Consider using a smart thermostat that allows you to control the heating remotely and set different temperatures for different times of the day. This can help you optimize your heating usage.
Rubric:

Appendix

Figure 17: The verifier incorrectly rejects a helpful rubric that appropriately emphasizes cost comparison and practical solutions. Despite the rubric correctly identifying Response B as more comprehensive, the verifier dismisses it and selects the brief, unsupported Response A.

Case 4: Incorrectly Trusting a Misleading Rubric (Qwen3-8B)
Prompt:
Write a C++ function vector<int> make_a_pile(int n) to solve the following problem: Given a positive integer n, you have to make a pile of n levels of stones. The first level has n stones. The number of stones in the next level is: - the next odd number if n is odd. - the next even number if n is even. Return the number of stones in each level in a vector… Examples: make_a_pile(3) $\rightarrow$ {3, 5, 7}
Response A:
To solve this problem, we need to create a function make_a_pile … Here’s the implementation: vector<int> make_a_pile(int n) {
vector<int> pile; int current = n;
while (current > 0) {

Appendix

Figure 18: An example of a verification failure caused by model hallucination. Although the generated rubric correctly analyzed the code (identifying Assistant A as having the infinite loop), the verifier hallucinated the content of the responses: it explicitly stated “Assistant A uses a for loop” and “Assistant B uses a while loop,” effectively swapping the models. This led to the selection of the incorrect response despite the rubric’s accurate guidance.

Explore similar work

Mar 9, 2026

CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling

Reward modeling is essential for aligning Large Language Models(LLMs) with human preferences, yet conventional reward models suffer from poor interpretability and heavy reliance on costly expert annotations. While recent rubric-based approaches enhance evaluation transparency, they lack systematic quality control, yielding noisy and redundant criteria, failing to mitigate persistent biases (e.g., verbosity, position) in LLM evaluators, and creating a scalability-reliability trade-off. To address these limitations, we propose CDRRM (Contrast-Driven Rubric Reward Model), a framework built on a novel Contrast-then-Synthesis paradigm for high-quality rubric generation and guided preference judgment. CDRRM first conducts multi-dimensional contrastive profiling on preference pairs to identify causal discriminative factors, then synthesizes these insights into compact, context-aware rubrics to guide preference judg- ments. Extensive experiments on three authoritative benchmarks (RewardBench, RMBench, RMB) demonstrate that CDRRM achieves state-of-the-art performance across diverse domains and effectively mitigates aforementioned evaluation biases. Notably, our approach delivers exceptional data efficiency: training the rubric generator on only 3k high-quality samples empowers a frozen pre-trained judge model to outperform fully fine-tuned baselines. This work offers a scalable, interpretable, and data-efficient path for reward modeling.

Dengcan Liu, Fengkai Yang, Xiaohan Wang +7

University of Science and Technology of China · Peking University · Meituan +1

Reward Modeling Rubric-Based Evaluation Rubric-Guided Generation Bias in LLM-as-a-Judge

Ask PDF

Oct 13, 2026

Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges

Rubric-based reward models (RMs) evaluate responses against explicit criteria. However, existing methods often derive criteria from observed response candidates, introducing response-induced bias that may limit rubric generalizability. Their judges also struggle with mechanical constraints when relying on reading alone, and commonly used pairwise judges provide only relative preferences, complicating their direct use for policy optimization. We propose ReVeR, a pointwise rubric RM that derives hard rules and graded principles by reasoning about the instruction alone and evaluates responses independently through tool-augmented judging. Both components are trained through label-blind distillation, in which preference labels only filter a teacher's outputs, yielding ReVeR-Traces, a large-scale dataset of rubric-reasoning and judging trajectories. Experiments demonstrate that ReVeR outperforms pairwise rubric RMs in average preference accuracy, most clearly on instruction following, as well as in test-time scaling and downstream policy optimization. We further find that generator RL improves preference accuracy while reducing rubric fidelity, and that higher preference accuracy does not necessarily translate into better downstream policies. These findings highlight that rubric fidelity, preference accuracy, and downstream utility are distinct properties, shedding light on future RM research.

Tianci Liu, Haoyu Wang

University of Tennessee · University at Albany

Reward Modeling Rubric-Based Evaluation LLM-as-a-Judge

Ask PDF

May 19, 2026

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins

24

30

base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in

2.5

4\times

fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.

Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5

Scale AI · Persona

Reward Modeling Rubric-Based RL Reinforcement Learning with Verifiable Rewards Rubric-Based Evaluation Rubric-Guided Generation Reward Design for RL

Ask PDF