Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL

Hyun Ryu; Jong-Heon Kim; Yiqing Xie; Seungone Kim; Carolyn Rose; Eunho Yang

cs.CLOct 24, 2026

Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL

Authors: Hyun Ryu, Jong-Heon Kim, Yiqing Xie, Seungone Kim, Carolyn Rose, Eunho Yang

Organizations: KAIST AI · Carnegie Mellon University Language Technologies Institute

Abstract

Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench).

Figures & tables

Figure 1: Reward hacking in rubric-based RL and our mitigation method. With the standard coverage reward, recall (i.e., weighted coverage of rubrics) quickly saturates, while precision and win rate against a base model continually decreases and responses become more verbose. With a precision objective included in the reward ( Fork ), the model reaches similar recall while gradually increasing precision and win rate and generating less verbose responses. Here, we train Qwen2.5-7B-Instruct on RubricHub-Medical and evaluate it on an in-domain test set.

Figure 2: Training with Fork . Given a prompt, a policy model samples a response, which is then divided into units. An LLM grader matches rubrics to units to assign each unit a necessity credit and determines a credit threshold for unit necessity. In parallel, the grader assesses the factual correctness of all units. Precision is defined as the fraction of necessary and factually correct units, and recall follows the standard definition of weighted rubric coverage. We update the policy model using their weighted harmonic mean as a reward.

	RubricHub-Medical test set					HealthBench
	Rubric-based			Rubric-free		Rubric-based	Rubric-free
	Recall	Precision	$F_{1}$	Win Rate	Length	Recall	Win Rate
Base
Qwen2.5-7B-Instruct	22.6	46.6	26.3	N/A	2,432	13.4	N/A
Rubric-Coverage Rewards
Standard	35.7	25.5	26.4	3.1	9,195	25.9	7.1

Table 1: Main results on medical domain. We conduct rubric-based and rubric-free evaluations of our method and other rubric-based RL baselines on the RubricHub-Medical test set and HealthBench. Rubric-based evaluation measures macro rubric recall, macro precision, and the macro F1 score. Precision indicates the fraction of necessary and factually correct response units. Rubric-free evaluation measures win rate against the base policy model and average response length. We use GPT-5.4 as a reference grader for automating these evaluations.

	RubricHub-Science test set					ResearchQA
	Rubric-based			Rubric-free		Rubric-based	Rubric-free
	Recall	Precision	$F_{1}$	Win Rate	Length	Recall	Win Rate
Base
Qwen2.5-7B-Instruct	27.3	41.8	29.6	N/A	2,673	48.3	N/A
Rubric-Coverage Rewards
Standard	31.4	33.5	29.2	38.6	4,453	64.4	16.1

Table 2: Main results on science domain. We evaluate rubric-based RL methods on the RubricHub-Science test set and ResearchQA as in Table 1 .

Table 5

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	Rubric Sat.	Holistic Pref.	Necessity	Factual Corr.	Threshold	Avg.
gpt-oss-20b	0.591	0.796	0.348	0.350	0.679	0.553
gpt-oss-120b	0.606	0.825	0.563	0.677	0.688	0.672
GPT-4.1 Mini	0.639	0.846	0.447	0.579	0.711	0.644
Qwen3.5-122B-A10B	0.735	0.849	0.575	0.542	0.780	0.696
Qwen3.6-35B-A3B	0.719	0.853	0.534	0.703	0.760	0.714

Appendix

Table 7: Training grader selection results. We select five candidate training graders and measure their ability to make five types of judgments required for rubric-based RL methods. We use 40 randomly sampled base policy responses from RubricHub-Medical and report Pearson correlations between candidate and reference judgments.

	Completeness	Factual Corr.	Conciseness	Relevance	Safety	Overall
Base
Qwen2.5-7B-Instruct	4.83	4.90	5.58	5.95	5.46	4.94
Rubric-Coverage Rewards
Standard	5.34	3.77	1.14	4.05	4.98	3.29
OnlineRubrics	5.68	4.05	1.79	4.49	5.39	3.95
Universal Rubrics	5.64	3.76	1.85	4.57	5.11	3.79

Appendix

Table 8: Rubric-free evaluation results across response quality dimensions. We report each quality dimension scores of our method and rubric-coverage rewards in a 1–7 Likert scale following Mahmoud et al. (2026) . We also add a base model for reference. The overall score is determined implicitly by the reference grader (i.e., GPT-5.4).

Panel of Figure 5 — Figure 5: Prompt for Fork (Rubrics to Units Matching).

Panel of Figure 7 — Figure 7: Prompt for Fork (Necessity Threshold Decision) (2/2).

Panel of Figure 8 — Figure 8: Prompt for Fork (Factual Correctness Decision) (1/2).

Panel of Figure 9 — Figure 9: Prompt for Fork (Factual Correctness Decision) (2/2).

Panel of Figure 10 — Figure 10: Prompt for Reweighted Absence-Based Rubrics (1/2).

Panel of Figure 11 — Figure 11: Prompt for Reweighted Absence-Based Rubrics (2/2).

Panel of Figure 12 — Figure 12: Prompt for Holistic Preference (1/2).

Panel of Figure 13 — Figure 13: Prompt for Holistic Preference (2/2).

Panel of Figure 14 — Figure 14: Prompt for Generic Criteria.

Panel of Figure 15 — Figure 15: Prompt for Offline Criteria (1/2).

Panel of Figure 16 — Figure 16: Prompt for Offline Criteria (2/2).

Panel of Figure 17 — Figure 17: Prompt for Online Criteria (1/2).

Panel of Figure 18 — Figure 18: Prompt for Online Criteria (2/2).

Rubric-Based RL

Rubric-based reinforcement learning uses explicit, often task-specific criteria to reward model behavior beyond simple outcome checks.

See all 104 papers

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
May 19, 2026Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5Reward Modeling Rubric-Based RL
Scale AI · Persona
Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics
Sep 30, 2026Maoqi Liu, Junwei He, Bowen Zhang +5Reward Hacking Rubric-Based RL
Beijing University of Posts and Telecommunications · ByteDance
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-Based RL
Aug 12, 2026Minglai Yang, Xinyu Guo, Zhengliang Shi +8Reward Hacking Rubric-Based RL
Scale AI · University of Arizona · Carnegie Mellon University +2

May 19, 2026J/K move · Enter open · S save

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei, Daniel George+4

Scale AI · Persona

Reward Modeling Rubric-Based RL Reinforcement Learning with Verifiable Rewards Rubric-Based Evaluation Rubric-Guided Generation Reward Design for RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins

24

30

base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in

2.5

4\times

fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.

Abstract

Rubric-Based RL

Explore similar work

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-Based RL