Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics

Maoqi Liu; Junwei He; Bowen Zhang; Feiran Li; Wentao Ma; Rongyi Lin; Shuhan Zhong; Quan Fang

cs.LGSep 30, 2026

Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics

Authors: Maoqi Liu, Junwei He, Bowen Zhang, Feiran Li, Wentao Ma, Rongyi Lin, Shuhan Zhong, Quan Fang

Organizations: Beijing University of Posts and Telecommunications · ByteDance

Abstract

Rubric-based reinforcement learning (Rubric-RL) trains language models where no verifier exists. A judge checks each criterion of a rubric, and the verdicts are aggregated into a reward, most often by a weighted sum. We show that this additive aggregation is the weak point. Under a sum, criteria compensate for one another: a policy that misses the one decision that matters can buy the points back with advice nobody asked for. On clinical consultation, such a policy scores higher and answers worse. Rubric coverage rises while appropriateness on held-out physician criteria falls below the untrained model. The medical criteria are not to blame. Grouped so that they must hold together, the same criteria, unchanged to the word, recover a third of the loss; shorter answers recover almost none. We therefore propose Protocol-level Rubrics (ProRubric), which keeps what the criteria ask for and changes how they are aggregated. It groups a checklist into a few protocol-level dimensions. A dimension counts only when all of its criteria hold and its failure clause does not fire. The grouping is done once, offline, and leaves the optimizer unchanged. ProRubric raises appropriateness by 10.8 points without losing coverage and has the best seven-benchmark average at both scales. Reward validity is set not only by what a rubric verifies, but by how it aggregates. Code is available at https://github.com/Estrellajer/ProRubric

Figures & tables

Figure 1: Comparison of reward verification paradigms in clinical protocol RL. Top: a real prompt and a Rubric-RL response in which the call to 911 first appears after 2,165 characters. (a) Conventional additive Rubric-RL scores atomic criteria independently, rewarding superficial coverage while masking critical omissions. (b) ProRubric counts a group of criteria only when all of them hold, so cheap criteria can no longer offset a missed critical one.

Figure 2: Reward hacking under additive aggregation. Coverage vs. appropriateness across the medical 4B models.

(a) After Rubric-RL training: untrained answer preferred (%)			(b) Before training: reward change per edit
Domain	DeepSeek-V4-Pro	GPT-5.6-luna	Edit to the answer	Rubric-RL	raw-AND	ProRubric
Medicine	$\mathbf{65.7}$ $\pm$ 13.2	$\mathbf{93.6}$ $\pm$ 2.3	Name an item	$\mathbf{+7.3}$	$-0.3$	$+2.2$
Science	$1.2$ $\pm$ 1.2	$\mathbf{90.3}$ $\pm$ 9.0	Name the topic	$-0.4$	$+0.6$	$-3.8$
Dialogue	$24.3$ $\pm$ 5.8	$44.9$ $\pm$ 6.8	Add a needless test	$+0.3$	$-1.4$	$\mathbf{-5.8}$
Writing	$0.0$ $\pm$ 0.0	$1.4$ $\pm$ 1.6	Drop the key advice	$-0.9$	$-0.8$	$-2.2$

Table 1: Evidence of reward hacking without a rubric and before training. (a) Blind pairwise win rates against the untrained Base (bold denotes Base preferred). (b) Reward change per edit.

Method	Writing		Dialogue	Clinical Medicine		Scientific Reasoning		Overall
Method	WritingBench	Creative-v3	Arena-Hard	HealthBench	MedQA	GPQA	ResearchQA	Average
Qwen3-4B
Base	49.2	27.9	12.9	26.1	65.2	41.1	51.8	39.2
+ SFT	63.9 $+$ 14.7	35.5 $+$ 7.6	21.6 $+$ 8.7	35.1 $+$ 9.1	59.5 $-$ 5.7	36.0 $-$ 5.1	63.2 $+$ 11.4	45.0 $+$ 5.8
+ OPSD	49.7 $+$ 0.4	27.7 $-$ 0.1	9.2 $-$ 3.7	25.1 $-$ 0.9	60.8 $-$ 4.4	36.4 $-$ 4.7	47.5 $-$ 4.3	36.6 $-$ 2.5
+ Rubric-RL	64.6 $+$ 15.3	35.0 $+$ 7.2	17.3 $+$ 4.4	31.9 $+$ 5.8	62.6 $-$ 2.6	44.7 $+$ 3.6	62.9 $+$ 11.1	45.6 $+$ 6.4

Table 2: Cross-domain evaluation results at 4B and 8B scales. Deltas are relative to the untrained Base; bold and underline mark the best and second-best results.

Reward	$\bm{c}$	$\bm{g}$
ProRubric variants
ProRubric	36.8 $\pm$ 2.7	35.0 $\pm$ 1.4
$-$ failure clauses	34.5 $\pm$ 1.1	34.9 $\pm$ 0.3
partial credit (Graded)	36.3 $\pm$ 1.8	35.1 $\pm$ 0.9
one dimension ( $K{=}1$ )	36.9 $\pm$ 1.9	33.5 $\pm$ 1.2
Same criteria, other aggregation

Table 3: Medical ablations (4B).

c

: appropriateness;

g

: coverage; both by DeepSeek-V4-Pro, averaged over three seeds on the common set (untrained Base:

54.1

26.6

Figure 3: Analysis of confounding factors and cross-domain generalization. (a) Length matching does not restore appropriateness. (b) KL penalty. (c) Appropriateness retention across domains.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Rubric-RL / ProRubric	RuscaRL	OPSD	SFT
Training steps	300	300	485	1,224 (3 epochs)
Prompts per batch	64	512	128	64
Responses per prompt	8	1	1	—
Mini-batch size	32	256	32	—
Learning rate	$10^{-6}$	$10^{-6}$	$4.2\times 10^{-6}$	$10^{-5}$
Prompt limit	4,096	6,144	4,096	20,000

Appendix

Table 4: Training hyperparameters and generation budgets of each method.

Metric	Emergency referrals	Global health	Communi- cation	Context seeking	Hedging	Health data	Complex responses	All criteria
Criteria ( $n$ )	6	3	4	4	9	4	4	34
DeepSeek-V4-Pro	0.65	0.69	0.63	0.61	0.61	0.71	0.45	0.62
Physicians	0.65	0.65	0.62	0.64	0.66	0.74	0.57	0.65

Appendix

Table 5: Agreement with physicians on HealthBench’s meta-evaluation (balanced F1).

Medicine	Science	Writing	Dialogue
HealthBench-consensus (3,140)	GPQA-Diamond (198)	WritingBench (1,000)	Arena-Hard v2 (748)
HealthBench-full, lite (4,392)	ResearchQA (702)	Creative-v3 (96)
HealthBench-full, pro (4,138)
MedQA (1,273)

Appendix

Table 6: Sample sizes of common evaluation sets across benchmarks and judges.

	DeepSeek-V4-Pro		GPT-5.6-luna
Domain	W / T / L	Win rate (%)	W / T / L	Win rate (%)
Base vs. Rubric-RL
Medicine	136 / 93 / 71	65.7 $\pm$ 13.2	229 / 55 / 16	93.6 $\pm$ 2.3
Science	3 / 35 / 262	1.2 $\pm$ 1.2	151 / 134 / 15	90.3 $\pm$ 9.0
Dialogue	45 / 117 / 138	24.3 $\pm$ 5.8	81 / 114 / 99	44.9 $\pm$ 6.8
Writing	0 / 8 / 292	0.0 $\pm$ 0.0	4 / 30 / 266	1.4 $\pm$ 1.6

Appendix

Table 7: Full win/tie/loss counts and win rates for rubric-free pairwise comparisons.

Figure 4: Response length and budget constraints. (a) Appropriateness vs. mean response length (filled: DeepSeek-V4-Pro; open: Doubao-lite). (b) Pairwise win rates at a 2,048-token budget (DeepSeek-V4-Pro; vs. Rubric-RL: three seeds, others: seed 42).

Benchmark	Rubric-RL		ProRubric		Paired $\bm{\Delta}$
Benchmark	per seed	mean	per seed	mean	Paired $\bm{\Delta}$
Writing
WritingBench	66.3 / 63.6 / 63.7	64.6 $\pm$ 1.5	67.9 / 66.8 / 66.5	67.1 $\pm$ 0.7	+2.50 $\pm$ 0.85
Creative-v3	36.5 / 34.1 / 34.5	35.0 $\pm$ 1.3	39.1 / 36.8 / 37.4	37.7 $\pm$ 1.2	+2.73 $\pm$ 0.17
Dialogue
Arena-Hard	18.2 / 16.1 / 17.4	17.3 $\pm$ 1.1	18.4 / 17.5 / 19.8	18.6 $\pm$ 1.1	+1.29 $\pm$ 1.07

Appendix

Table 8: Per-seed cross-domain evaluation results at 4B scale (seeds 42, 43, and 44).

Method	$\bm{c_{\mathrm{pro}}}$ per seed	$\bm{c_{\mathrm{lite}}}$ per seed	$\bm{g_{\mathrm{lite}}}$ per seed
Baselines
Rubric-RL	24.3 / 28.4 / 25.1 $\pm$ 2.2	69.9 / 71.3 / 68.3 $\pm$ 1.5	55.7 / 56.1 / 55.4 $\pm$ 0.4
RuscaRL	24.6 / 25.8 / 29.5 $\pm$ 2.6	66.4 / 70.3 / 71.9 $\pm$ 2.8	54.2 / 56.4 / 55.6 $\pm$ 1.1
OPSD	40.7 / 42.9 / 41.4 $\pm$ 1.1	64.8 / 67.9 / 66.3 $\pm$ 1.6	35.4 / 36.9 / 36.5 $\pm$ 0.8
Aggregation changed
Implicit	36.7 / 36.6 / 36.8 $\pm$ 0.1	78.0 / 77.5 / 77.7 $\pm$ 0.2	56.4 / 56.1 / 55.6 $\pm$ 0.4

Appendix

Table 9: Per-seed medical ablation results at 4B scale under both judges. Subscripts: pro, DeepSeek-V4-Pro; lite, Doubao-lite.

Configuration	Reward structure		Score				Len.	$n$
Configuration	grouped	appr. crit.	$\bm{g_{\mathrm{lite}}}$	$\bm{g_{\mathrm{pro}}}$	$\bm{c_{\mathrm{lite}}}$	$\bm{c_{\mathrm{pro}}}$	Len.	$n$
Baselines
Base	–	–	38.1	26.6	78.0	54.1	3.0k	1
Rubric-RL	$\circ$	$\circ$	55.8 $\pm$ 0.4	32.1	69.8 $\pm$ 1.5	26.0 $\pm$ 2.2	11.7k	3
RuscaRL	$\circ$	$\circ$	55.4 $\pm$ 1.1	32.1	69.5 $\pm$ 2.8	26.7 $\pm$ 2.6	11.9k	3
OPSD	–	–	36.3 $\pm$ 0.8	25.6	66.3 $\pm$ 1.6	41.7 $\pm$ 1.1	4.7k	3

Appendix

Table 10: Full medical ablations under both judges (4B). Subscripts: pro, DeepSeek-V4-Pro; lite, Doubao-lite. Appr. crit.: the fixed appropriateness criterion (Appendix D.3 ); Len.: mean response length in characters;

n

: number of seeds.

	Baselines		Aggregation changed				+ Appr. crit.		KL penalty
Metric	Base	Rubric-RL	Implicit	raw-AND	Graded	ProRubric	Rubric-RL	ProRubric	Rubric-RL	Rubric-RL	ProRubric
							+ appr.	+ appr.	$\beta{=}0.01$	$\beta{=}0.04$	$\beta{=}0.01$
Score	33.7	14.4	21.6	22.4	20.1	20.3	18.0	22.6	20.2	29.2	28.5
Items	3,655	3,670	3,652	3,669	3,670	3,671	3,668	3,669	3,652	3,654	3,653

Appendix

Table 11: Cross-family ranking validation using GPT-5.6-luna (3,652–3,671 items per method).

	DeepSeek-V4-Pro			Doubao-lite
Seed	Rubric-RL	ProRubric	$\bm{\Delta}$	Rubric-RL	ProRubric	$\bm{\Delta}$
42	24.3	39.8	+15.5	69.9	78.0	+8.1
43	28.4	36.2	+7.7	71.3	74.9	+3.6
44	25.1	34.5	+9.4	68.3	74.9	+6.6
Mean	26.0	36.8	+10.8	69.8	75.9	+6.1

Appendix

Table 12: Per-seed HealthBench-consensus results at 4B (

\Delta=\text{ProRubric{}}-\text{Rubric-RL}

Edit to the answer	Rubric-RL	raw-AND	Graded	ProRubric	Rubric-RL	ProRubric	Implicit	Writing
Edit to the answer	Rubric-RL	raw-AND	Graded	ProRubric	+ appr.	+ appr.	Implicit	(Rubric-RL)
Name an item	$\mathbf{+7.3}$	$-0.3$	$+1.8$	$+2.2$	$+3.0$	$-3.9$	$+1.4$	$+1.3$
	[ $+5.9$ , $+8.7$ ]	[ $-1.9$ , $+1.4$ ]	[ $+0.8$ , $+2.8$ ]	[ $-0.1$ , $+4.5$ ]	[ $+1.5$ , $+4.5$ ]	[ $-6.4$ , $-1.4$ ]	[ $-0.2$ , $+2.8$ ]	n.s.
Name the topic	$-0.4$	$+0.6$	$-1.2$	$-3.8$	$-1.6$	$-7.6$	$-1.7$	$-0.3$
	[ $-1.7$ , $+0.8$ ]
Satisfy a criterion in part	$+2.2$	$+1.1$	$+1.8$	$+3.3$	$+1.9$	$+0.4$	$+0.9$	–

Appendix

Table 13: Reward change per edit before training, for every aggregation. + appr.: with the fixed appropriateness criterion (Appendix D.3 ).

Figure 5: Appropriateness during training. Medical 4B, 300 HealthBench-consensus prompts; three-seed mean

\pm

standard deviation, step 0 is the untrained model.

	Medicine		Dialogue		Science		Writing
Method	Score	$\bm{\Delta}$ [95% CI]	Score	$\bm{\Delta}$ [95% CI]	Score	$\bm{\Delta}$ [95% CI]	Score	$\bm{\Delta}$ [95% CI]
Base	50.0	$+31.1^{*}$ [ $+27.0$ , $+35.1$ ]	45.6	$+7.4^{*}$ [ $+3.6$ , $+11.1$ ]	52.2	$+11.2^{*}$ [ $+8.4$ , $+14.4$ ]	38.9	$-19.5^{*}$ [ $-23.9$ , $-15.1$ ]
Rubric-RL	18.9	—	38.2	—	41.0	—	58.4	—
ProRubric	39.2	$+20.4^{*}$ [ $+16.9$ , $+23.9$ ]	44.9	$+6.6^{*}$ [ $+3.5$ , $+9.9$ ]	51.0	$+10.0^{*}$ [ $+7.4$ , $+12.6$ ]	64.1	$+5.8^{*}$ [ $+3.2$ , $+8.2$ ]
raw-AND	29.9	$+11.0^{*}$ [ $+8.0$ , $+14.1$ ]	45.9	$+7.6^{*}$ [ $+4.4$ , $+10.9$ ]	43.6	$+2.6^{*}$ [ $+0.1$ , $+5.2$ ]	59.9	$+1.5$ [ $-1.2$ , $+4.2$ ]
Implicit, seed 42	–	–	43.8	$+5.5^{*}$ [ $+2.5$ , $+8.8$ ]	44.2	$+3.2^{*}$ [ $+0.8$ , $+5.9$ ]	66.0	$+7.6^{*}$ [ $+5.4$ , $+9.9$ ]

Appendix

Table 14: Cross-domain appropriateness: scores and differences from Rubric-RL across four domains.

(a) Appropriateness change from the untrained model
Domain	Untrained	Rubric-RL		ProRubric		ProRubric $-$ Rubric-RL
	score	42 / 43 / 44	mean	42 / 43 / 44	mean	mean (lowest seed)
Medicine	50.0	$-31.1$ / $-26.5$ / $-29.0$	$-28.9$ $\pm$ 2.3	$-10.8$ / $-16.6$ / $-19.5$	$-15.6$ $\pm$ 4.5	$+13.3$ ( $+9.5$ )
Science	52.2	$-11.2$ / $-11.6$ / $-10.8$	$-11.2$ $\pm$ 0.4	$-1.2$ / $-6.6$ / $-3.2$	$-3.7$ $\pm$ 2.7	$+7.5$ ( $+5.0$ )
Dialogue	45.6	$-7.4$ / $-9.8$ / $-4.1$	$-7.1$ $\pm$ 2.8	$-0.8$ / $-0.2$ / $+0.2$	$-0.3$ $\pm$ 0.5	$+6.8$ ( $+4.4$ )
Writing	38.9	$+19.5$ / $+13.9$ / $+14.6$	$+16.0$ $\pm$ 3.1	$+25.2$ / $+25.0$ / $+22.6$	$+24.3$ $\pm$ 1.4	$+8.3$ ( $+5.8$ )

Appendix

Table 15: Cross-domain appropriateness over three seeds, and the incentive audit in every domain. (a) Change from the untrained model on the 200 prompts per domain of Table 14 , seeds 42 / 43 / 44, mean

\pm

s.d.; the last column pairs each seed with the Rubric-RL model of the same seed. (b) The untrained model’s answers to 300 training prompts per domain, perturbed as in Section 3.3 : reward change (

\times 100

) under the Rubric-RL reward for naming an item and for adding unrequested substance; the naming payment as a share of the substance payment under each reward; the naming edits that leave the Rubric-RL reward unchanged; and the change in the redundancy criterion’s pass rate (points; all prompts with every variant, 293–300 per domain).

Explore similar work

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
Aug 12, 2026Minglai Yang, Xinyu Guo, Utkarsh Tyagi +6Task-Specific Rubrics Rubrics
Scale AI · University of Arizona · University of Texas at Dallas
Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
May 19, 2026Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5Rubric-Based Scoring Reward Functions
Scale AI · Persona
Reward Hacking in Rubric-Based Reinforcement Learning
May 12, 2026Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang +3Verifiable Rewards
Scale AI

Aug 12, 2026, cs.LGJ/K move · Enter open · S save

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

Minglai Yang, Xinyu Guo, Utkarsh Tyagi, Mian Zhang+5

Scale AI · University of Arizona · University of Texas at Dallas

Task-Specific Rubrics Rubrics Simple

Open paper AskPDF arXiv

Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer. The rubric, however, is a fixed proxy for quality, never a complete description of it, and a policy trained against it long enough will learn to exploit the difference. We measure this directly. Training Qwen3-8B with Group Relative Policy Optimization (GRPO) on medical and science rubrics and grading out-of-distribution (OOD) benchmarks with both the training judge and a stronger gold judge, we find that the two scores diverge during training. The training judge's score keeps climbing while the gold judge's score peaks and then falls, by 3 points on HealthBench-Hard and by 22 points on ResearchQA. A judge with a fixed bias would shift the gold curve by a constant, not send it down while the training score rises, so the divergence is reward hacking, not judge noise. We propose Rubric Dropout, a one-line fix borrowed from neuron dropout. At every step, we randomly drop a subset of the rubric's criteria before computing the reward, so the policy never optimizes the same rubric twice. The dropped subset is shared across each rollout group, so GRPO's group-relative advantages stay comparable, and evaluation always uses the full rubric. Comparing no dropout against dropout at 30% and 50% on both benchmark pairs, dropout raises the OOD gold score at every matched checkpoint (+1 to +2 points on HealthBench-Hard, +6 to +7 points on ResearchQA), lowers the two hacking measures we track, and costs nothing in domain. Sweeping the dropout fraction shows a broad 30-50% sweet spot, while the natural alternative, reweighting criteria by how useful they are to training, performs worse than no intervention at all in our setting.

Abstract

Explore similar work

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Reward Hacking in Rubric-Based Reinforcement Learning