Rubric Rewards from Item Response Theory

Milad Yazdani; Yaser Souri; Xiren Zhou; Pranit Chawla; Dena Shahriari; Subhojit Som; Xia Song

cs.CLSep 28, 2026

Rubric Rewards from Item Response Theory

Authors: Milad Yazdani, Yaser Souri, Xiren Zhou, Pranit Chawla, Dena Shahriari, Subhojit Som, Xia Song

Organizations: Department of Electrical and Computer Engineering, University of British Columbia · Microsoft · School of Biomedical Engineering, University of British Columbia

Abstract

Many language tasks have no single answer that can be checked automatically. Rubrics provide criteria for judging responses to these tasks. For reinforcement learning, the resulting verdicts must be combined into a scalar reward. A common approach sums the points assigned to satisfied criteria. Distinct verdict patterns can thus receive the same reward, and the fixed points encode how much each criterion should count, not how strongly its verdict distinguishes the current rollouts. Beyond this aggregation problem, judging the full rubric needs more judge requests as the criterion count grows. To address these limitations, Rubric Response Theory (RRT) measures quality and selects criteria when rubric criteria are monotone indicators of a shared target. Rather than adding assigned points, RRT uses a two parameter item response model that treats the verdict pattern as evidence about scalar quality specific to the rubric. Under this model, its likelihood score maximizes the local signal-to-noise ratio for quality. Its Response Parameter Network (RPN) reads the prompt and criterion text to predict criterion difficulty and discrimination. As the policy distribution changes during training, RRT uses online expectation maximization to update the RPN from current rollout verdicts. With Qwen3.5-4B as the policy, RRT's macro criterion score across Medical, Science, Rubrics as Rewards Science, and RubricBench is 1.7 points above that of group relative policy optimization (GRPO). On hard and very hard criteria in Medical and Science, RRT gains 2.8 to 5.6 points over GRPO. At half the criterion budget, adaptive Fisher selection with a frozen RPN keeps the macro criterion score across four datasets within 0.1 points of GRPO with full judging. These results show RRT can reduce judge requests while remaining competitive with GRPO.

Figures & tables

Figure 1: Criterion selection in RRT using Fisher information. (a) Criterion information for five criteria and four rollouts after judging two criteria. Gold diamonds show inferred rollout qualities. Dots show information from unjudged criteria at these qualities, dashed gray curves show judged criteria, and

c_{5}

is selected next. (b) Criterion scores after RRT with adaptive Fisher selection, normalized to full judging for each dataset. Shading shows the macro gap from full judging.

Figure 2: RRT reward inference for an illustrative rubric. (a) Verdict vectors for equally weighted criteria ordered by difficulty, with both rewards shown. (b) E-step for rollout

i=1

. The thick curve combines the verdict likelihoods with the Gaussian prior, and its mode is the RRT reward.

Prediction method	Science	RaR Science	Medical	RubricBench	Macro mean
Rollout evidence: ROC-AUC within each criterion
RPN without rollout evidence	50.0	50.0	50.0	50.0	50.0
Mean of other verdicts	69.6	67.2	63.8	62.5	65.8
RRT, $a_{j}=1$	69.7	67.0	63.9	62.8	65.8
RRT	70.0 (20.0)	67.4 (17.4)	64.0 (14.0)	61.8 (11.8)	65.8 (15.8)
Joint rollout and criterion ranking: pooled ROC-AUC

Table 1: Leave-one-criterion-out verdict prediction by ROC-AUC. Results within each criterion use the RPN without rollout evidence as the reference, and pooled results use the mean of the other verdicts. Blue marks RRT, and green parentheses give gains from the reference.

	Medical		Science		RaR Science		RubricBench	Macro mean
Condition	Criterion score	Normalized points score	Criterion score	Normalized points score	Criterion score	Normalized points score	Criterion score	Criterion score	Normalized points score
Base policy	64.1	63.3	76.5	70.9	68.7	71.0	74.7	71.0	70.0
Vanilla GRPO	67.7 (3.6)	66.8 (3.5)	78.3 (1.8)	77.8 (6.9)	70.6 (1.9)	73.7 (2.7)	77.0 (2.3)	73.4 (2.4)	73.8 (3.8)
POW3R	70.6 (6.5)	69.6 (6.3)	79.0 (2.5)	78.7 (7.8)	70.1 (1.4)	73.2 (2.2)	77.4 (2.7)	74.3 (3.3)	74.7 (4.7)
DIVA	64.9 (0.8)	64.4 (1.1)	78.8 (2.3)	78.3 (7.4)	70.6 (1.9)	73.8 (2.8)	75.8 (1.1)	72.5 (1.5)	73.1 (3.1)
Same RRT reward $R_{i}=\hat{z}_{i}$ and E-step across criterion parameter settings

Table 2: Criterion and normalized points scores. Bold marks the highest score in each column. RRT variants use parameters from empirical pass rates or the RPN, with the same reward and E-step. RubricBench uses one point per criterion, so both scores are equal.

Selection method	Medical	Science	RaR Science	RubricBench	Macro mean
Random	10.7%	15.2%	5.1%	8.9%	10.0%
Discrimination ( $a_{j}^{2}$ )	20.0% (9.3)	18.0% (2.7)	5.1% (0.0)	8.9% (0.0)	13.0% (3.0)
Static Fisher	20.0% (9.3)	20.5% (5.3)	18.3% (13.2)	18.6% (9.6)	19.3% (9.3)
Adaptive Fisher	20.8% (10.1)	22.9% (7.6)	18.3% (13.2)	21.9% (13.0)	21.0% (11.0)

Table 3: Unjudged criteria at the smallest budget reaching 95.0% correlation with GRPO advantages from the full rubric on rollouts from trained policies. Parentheses give gains over random selection.

	Medical		Science		RaR Science		RubricBench	Macro mean
Condition	Criterion score	Normalized points score	Criterion score	Normalized points score	Criterion score	Normalized points score	Criterion score	Criterion score	Normalized points score
Vanilla GRPO	67.7	66.8	78.3	77.8	70.6	73.7	77.0	73.4	73.8
RRT, full judging (1.00)	70.1 (2.4)	69.8 (3.0)	79.5 (1.2)	79.2 (1.4)	70.7 (0.1)	73.8 (0.1)	77.9 (0.9)	74.6 (1.2)	75.2 (1.4)
Random selection (0.50)
$\hookrightarrow$ Vanilla GRPO	65.9 (1.8)	64.0 (2.8)	74.7 (3.6)	74.8 (3.0)	67.5 (3.1)	70.3 (3.4)	76.3 (0.7)	71.1 (2.3)	71.4 (2.5)
$\hookrightarrow$ RRT	67.8 (0.1)	67.0 (0.2)	77.2 (1.1)	77.0 (0.8)	70.3 (0.3)	72.2 (1.5)	76.5 (0.5)	73.0 (0.5)	73.2 (0.7)

Table 4: Scores under full and partial judging. Budget is the fraction of criteria judged. Parentheses give differences from Vanilla GRPO with full judging. Only criterion score is shown for RubricBench.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Criterion information and likelihood scores under the Gaussian CDF

F=\Phi

as functions of pass probability

P_{ij}

. The left panel shows normalized information

I_{j}(z_{i})/a_{j}^{2}

. The right shows

a_{j}^{-1}\partial\ell_{ij}/\partial z_{i}

for a pass and failure. The dotted line marks

P_{ij}=1/2

, and shading marks verdicts with low probability.

	Verdict included in fit (ROC-AUC)
RPN configuration	Medical	Science	RaR Science	RubricBench	Macro mean
$F$
$\Phi$ (Gaussian CDF)	80.6	84.2	91.7	91.9	87.1
$\sigma$ (logistic CDF)	80.5	84.1	90.1	90.6	86.3
$\mathrm{cll}$ (complementary log-log)	80.5	84.3	92.2	91.8	87.2
Number of criterion parameters

Appendix

Table 5: Verdict ROC-AUC of RPN configurations across datasets when the evaluated verdict is included. Each block varies one component: the response function, number of criterion parameters, text conditioning, text embedder, or rollout policy. The remaining components use the standard configuration. Bold marks the highest macro mean in each block.

Figure 4: t-SNE projections of RPN representations. Rows show the four datasets. The first two columns color pairs by predicted criterion difficulty $b_{j}$ and empirical criterion difficulty $1-\bar{G}_{j}$ . The last two color sampled verdicts at jittered pair coordinates by the pass probability $\widehat{P}_{ij}^{\mathrm{MAP}}$ fitted using the RPN and inferred rollout quality, and by verdict $G_{ij}$ . Empirical criterion difficulty and fitted pass probability share a color scale.

Dataset	Difficulty $\rho_{\mathrm{S}}$	T@10	$b_{j}$	$1-\bar{G}_{j}$	$\widehat{P}_{ij}^{\mathrm{MAP}}$	$G_{ij}$
			Moran’s $I$
RubricBench	38.0	97.2	98.8	21.5	74.0	45.9
Medical	47.9	99.2	99.7	25.9	77.0	32.9
Science	47.2	99.2	99.3	28.2	85.8	38.2
RaR Science	38.5	99.1	99.1	19.4	74.6	44.1

Appendix

Table 6: Criterion difficulty prediction and local t-SNE geometry. Columns report difficulty Spearman correlation, trustworthiness at 10 neighbors, and Moran’s

I

for predicted criterion difficulty, empirical criterion difficulty, pass probability fitted using the RPN and inferred rollout quality, and verdict. All values are percentages.

Dataset	$F=\Phi$ separated	$F=\sigma$ separated	Count order reversals
Medical	83.3%	0.0%	0
Science	39.3%	0.0%	0

Appendix

Table 7: Separation of rollout pairs by the Gaussian and logistic CDFs on Medical and Science. Both CDFs use

a_{j}=1

and the same criterion difficulties. Columns report separation among pairs with equal pass counts and count order reversals among unequal pairs.

Figure 5: Reward signal on matched rollouts. Columns show three policies. Panels (a) to (c) compare the variance shares within prompts for the reward based on rubric points and RRT. Panels (d) to (f) compare tied pair shares. Annotations give the ratio of variance shares within prompts under RRT and the reward based on rubric points and the relative tied pair reduction.

		Group size $n$
Quantity	Reward	2	4	8	16	24	32
Medical
Tied pair share	Rubric points	5.6	5.9	5.8	5.9	5.9	5.9
	RRT	3.4	3.7	3.6	3.6	3.6	3.6
Variance share within prompts	Rubric points	9.2	13.2	15.7	16.6	16.9	17.3
	RRT	10.3	14.8	17.5	18.6	18.9	19.3

Appendix

Table 8: Reward signal across group sizes on Medical and Science. Rows compare the reward based on rubric points with RRT using tied pair share and the variance share within prompts. Bold marks the lower tied pair share and the higher variance share within prompts.

Dataset	Qwen3.5-4B	Llama-3.1-8B-Instruct	Qwen3.5-2B
Medical	60.5	60.4	53.4
Science	69.3	54.6	52.3
RaR Science	80.5	52.5	60.6
RubricBench	73.4	57.9	56.1

Appendix

Table 9: Unanimous criterion share by dataset and policy.

	Medical		Science		Macro mean
Target	Frozen	Online	Frozen	Online	Frozen	Online
Checkpoint aggregate	53.2	55.2 (2.0)	56.1	56.7 (0.6)	54.7	56.0 (1.3)
Next policy step	56.9	59.6 (2.7)	58.9	59.6 (0.7)	57.9	59.6 (1.7)

Appendix

Table 10: Pearson correlation between empirical criterion pass rate

\bar{G}_{j}

and pass probability at the prior mean of quality

P_{j}^{(0)}

computed from criterion parameters predicted by the frozen and online RPNs. Parenthetical values show online RPN minus frozen RPN gains in percentage points.

Figure 6: EM calibration trajectories for one post hoc selected criterion in Medical and one in Science. Blue shows predicted criterion difficulty

b_{j}

, and the dashed line marks its first logged value. Green shows rolling empirical criterion pass rate

\bar{G}_{j}

over five consecutive checkpoints. Pearson’s

r

measures correlation between predicted criterion difficulty and observed verdicts before averaging. Endpoint annotations describe the displayed series.

Method	$\lambda_{a}$	Criterion loss	ROC-AUC
Hard EM	0	0.369	91.0
Hard EM	0.05	0.364	91.9
Soft EM	0	0.380	90.4
Soft EM	0.05	0.381	90.6

Appendix

Table 11: RubricBench RPN fit under hard and soft E-steps at two discrimination regularizer weights

\lambda_{a}

. Columns report criterion loss and ROC-AUC. Only ROC-AUC is a percentage.

Dataset	Pairwise agreement	Unanimous triples	$\kappa$	AC1	Uncertain agreement
Medical	94.6	91.9	88.3	90.0	90.5
Science	95.0	92.4	86.2	92.1	90.3
RaR Science	95.2	92.9	86.9	92.5	86.0
Macro mean	94.9	92.4	87.1	91.6	88.9

Appendix

Table 12: Agreement across three independent criterion verdicts by dataset. Columns report pairwise verdict agreement, unanimous triple share, Cohen’s

\kappa

, Gwet’s AC1, and pairwise verdict agreement on uncertain criteria. All values are percentages.

Dataset	Policy	Disagreement	Unanimous triples	PRESENT	NOT_PRESENT
Medical	Qwen3.5-4B	1.58	95.27	1.52	1.63
	Qwen3.5-2B	1.33	96.01	0.94	2.15
	Llama-3.1-8B-Instruct	1.16	96.53	0.76	2.35
Science	Qwen3.5-4B	1.39	95.83	2.57	0.95
	Qwen3.5-2B	1.51	95.46	1.69	1.37
	Llama-3.1-8B-Instruct	1.21	96.36	0.90	1.89

Appendix

Table 13: Disagreement among three repeated judge verdicts by dataset and policy. Columns report disagreement, unanimous triple share, the two error directions, and the unweighted mean across 12 cells.

	Tied pair share		Order preservation rate		Stable nonzero ordering share
Dataset	RRT + marginal calibration	Rubric points	RRT + marginal calibration	Rubric points	RRT + marginal calibration	Rubric points
Medical	1.7	2.5	82.2	77.5	80.8	75.6
Science	12.5	13.3	71.8	70.6	62.9	61.2
RaR Science	46.8	60.2	65.2	63.8	34.7	25.4
Macro mean	20.3	25.3	73.1	70.6	59.5	54.1

Appendix

Table 14: Stability of RRT with marginal calibration and the reward based on rubric points across replicate criterion verdicts. Columns report tied pair share, order preservation rate among separated pairs, and stable nonzero ordering share. Bold marks the lower tied pair share and higher ordering rates.

Corruption channel	Level	Mean $\Delta$	Ahead	Behind	Mean $\Delta$	Lower
		Stable nonzero ordering share			Order flip rate
RRT + marginal calibration, $a_{j}=1$
Triples with split verdicts	0.025	$+0.82$	2	2	$+0.85$	3
	0.050	$+1.09$	2	1	$+0.14$	3
	0.100	$+1.56$	3	0	$-0.73$	7
All triples	0.025	$+0.61$	2	3	$+1.18$	2

Appendix

Table 15: Reward stability under corruption of the consensus of three verdicts across 12 dataset and policy cells. Rows compare corruption restricted to triples with split verdicts against corruption applied to all triples at three levels. Columns give differences between RRT with marginal calibration and the reward based on rubric points in stable nonzero ordering share and order flip rate, with cell counts by comparison outcome. Bold marks the highest level in the block with estimated

a_{j}

Corruption channel	Structure	Mean $\Delta$ (points)	Ahead	Behind
			Cells
Symmetric, all criteria	None	$-0.8$	0	17
Lenient, all criteria	None	$+0.7$	15	0
Lenient, longer responses	Across rollouts	$-1.2$	0	19
Symmetric and lenient mixture	Both	$+0.1$	5	1
Symmetric, 30% of criteria	Across criteria	$+5.1$	19	0

Appendix

Table 16: Difference in advantage cosine similarity between RRT with marginal calibration and criterion score at corruption level 0.20. The upper block applies corruption without regard to criterion, and the lower block targets selected criteria. Mean differences are in percentage points. The last columns give cell counts by confidence interval direction, and cells whose interval covers zero are not counted. Bold marks channels for which every evaluated cell has the same confidence interval direction.

Dataset	Corruption	Criterion score	Rubric points	POW3R	DIVA	RRT + marginal calibration	$\Delta$
Medical	0.00	100.0	97.8	97.4	94.9	95.6	$-4.4$
	0.10	68.9	66.6	64.1	60.4	72.1	$+3.2$
	0.20	49.7	47.7	44.1	38.8	55.7	$+6.0$
	0.30	30.6	29.3	26.7	22.7	35.0	$+4.4$
Science	0.00	100.0	97.3	97.1	94.3	96.4	$-3.6$
	0.10	62.2	60.3	57.8	52.9	66.5	$+4.3$

Appendix

Table 17: Advantage cosine similarity with clean verdicts under corruption of a fixed 30% criterion subset. Rows vary corruption level by dataset. Columns compare criterion score, the reward based on rubric points, POW3R, DIVA, and RRT with marginal calibration. Similarities are percentages. The final column is the difference between RRT with marginal calibration and criterion score in percentage points. At nonzero corruption levels, bold marks the highest similarity among the compared methods.

Quantity	Range across datasets
Equal pass count in two blocks of eight	61.4% to 77.1%
Uncertain criteria with a majority disagreement	25.5% to 31.7%
Reliability at eight rollouts, all criteria	93.6% to 96.0%
Reliability at eight rollouts, uncertain criteria	70.9% to 71.9%
Difficulty correlation after attenuation adjustment	38.5% to 52.4%

Appendix

Table 18: Empirical criterion pass rate reliability from disjoint blocks of eight rollouts. Values are ranges across Medical, Science, RaR Science, and RubricBench.

Dataset	Rubric points	RRT	Difference
Medical	26.6	27.1	$+0.5\pm 0.3$
Science	21.5	25.1	$+3.6\pm 0.3$
RaR Science	12.4	27.5	$+15.1\pm 3.5$
RubricBench	25.1	27.5	$+2.4\pm 0.6$

Appendix

Table 19: Rank correlation across criterion splits for reward ordering within prompts on training rollout groups from RRT runs. Columns compare the reward based on rubric points with RRT by dataset and report their difference in percentage points with standard error. Bold marks the higher rank correlation in each dataset.

Rank correlation quintile	RRT gain
1, lowest rank correlation	$+14$
2	$+7$
3	$+3$
4	$+2$
5, highest rank correlation	$+1$

Appendix

Table 20: Rank correlation across criterion splits for RRT and the reward based on rubric points on training rollout groups across rank correlation quintiles. The column reports RRT gains in percentage points.

Dataset	$\bar{r}$	$\bar{r}_{\text{model}}$	Explained	Observed	Bootstrap	Redundant	Deficient	$s<0.05$	$s>0.30$
	Pairwise dependence			Residual $Q_{3}$		Residual share		Redundant share by $s$
Medical	4.3	6.3	72.8	1.7	1.8	6.0	6.0	3.9 (4.2)	23.5 (11.6)
Science	9.0	11.2	76.4	3.4	3.8	6.8	7.7	4.2 (3.6)	20.2 (12.2)
RaR Science	12.1	18.1	75.4	6.8	10.8	5.6	11.2	3.1 (2.7)	13.7 (10.3)
RubricBench	10.6	14.6	85.3	6.2	7.8	5.1	7.5	3.4 (4.1)	16.4 (10.7)
Macro mean	9.0	12.5	77.5	4.5	6.1	5.9	8.1	3.7 (3.7)	18.5 (11.2)

Appendix

Table 21: Local independence diagnostics for criterion verdict pairs by dataset. Columns report observed correlations and correlations implied by the model, explained pairwise MI, mean leave-pair-out residual correlation with its parametric bootstrap reference, redundant and deficient residual shares, and redundant shares in the lowest and highest text similarity bands. Parentheses give bootstrap null rates.

Condition	Medical	Science	RaR Science	RubricBench	Macro mean
Qwen3.5-2B
Base policy	37.2	44.4	58.8	57.7	49.5
Vanilla GRPO	53.8 (16.7)	59.9 (15.4)	63.1 (4.3)	60.6 (2.9)	59.4 (9.8)
RRT	53.3 (16.1)	59.1 (14.7)	63.7 (4.9)	60.8 (3.1)	59.2 (9.7)
Llama-3.1-8B-Instruct
Base policy	31.1	29.8	46.7	61.9	42.4

Appendix

Table 22: Criterion scores for Qwen3.5-2B and Llama-3.1-8B-Instruct by dataset and macro mean. Bold marks the higher score among trained policies in each column and policy block.

Figure 7: Medical and Science criterion scores by empirical criterion difficulty under the base policy. Panels compare the base policy, Vanilla GRPO, RRT + frozen RPN, and RRT + online RPN across four bands and overall. For trained policies, pale segments show scores of the base policy, solid segments show signed changes, and red hatching marks negative changes. The value

n

gives the criterion count per band.

Condition	HealthBench	ResearchQA	Macro mean
Qwen3.5-4B
Base policy	58.8	78.9	68.9
Vanilla GRPO	59.8 (1.0)	79.7 (0.8)	69.8 (0.9)
RRT	60.9 (2.1)	80.0 (1.1)	70.5 (1.6)
Qwen3.5-2B
Base policy	42.7	64.5	53.6

Appendix

Table 23: Criterion scores on HealthBench and ResearchQA by policy. Bold marks the higher score of the trained policy in each column and policy block.

Figure 8: Median generated tokens, in units of

10^{3}

tokens. Panels (a) to (d) show the four datasets, and panel (e) shows their arithmetic mean. Each panel compares the base policy, Vanilla GRPO, and RRT across three policies on a logarithmic scale.

Selection method	Medical	Science	RaR Science	RubricBench	Macro mean
Random	10.7%	11.5%	4.9%	8.8%	9.0%
Discrimination ( $a_{j}^{2}$ )	16.8% (6.2)	14.2% (2.7)	4.9% (0.0)	8.8% (0.0)	11.2% (2.2)
Static Fisher	19.3% (8.7)	20.7% (9.3)	18.2% (13.3)	19.9% (11.1)	19.6% (10.6)
Adaptive Fisher	19.3% (8.7)	21.7% (10.2)	18.2% (13.3)	14.7% (5.9)	18.5% (9.5)

Appendix

Table 24: Share of criteria left unjudged at the smallest criterion budget reaching 95.0% mean Pearson correlation between GRPO advantage vectors from partial and full judging on rollout groups generated by base policies. Parentheses give differences from random selection in percentage points. Blue shading marks adaptive Fisher selection.

Figure 9: Mean Pearson correlation between GRPO advantage vectors from partial judging

\mathbf{A}^{(m)}

and full judging

\mathbf{A}^{(K)}

as the criterion budget increases. Panels show four datasets on rollout groups from the base policy. Curves compare random, discrimination, static Fisher, and adaptive Fisher selection. The dashed line marks 95.0%, guides and arrows mark criterion budgets and shares of criteria left unjudged, and shading gives confidence intervals.

Dataset	Source pool	Training	Validation	Test	Split role
Medical	29,681	5,000	500	500	Validation selects checkpoints. Test gives final results.
Science	29,418	5,000	500	500	Validation selects checkpoints. Test gives final results.
RaR Science	22,917	5,000	500	500	Validation selects checkpoints. Test gives final results.
RubricBench	1,147	847	150	150	Validation selects checkpoints. Test gives final results.
HealthBench	5,000	0	0	5,000	Full set for Medical generalization.
ResearchQA	21,414	0	0	21,414	Full set for Science generalization.

Appendix

Table 25: Released source pools and prompt splits by dataset. Counts are numbers of dataset rows. Each training dataset has separate validation and test splits. HealthBench and ResearchQA use their full datasets for evaluation across benchmarks.

Experiment case	Training role	Evaluation role
RPN experiments	Training prompts fit the RPNs used in policy training.	Validation selects those checkpoints.
Policy experiments	Training prompts update the policy.	Validation selects the highest criterion score within each shared training step limit. Test gives final policy results.
Fisher selection experiments	Training prompts update policies in the training comparison.	Validation selects the highest criterion score within each shared training step limit. Test gives final policy results for the training comparison.
Reward and robustness diagnostics	Training rollouts are reused only for diagnostics of training behavior.	Other diagnostics use the test set. They do not select checkpoints or give final policy results.
Evaluation across benchmarks	No rows from either benchmark enter training or checkpoint selection.	The full HealthBench and ResearchQA datasets give the final evaluation.
Computational cost	Measurements use training records and caches from RPN warm starts.	Checkpoint evaluation timings enter where reported. Test responses are not used.

Appendix

Table 26: Data roles for the experiment groups. The training, validation, and test prompt counts are in Table 25 .

Setting	Value
Job layout	Eight nodes for each full batch, with one independent run per node. Each node has eight NVIDIA A100 80 GB GPUs.
Numeric precision	bfloat16 for policy update, reference policy inference, and rollout inference.
Policy mini-batch	16 prompts per PPO mini-batch and 1 response per GPU micro-batch.
Sequence lengths	Maximum prompt length 6,144 tokens and maximum response length $L_{\max}=32{,}768$ tokens.
Sampling	Temperature 1.0, top- $p$ 1.0, top- $k$ disabled, repetition penalty 1.1 for Qwen and 1.0 for Llama.
GRPO loss	Clipped surrogate with $\epsilon_{c}=0.2$ and mean token loss within each sequence averaged across sequences.

Appendix

Table 27: Policy training and evaluation configuration. Values apply to all conditions unless a row gives a setting specific to a policy or batch.

Primary comparison
Policy	Condition	Medical	Science	RaR Science	RubricBench
Qwen3.5-4B	Vanilla GRPO	120	130	105	50
	RRT + frozen RPN	145	150	120	45
	RRT + online RPN	115	110	110	50
Qwen3.5-2B	Vanilla GRPO	100	150	120	45
	RRT	125	150	110	45

Appendix

Table 28: Selected policy checkpoint steps. Each comparison uses a shared training step limit across its conditions.

	Medical		Science
Method	Judge requests	Input tokens	Judge requests	Input tokens
Vanilla GRPO	7,716	11.1	6,908	16.3
RRT, full judging (1.00)	7,685	11.1	6,895	16.5
RRT, adaptive Fisher
$\hookrightarrow$ 0.95	7,447	10.8	6,670	16.1
$\hookrightarrow$ 0.80	6,265	9.1	5,622	12.8

Appendix

Table 29: Mean Medical and Science judge requests and input tokens per policy step over the first 30 steps. Input tokens are in millions. Adaptive Fisher rows give the criterion budget in parentheses.

Input tokens per request
Quantity	Medical	Science
Completed judge requests	6,728	4,984
$\hookrightarrow$ mean	1,137	3,800
$\hookrightarrow$ median	984	1,216
$\hookrightarrow$ p10 to p90	496 to 1,664	640 to 2,846
$\hookrightarrow$ maximum	29,399	32,115

Appendix

Table 30: Judge API usage and request failures on Medical and Science for one policy step per dataset. Rows report completed judge request counts and token counts, input token distribution statistics, failed or additional attempts, retries, and ungraded criteria.

Method	Added seconds	Overhead of the policy step
Vanilla GRPO	0	0%
RRT +
$\hookrightarrow$ batch pass rate	0.10	0.004%
$\hookrightarrow$ cached pass rate	0.10	0.004%
$\hookrightarrow$ frozen RPN	0.17	0.007%
$\hookrightarrow$ online RPN	2.79	0.123%

Appendix

Table 31: Added RRT computation for one Medical policy step, in seconds and as a percentage of the time for the Vanilla GRPO policy step.

	Seconds per policy step, median with p10 to p90
Condition	Generation	Judging	Log probability	Policy update	Total policy step	p10 to p90 of policy step
Medical
Vanilla GRPO	460	1,451	36	107	2,274	1,136 to 3,603
RRT +
$\hookrightarrow$ frozen RPN	647	1,457	38	124	2,336	1,815 to 3,092
$\hookrightarrow$ online RPN	703	1,453	66	217	2,454	1,628 to 4,193

Appendix

Table 32: Median time per policy step and selected stage times. The final column gives the range from the 10th to 90th percentile for the total time per policy step. Blue shading marks RRT + online RPN.

Dataset	Optimizer steps	GPU hours, 0.6B embedder	GPU hours, 8B embedder
Medical	3,000	5.30	7.64
Science	1,254	2.48	4.71
RaR Science	1,254	1.12	2.33
RubricBench	1,067	0.89	1.57

Appendix

Table 33: Cost of an RPN warm start on one A100 80 GB GPU. Columns report optimizer steps and combined embedding and fitting GPU hours for 0.6B and 8B frozen text embedders.

Policy	Added deployment parameters	Added inference components
Base policy	0	None
Policy trained with RRT	0	None

Appendix

Table 34: Added deployment parameters and inference components for the base policy and a policy trained with RRT.

Explore similar work

May 19, 2026cs.AI

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins

24

30

base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in

2.5

4\times

fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.

Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5

Scale AI · Persona

Rubric-Based Scoring Reward Functions

Ask PDF

Aug 12, 2026cs.LG

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer. The rubric, however, is a fixed proxy for quality, never a complete description of it, and a policy trained against it long enough will learn to exploit the difference. We measure this directly. Training Qwen3-8B with Group Relative Policy Optimization (GRPO) on medical and science rubrics and grading out-of-distribution (OOD) benchmarks with both the training judge and a stronger gold judge, we find that the two scores diverge during training. The training judge's score keeps climbing while the gold judge's score peaks and then falls, by 3 points on HealthBench-Hard and by 22 points on ResearchQA. A judge with a fixed bias would shift the gold curve by a constant, not send it down while the training score rises, so the divergence is reward hacking, not judge noise. We propose Rubric Dropout, a one-line fix borrowed from neuron dropout. At every step, we randomly drop a subset of the rubric's criteria before computing the reward, so the policy never optimizes the same rubric twice. The dropped subset is shared across each rollout group, so GRPO's group-relative advantages stay comparable, and evaluation always uses the full rubric. Comparing no dropout against dropout at 30% and 50% on both benchmark pairs, dropout raises the OOD gold score at every matched checkpoint (+1 to +2 points on HealthBench-Hard, +6 to +7 points on ResearchQA), lowers the two hacking measures we track, and costs nothing in domain. Sweeping the dropout fraction shows a broad 30-50% sweet spot, while the natural alternative, reweighting criteria by how useful they are to training, performs worse than no intervention at all in our setting.

Minglai Yang, Xinyu Guo, Utkarsh Tyagi +6

Scale AI · University of Arizona · University of Texas at Dallas

Task-Specific Rubrics Rubrics Simple

Ask PDF

Aug 30, 2026cs.CL

Small Language Models as Judges for Rubric-Based Reinforcement Learning

Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges. Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods, outperforming Generative and Logprob judges. Used as a GRPO reward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7

\times

more reward-judge time. Task and domain transfer experiments further suggest that Probe judges preserve criterion-level reward structure across settings.

Fengyu Xie, Yilun Zhao, Bingsen Chen +2

New York University · Yale University

Judges Reward Functions

Ask PDF