Judging the Judges: Confidence and Disagreement in Rubrics as Rewards

Michael Tian-Yue Liu; Anne Harrington; Trevor Darrell; Yutong Bai

cs.LGOct 15, 2026

Judging the Judges: Confidence and Disagreement in Rubrics as Rewards

Authors: Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell, Yutong Bai

Organizations: UC Berkeley

Abstract

Rubrics as Rewards (RaR) extends reinforcement learning to non-verifiable domains by using an LLM judge to score responses against a structured rubric. Prior work has focused on improving the rubric or strengthening the single judge, with comparatively little attention to how different single or multiple judges might be used during RaR training. We first show that judge identity alone produces a substantial spread in scores when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges. Building on this, we propose RaRE (Rubrics as Rewards with Ensembles), a novel judge ensemble strategy for RaR with two mechanisms: a soft-Likert reward that uses judge logprobs to weight more confident judges more heavily, and a drop-on-disagreement mask that excludes high-disagreement rollouts from the GRPO update. We find that this design is critical: while a naive average of judge scores fails to outperform the best single judge, RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA Diamond, with per-theme analysis showing that it inherits the complementary strengths of its constituent judges. Our findings establish judge ensembling as a compelling axis for improving RaR.

Figures & tables

Figure 1: Overview of RaRE (Rubrics as Rewards with Ensembles), instantiated with two judges. (1) Inputs: the user prompt, rubric, and a model rollout sampled from the policy (Qwen3-4B-Base) are passed to the judges. (2) Two-judge ensemble: we use an ensemble of two diverse LLM judges (selected from a pool of frontier and open-weight models). Each judge scores the rollout against the rubric on a 1–5 Likert scale, and we read its top-5 logprobs to recover an expected rating $E_{i}$ and a per-judge confidence $C_{i}$ . (3) Reward construction: our soft-Likert reward (A) combines the two expected ratings into a single scalar by weighting each judge by its inverse-variance confidence. Our drop-on-disagreement mask (B) excludes rollouts where the two judges’ expected ratings differ by more than $\tau\cdot(5-1)$ from the GRPO update.

Method	Rubric	Ensemble	Confid.	Disag.
RLVR	$\times$	$\times$	$\times$	$\times$
RaR	✓	$\times$	$\times$	$\times$
Naive ensemble	✓	✓	$\times$	$\times$
+ drop	✓	✓	$\times$	✓
+ soft-Likert	✓	✓	✓	$\times$
RaRE (ours)	✓	✓	✓	✓

Table 1: Comparison of methods. Rubric: reward comes from an LLM judge scoring a structured rubric. Ensemble: reward aggregates ensemble of

K{>}1

judges (§ 3.1.1 ). Confid.: soft-Likert inverse-variance confidence weighting (§ 3.1.2 ). Disag.: drop-on-disagreement mask (§ 3.1.3 ).

Figure 2: HealthBench scores under RaR training across baselines (Qwen3-4B-Base, Qwen3-4B), five single-judge variants, and our RaRE ensembles. Gem+DS pairs Gemini-2.5-Flash-lite with DeepSeek-V3.1; DS+Mini pairs DeepSeek-V3.1 with GPT-4o-mini. Ensemble configurations: mean averages judge ratings; soft applies our soft-Likert reward; drop applies our drop-on-disagreement mask (

\tau=0.30

). Best result in bold.

Figure 3: GPQA Diamond accuracy on a 50% test split under RaR training across baselines (Qwen3-4B-Base, Qwen3-4B), five single-judge variants, and our RaRE ensembles. Error bars show the mean over five training seeds with 95% confidence intervals. Gem+DS pairs Gemini-2.5-Flash-lite with DeepSeek-V3.1; DS+Mini pairs DeepSeek-V3.1 with GPT-4o-mini. Ensemble configurations: mean averages judge ratings; soft applies our soft-Likert reward; drop applies our drop-on-disagreement mask (

\tau=0.30

for Gem+DS ;

\tau=0.40

for DS+Mini , selected on a held-out validation set). Best RaR-trained result in bold.

No-RL baselines
Judge	Score	Chars	% $\geq$ 4k
Qwen3-4B-Base	0.266	1,610	4.2%
Qwen3-4B	0.308	1,340	0.5%
Single-judge RaR
gpt-oss-120b	0.208	4,742	97.0%
gpt-4o-mini	0.296	3,869	42.9%

Table 2: Reward vs. response-length statistics on HealthBench 1K subset by training judge. Chars is the mean response length in characters; %

\geq

4k is the fraction of responses that hit the 4,000-character truncation cap. Single-judge RaR uses full-parameter fine-tuning by default; the bottom block shows two interventions on the gpt-4o-mini judge that mitigate length hacking without sacrificing reward.

Panel of Figure 4 — Figure 4: Mean-reward delta on English-only HealthBench (n=3931) vs. Qwen3-4B-Base across themes (Comm. = communication; Emerg. = emergency referrals). Columns: single judges (DeepSeek-V3.1, Gemini-2.5-flash-lite, GPT-4o-mini) and ensembles (DeepSeek+Gemini, naive mean; DeepSeek+GPT-4o-mini, soft-Likert; both with drop). Bold: row winner.

$\tau$	drop %	Score	95% CI
none (mean only)	0%	$0.314$	$[0.295,\,0.335]$
$0.15$	14%	$0.251$	$[0.233,\,0.271]$
$0.25$	9%	$0.301$	$[0.281,\,0.323]$
$\mathbf{0.30}$	5%	$\mathbf{0.330}$	$[0.312,\,0.351]$
$0.35$	2%	$0.273$	$[0.254,\,0.294]$

Table 3: Drop-on-disagreement threshold sweep on the Gemini+DeepSeek pair (hard-Likert reward path) on the HealthBench 1K subset. drop % is the mean fraction of rollouts masked across training steps;

\tau{=}0.30

achieves the best score. 95% CIs computed via bootstrap.

$\tau$	drop %	Validation score
none (mean only)	0%	$0.349$
$0.30$	29%	$0.362$
$\mathbf{0.40}$	24%	$\mathbf{0.380}$
$0.45$	25%	$0.356$
$0.50$	12%	$0.374$

Table 4: Drop-on-disagreement threshold sweep on the DeepSeek+GPT-4o-mini pair (soft-Likert reward) on GPQA Diamond. Scores are 5-seed means on a 50% validation split; drop % is the mean fraction of rollouts masked across training steps.

\tau{=}0.40

is selected as the best validation threshold and is evaluated on the disjoint 50% test split in the main results.

Confidence + aggregation	Score	95% CI
inverse-var, weighted avg	$\mathbf{0.336}$	$[0.316,\,0.355]$
top-logit, weighted avg	$0.214$	$[0.195,\,0.233]$
top-logit, winner-takes-all	$0.198$	$[0.179,\,0.217]$

Table 5: Confidence-aggregation ablation on the DeepSeek+GPT-4o-mini pair with drop030, evaluated on the HealthBench 1K subset. Replacing inverse-variance (Eq. 6 ) with simpler schemes drops the score by 12–14 pp and below the untrained base (

0.266

). 95% CIs computed via bootstrap.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Per-category delta in English-only HealthBench mean reward vs. Qwen3-4B-Base. Single judges: ds = DeepSeek-V3.1, gem = Gemini-2.5-flash-lite, mini = GPT-4o-mini, qw = Qwen3-235B-Instruct, oss = GPT-OSS-120B. Ensembles: gemds = naive mean of Gemini+DeepSeek with drop030; dsmini = soft-Likert of DeepSeek+GPT-4o-mini with drop030. Diverging red–blue scale; bold marks the per-category winner. The dsmini ensemble wins 12 of 17 categories outright. emergent is the universal pain point: all seven RaR variants worsen relative to the no-RL base.

Panel of Figure 6 — Figure 6: Per-topic delta in GPQA Diamond accuracy vs. Qwen3-4B-Base, averaged over 5 seeds; gemds is a single seed. Single judges: ds = DeepSeek-V3.1, gem = Gemini-2.5-flash-lite, mini = GPT-4o-mini, qw = Qwen3-235B-Instruct, oss = GPT-OSS-120B. Ensembles: gemds = naive mean of Gemini+DeepSeek with drop030; dsmini = soft-Likert of DeepSeek+GPT-4o-mini with drop040 (the GPQA-tuned threshold). Diverging red–blue scale; bold marks the per-row winner. Note that Biology only has n=19 questions, so single-question swings move that row by $\sim$ 5pp, making comparisons relatively unreliable.

Panel of Figure 7 — Figure 7: Per-theme delta in English-only HealthBench reward vs. Qwen3-4B-Base, as a grouped bar chart. Same data as Figure 4 . Single judges: ds = DeepSeek-V3.1, gem = Gemini-2.5-flash-lite, mini = GPT-4o-mini. Ensembles: gemds = naive mean of Gemini+DeepSeek with drop030; dsmini = soft-Likert of DeepSeek+GPT-4o-mini with drop030.

Judge	Score	95% CI
— (Qwen3-4B-Base)	$0.257$	$[0.246,\,0.267]$
— (Qwen3-4B)	$0.308$	$[0.297,\,0.318]$
gemini-2.5-flash-lite	$0.296$	$[0.286,\,0.307]$
qwen3-235B-Instruct	$0.309$	$[0.299,\,0.320]$
gpt-oss-120b	$0.322$	$[0.312,\,0.332]$
deepseek-v3.1	$0.338$	$[0.328,\,0.349]$

Appendix

Table 6: Judge comparison for the RaR-trained model on HealthBench (English-only). Top block: baseline policies. Middle block: single-judge RaR. Bottom block: two-judge ensemble. HB 95% CIs computed via bootstrap. Judge pairs: gemds = Gemini-2.5-flash-lite + DeepSeek-V3.1; dsmini = DeepSeek-V3.1 + GPT-4o-mini. Methods: mean averages the two judges’ integer ratings (Sec. 3.1.1 ); soft-Likert reweights by inverse-variance confidence from logprobs (Sec. 3.1.2 ); Best per benchmark in bold.

Training judge(s)	GPT-4o-mini	GLM-5.3-Flash
RaRE ( dsmini , soft + drop)	$\mathbf{36.35}$	$\mathbf{35.45}$
DeepSeek-v3.1	$33.81$	$33.15$
GemDS mean + drop	$34.68$	$32.18$
GPT-4o-mini	$33.89$	$30.80$
Qwen3-235B-Instruct	$30.95$	$29.00$
GPT-4o	$33.54$	$28.47$

Appendix

Table 7: HealthBench (English-only) scores (in percent) for the same saved policy responses under GPT-4o-mini and GLM-5.3-Flash evaluation. Rows are single-judge RaR policies unless marked as an ensemble. GPT-4o is an additional single-judge RaR policy (Appendix F ).

Reward	HealthBench
GPT-4o-mini (single-judge RaR)	$33.9$
GPT-4o-mini + two-judge drop	$34.2$
RaRE (soft-Likert ensemble + drop)	$\mathbf{36.4}$

Appendix

Table 8: Single-judge reward with two-judge disagreement filtering on HealthBench (English-only, percent, GPT-4o-mini evaluator). The filtering judges are GPT-4o-mini and DeepSeek-V3.1.

Training judge(s)	API cost	HealthBench
RaRE	$4.46	$\mathbf{36.4}$
GPT-4o (single judge)	$18.42	$33.5$

Appendix

Table 9: Training-time judge API cost and HealthBench score (percent).

Policy	Mean (%)	95% interval
Qwen3-4B-Base	$37.0$	$[36.0,\,37.9]$
Qwen3-4B	$39.9$	$[38.5,\,41.2]$
RaRE	$39.3$	$[37.6,\,41.1]$
gpt-4o-mini	$38.4$	$[37.3,\,39.5]$
gpt-oss-120b	$38.4$	$[37.0,\,39.8]$
deepseek-v3.1	$38.0$	$[36.7,\,39.3]$

Appendix

Table 10: GPQA Diamond accuracy averaged over ten answer-order permutations (all 198 questions). Top block: baseline policies. Bottom block: RaRE and single-judge RaR, labeled by training judge. Intervals are the mean

\pm

2 standard errors over the ten permutations.

Figure 8: GPQA Diamond accuracy averaged over ten answer-order permutations. Error bars show the mean

\pm

2 standard errors over the ten permutations. Best RaR-trained result in bold.

Likert scale	Score
$L=5$	$0.263$
$L=7$	$0.254$

Appendix

Table 11: Soft-Likert scale ablation:

L=5

L=7

for the dsmini ensemble (DeepSeek-V3.1 + GPT-4o-mini, soft-Likert weighting, no disagreement masking) evaluated on the HealthBench 1K subset. Result: no improvement; the wider scale is slightly worse on the point estimate.

$\tau$	Validation score
none (mean only)	$0.331$
$0.25$	$0.235$
$\mathbf{0.30}$	$\mathbf{0.365}$
$0.35$	$0.337$

Appendix

Table 12: Drop-on-disagreement threshold sweep on the DeepSeek+GPT-4o-mini pair (soft-Likert reward) on HealthBench. Scores are means on a held-out 50% validation split.

\tau{=}0.30

is selected on validation.

Reward / training	full-FT	LoRA
mini (single judge)	$0.296$	$0.295$ $[0.275,\,0.314]$
ds (single judge)	$0.317$	$0.295$ $[0.276,\,0.316]$
gemds + drop030	$0.330$	$0.295$ $[0.276,\,0.314]$

Appendix

Table 13: Full-parameter vs. LoRA at fixed budget. Numbers are HB-1k mean reward with 95% bootstrap CIs (

n{=}1{,}000

, judge gpt-4o-mini). LoRA also produces

\sim

2–4

\times

shorter responses on average, useful when the policy is tending to length-hack.

Pairwise judge	HB-1k
mini	$0.312$
ds	$0.306$
gem	$0.172$
gem (no rubric)	$0.246$

Appendix

Table 14: Pairwise (non-rubric) LLM-as-a-judge reward, HB-1k. Pairwise mini/ds reach roughly the same level as their hard-Likert RaR counterparts; pairwise gem collapses, suggesting Gemini’s pairwise preference is poorly calibrated. The rubric absorbs most of the calibration burden in the RaR setting.

Policy / model
Base model	Qwen3-4B-Base
Parameters fine-tuned	all (full-FT)
	or LoRA $r{=}32$ , $\alpha{=}64$
Optimizer
Optimizer	AdamW-8bit (bnb)
Learning rate	$5{\times}10^{-6}$

Appendix

Table 15: Full GRPO hyperparameters. Settings that vary between conditions (LoRA on/off, reward type,

\tau

) are listed in the soft-Likert / drop sub-block; all other settings are shared across all other runs reported in the main paper.

Figure 9: Training reward over training steps for all single-judge and ensemble configurations (rolling mean, window=5). Soft-Likert configurations (dsmini soft-Likert and dsmini soft+drop030) reach higher training reward earlier than hard-Likert configurations, since the continuous reward signal allows finer-grained credit assignment. The mini+LoRA configuration trains more slowly, which is consistent with its lower final score but also its dampened length expansion (Figure 10 ).

Figure 10: Mean completion length (tokens) over training steps for all configurations (rolling mean, window=5). All configurations show a brief initial drop in length, followed by sustained expansion under single-judge RaR. Gemini (gem) is the notable exception: its completion length stays low throughout training, consistent with its constituent role in the shortest-output ensemble (gemds+drop030). mini+LoRA also keeps length bounded, supporting the LoRA-as-length-dampener finding in Table 2 .

Rubric-Based RL

Rubric-based reinforcement learning uses explicit, often task-specific criteria to reward model behavior beyond simple outcome checks.

See all 104 papers

Small Language Models as Judges for Rubric-Based Reinforcement Learning
Aug 30, 2026Fengyu Xie, Yilun Zhao, Bingsen Chen +2LLM-as-a-Judge Language Model Generation Evaluation
New York University · Yale University
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-Based RL
Aug 12, 2026Minglai Yang, Xinyu Guo, Zhengliang Shi +8Reward Hacking Rubric-Based RL
Scale AI · University of Arizona · Carnegie Mellon University +2
Rubric Rewards from Item Response Theory
Sep 28, 2026Milad Yazdani, Yaser Souri, Xiren Zhou +4Reward Modeling Rubric-Based RL
Department of Electrical and Computer Engineering, University of British Columbia · Microsoft · School of Biomedical Engineering, University of British Columbia

Aug 30, 2026J/K move · Enter open · S save

Small Language Models as Judges for Rubric-Based Reinforcement Learning

Fengyu Xie, Yilun Zhao, Bingsen Chen, Arman Cohan+1

New York University · Yale University

LLM-as-a-Judge Language Model Generation Evaluation Rubric-Based RL Rubric-Based Evaluation Language Model Probing

Open paper AskPDF arXiv

Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges. Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods, outperforming Generative and Logprob judges. Used as a GRPO reward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7

\times

more reward-judge time. Task and domain transfer experiments further suggest that Probe judges preserve criterion-level reward structure across settings.

Abstract

Rubric-Based RL

Explore similar work

Small Language Models as Judges for Rubric-Based Reinforcement Learning

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-Based RL

Rubric Rewards from Item Response Theory