MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning

Yuxuan Fan; Jaehong Yoon

cs.AIOct 2, 2026

MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning

Authors: Yuxuan Fan, Jaehong Yoon

Organizations: NTU Singapore

Abstract

Rubric-based reinforcement learning extends reward-driven optimization to open-ended tasks by assigning partial credit to individual response requirements. However, rubric judges can assign a high criterion score even when the information or action it requires is absent from the response, a failure mode we term Vacuous Credit. Such awards persist after the required information is removed and can reverse the sign of a response's GRPO advantage. To address this problem, we introduce MetaRubric, which alternates evidence-aware policy optimization with response-guided rubric adaptation. We construct counterfactual counterparts by changing one task-relevant fact in each prompt. During policy optimization, credit is assigned only when the response contains sufficient evidence to satisfy the required rubric criterion. After each policy-optimization stage, current policy responses guide revisions to original and counterfactual criteria while preserving the meaning of the original prompt's initial rubric as interpreted under each prompt's facts. We also adapt criterion weights at stage boundaries to better address observed policy errors. Across multiple backbones, MetaRubric improves PubMedQA accuracy by 6.00--20.40 percentage points over static-judge GRPO, with further gains on HealthBench-Hard and two multimodal medical benchmarks.

Figures & tables

Figure 1: Motivation and overview of MetaRubric. (a) Vacuous Credit in rubric-based RL. Under an incomplete, underspecified criterion, the judge awards full credit to a safe, conservative response that leaves the query unanswered. (b) MetaRubric addresses this by jointly improving the policy and rubric: the inner loop optimizes evidence-aware responses, while the outer loop reweights criterion groups and revises criterion descriptions.This produces clearer criteria and more concrete answers.

Figure 2: Reward growth and rubric satisfaction. (a,b) Training reward and rubric scores during training; evaluation uses a panel of three stronger judges on a separate test set. Panels (a,c) use GPT-4o-mini as the training judge; panels (b,d) use GPT-5.4-mini. (c,d) Changes from step 0 in fully satisfied requirement weight and Vacuous Credit identified using a stronger model to audit response evidence. Both are normalized by total available rubric credit

P_{x}

and expressed in percentage points.

	Deleted content
Reward judge	Required	Generic
	content	statements
GPT-4o-mini	$82.0\%$	$93.1\%$
GPT-5.4-mini	$69.8\%$	$95.0\%$

Table 1: Retention rate of target-criterion awards in the deletion test.

Figure 3: Overview of MetaRubric. Each stage alternates $\mathcal{N}$ evidence-aware GRPO updates (Inner) with rubric adaptation (Outer). Accepted rubric updates carry into the next stage.

Reference Models
Model / training method	PubMedQA	HealthBench-Hard		MMOral-X		MMOral-OPG
	Acc.	Acc.	Aux. Acc.	Avg.	Aux. Acc.	Avg.	Aux. Acc.
LLaVA-OneVision	70.40	0.00	6.92	6.31	32.75	24.86	36.49
InternVL3.5-8B-Instruct	69.80	1.85	10.20	3.13	24.93	31.49	48.32
MiniCPM-V 4.5	54.20	1.59	10.30	3.12	20.19	26.52	35.17
MedGemma 1.5-4B	55.20	4.88	18.24	5.77	25.82	24.46	32.21

Table 2: Performance on four medical benchmarks (%;

\uparrow

). Metrics are PubMedQA accuracy, HealthBench-Hard accuracy-axis score, MMOral-X mean score, and MMOral-OPG overall score. Aux. Acc. is auxiliary QA accuracy measured with the Qwen3-1.7B reader. Qwen3 families use Qwen3 for text and Qwen3-VL of the same size for multimodal tasks.

Figure 4: Component ablations. HealthBench-Hard accuracy-axis scores with Qwen3-4B and MMOral-OPG overall scores with Qwen3-VL-4B (0–100;

\uparrow

Table 7

Figure 5: Illustrative rubric and weight adaptation for a PubMedQA prostate-bed motion question. Each row aligns an abstract finding, a response, and the criterion before and after revision. The example group-weight update places greater relative emphasis on overstatement errors.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Source entries	Covered entries	Questions	Questions/entry
HealthBench-Hard	1,000	965	4,117	4.12
MMOral-X	300	300	828	2.76
MMOral-OPG	578	578	981	1.70

Appendix

Table 5: Auxiliary-question construction statistics for evaluation. Covered entries have at least one question; questions per entry is averaged over all source entries.

Parameter	Value
Policy	Qwen3-4B-Instruct-2507; thinking disabled
Optimizer	AdamW, $\beta_{1}=0.9$ , $\beta_{2}=0.999$
Learning rate / weight decay	$10^{-6}$ / $0.01$
Learning-rate schedule	4 warmup updates, then constant
Gradient norm clipping	$1.0$
Prompt batch / PPO mini-batch	18 / 18

Appendix

Table 6: HealthBench training hyperparameters in the supplementary configuration. Batch sizes count prompts before rollout expansion. Token budgets are per GPU.

Rubric-Based RL

Rubric-based reinforcement learning uses explicit, often task-specific criteria to reward model behavior beyond simple outcome checks.

See all 104 papers

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
May 19, 2026Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5Reward Modeling Rubric-Based RL
Scale AI · Persona
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-Based RL
Aug 12, 2026Minglai Yang, Xinyu Guo, Zhengliang Shi +8Reward Hacking Rubric-Based RL
Scale AI · University of Arizona · Carnegie Mellon University +2
Rubric Rewards from Item Response Theory
Sep 28, 2026Milad Yazdani, Yaser Souri, Xiren Zhou +4Reward Modeling Rubric-Based RL
Department of Electrical and Computer Engineering, University of British Columbia · Microsoft · School of Biomedical Engineering, University of British Columbia

May 19, 2026J/K move · Enter open · S save

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei, Daniel George+4

Scale AI · Persona

Reward Modeling Rubric-Based RL Reinforcement Learning with Verifiable Rewards Rubric-Based Evaluation Rubric-Guided Generation Reward Design for RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins

24

30

base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in

2.5

4\times

fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.

Abstract

Rubric-Based RL

Explore similar work

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-Based RL

Rubric Rewards from Item Response Theory