Rubrics Beyond Evaluation: Exploiting Rubric Structure in Reinforcement Learning

Yuxun Huang; Yunyao Zhang; ZeLiang Chen; Yi-Ping Phoebe Chen; Junqing Yu; Zikai Song

cs.LGOct 19, 2026

Rubrics Beyond Evaluation: Exploiting Rubric Structure in Reinforcement Learning

Authors: Yuxun Huang, Yunyao Zhang, ZeLiang Chen, Yi-Ping Phoebe Chen, Junqing Yu, Zikai Song

Organizations: Huazhong University of Science and Technology, Wuhan, China · La Trobe University, Melbourne, Australia

Abstract

Rubrics are widely used to evaluate open-ended question answering by breaking overall quality into explicit criteria. These explicit criteria make rubrics a rich source of learning guidance, yet their potential for guiding the training process has been largely overlooked. Existing methods use rubrics mainly as direct feedback signals, without explicitly modeling the capabilities behind their criteria. This leaves their capability structure largely unused in guiding the training process. We introduce Rubric-Structured Reinforcement Learning (RSRL), an RL framework that unlocks the full potential of rubrics by exploiting their structure throughout the training process, rather than treating them simply as direct reward signals. At its core, RSRL builds on three key designs: (1) Rubric-to-Capability Structuring first recovers the capability structure hidden in rubric criteria, (2) Self-Sufficient Capability Learning then uses this structure to bootstrap underdeveloped capabilities into effective learning, (3) Capability-Residual Credit Routing finally preserves capability-specific gains during optimization. Experiments show that RSRL consistently improves both overall performance and underdeveloped capabilities, boosting downstream scientific question-answering performance by up to 7.43 over the initial policy and 7.60 over non-rubric structured RL baselines, showing that reinforcement learning can substantially benefit from the capability structure encoded in rubrics.

Figures & tables

Figure 1. Overview of RSRL. Rubric-to-Capability Structuring identifies targets from local and cross-question evidence. Self-Sufficient Capability Learning gates guided exploration. Capability-Residual Credit Routing adjusts response credit, with a guard selecting the routed candidate or baseline. Sampling weights account for the proposal shift; updates use the original question. RSRL links rubric-to-capability structuring, self-sufficient capability learning, and capability-residual credit routing.

Method	ResearchQA	SQAv2	GPQA-Diamond	RubricHub
Closed-source Frontier
GPT-5	77.60	—	85.70	—
Claude Opus 4.1	74.30	86.10	77.30	—
GPT-4.1	70.80	—	50.50	—
DeepSeek V3.1	75.90	—	68.30	—
Open-weight

Table 1. Scientific question-answering scores. Upper blocks provide external references; Qwen3-4B methods use our evaluation. Higher is better. Initial is the starting checkpoint. Parentheses are differences from Initial: score points, or percentage points for GPQA-Diamond. Within Qwen3-4B, bold/underlined scores mark best/second-best, including ties. Green/red parentheses mark gains/drops. Darker purple behind Ours indicates larger gains from Initial.

Figure 2. Effects of component removal. Points show the four performance metrics in Table 2 . Within each metric, methods are ordered from highest to lowest score. Colors and symbols identify methods; horizontal axes use metric-specific ranges. Four dot plots compare full RSRL with guided-exploration and residual-routing ablations. Full RSRL has the highest values in all four metrics.

Method	W.C.	F.W.	O.W.	W.S.	Overall
RSRL	45.34	49.43	55.47	$\mathbf{+1.38}$	55.40
w/o guided exploration	42.86 (-2.48)	43.60 (-5.83)	52.40 (-3.07)	$-0.32$	48.37 (-7.03)
w/o residual routing	42.37 (-2.97)	42.21 (-7.22)	53.61 (-1.86)	$-0.64$	48.48 (-6.92)

Table 2. Component ablations on the 128-question development probe. Metrics follow Appendix D.1 ; scores/AULCs use 0–100 and changes use score points. Bold marks column maxima. Parentheses show differences from full RSRL; red indicates decreases.

Method	W.C.	F.W.	O.W.	W.S.	Overall
Rubric-RL	44.23	47.06	55.02	$-0.45$	54.31
RuscaRL	44.99	45.20	56.08	$-2.31$	54.00
RSRL	45.34	49.43	55.47	$\mathbf{+1.38}$	55.40

Table 3. Capability tracking on 128 development questions. W.C./O.W.: current-worst/initially weak channel AULC; F.W.: final worst score; W.S.: minimum strong-channel change; Overall: mean question score. Scores/AULCs use 0–100; changes use points. Bold marks maxima.

Figure 3. Worst-channel scores over 100 updates: 27 checkpoints on 128 development questions. Worst-channel development scores for RSRL, Rubric-RL, and RuscaRL over 100 updates.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Starting policy / training questions	Qwen3-4B / 2,460 science questions
Optimizer updates / questions per update	100 / 32
Diagnostic / optimization responses	4 / 8
Active guided-sampling allocation	4 unguided + 4 guided
Response cap / generation mode	3,072 tokens / non-thinking
LoRA rank / alpha / dropout	32 / 32 / 0

Appendix

Table 4. Training configuration. Response budgets are per question. The lower block records additional RuscaRL settings.

Parameter	Value
Capability threshold $\tau$ / pooling $\kappa$	0.75 / 8
Domain retention $\beta_{D}$ / mapping acceptance	0.9 / 0.8
Guidance activation / saturation deficit	0.05 / 0.30
Guidance $\eta_{\max}$ / contrast clip $b$	1 / 4
Positive-score threshold / Low fraction	0.5 / 0.25
Partial projection $\xi$	0.5

Appendix

Table 5. Fixed diagnosis, exploration, and routing parameters.

Statistic	$1\times$	$3\times$	$4\times$
Mean target gain	0.001747	$-0.004924$	0.012816
Mean non-target gain	$-0.011115$	$-0.006001$	0.002867
Median ESS / 8	0.990510	0.920261	0.865988
Mean guided weight share	0.489837	0.427703	0.381443
Mean effective-signal proxy	0.044115	0.039832	0.038713

Appendix

Table 6. Offline guidance diagnostics on 900 common-valid questions. Gains use the 0–1 score scale; ESS is divided by the eight-response group size.

Method	AULC	worst	AULC	$\Delta_{\min}$	macro
	Worst	Final	Initial weak	Strong	Capability
Full $1\times$	42.6268	42.2695	52.6774	$+0.0620$	50.2890
Full $4\times$	42.3726	40.9658	52.9860	$-1.2417$	50.8242
RuscaRL	43.2669	44.9219	53.1383	$-0.8908$	51.1511

Appendix

Table 7. Earlier 50-update training comparison. AULCs are normalized by 50. Capability macro averages the six channel scores.

Rubric-Based RL

Rubric-based reinforcement learning uses explicit, often task-specific criteria to reward model behavior beyond simple outcome checks.

See all 104 papers

QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards
Jun 2, 2026Rongzhi Zhang, Rui Feng, Zhihan Zhang +8Reinforcement Learning RL for Language Model Reasoning
Amazon · Georgia Institute of Technology
Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
May 8, 2026Manish Bhattarai, Ismael Boureima, Nishath Rajiv Ranasinghe +2Reinforcement Learning RL for Language Model Reasoning
Los Alamos National Laboratory, Los Alamos, NM, USA.
Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
May 19, 2026Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5Reward Modeling Rubric-Based RL
Scale AI · Persona

Jun 2, 2026J/K move · Enter open · S save

QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

Rongzhi Zhang, Rui Feng, Zhihan Zhang, Jingfeng Yang+7

Amazon · Georgia Institute of Technology

Reinforcement Learning RL for Language Model Reasoning Rubric-Based RL Rubric-Based Evaluation Reward Design for RL

Open paper AskPDF arXiv

Rubric-based RL is a promising route for extending reinforcement learning beyond verifiable rewards, yet existing methods optimize rubrics while treating the query distribution as fixed. We identify a structural bottleneck: rubric quality is constrained by query structure. Open-ended queries yield vague rubrics; naively narrowing them introduces fabricated references that no model can verify, so all responses fail and training receives no reward signal. We present QUBRIC, a framework that co-designs queries and rubrics. Teacher-derived key points ground the rewriting of open-ended queries into scenario-based, evaluable questions. Contrastive rubric generation then turns teacher-policy gaps into query-level criteria, and learnability filtering retains only informative query-rubric pairs for GRPO training. QUBRIC achieves a +5.5 point gain on ArenaHard over the SFT baseline. Trained only on instruction-following data, it further transfers to three held-out benchmarks spanning legal, moral, and narrative reasoning (+6.3 points on average), with improvements concentrated in reasoning-related dimensions. These results provide evidence that co-designing queries and rubrics can make rubric-based RL a practical complement to RLVR beyond strictly verifiable tasks.

Abstract

Rubric-Based RL

Explore similar work

QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR