cs.LGNov 4, 2026

Rubric Curriculum RL: Exploiting the Generation-Verification Gap in Non-Verifiable Domains

Authors: Sumeet Ramesh Motwani, Tejas Krishnan, Alistair Letcher, Charles London, Suhaas Bhat, Huitian Jiao, Philip Torr, Riashat Islam, 

Organizations: University of Oxford · Microsoft Research

Abstract

Reinforcement learning with verifiable rewards (RLVR) has led to significant gains in math and code generation. Extending these gains to open-ended domains remains challenging: ground-truth verification is unavailable, human annotation is expensive, and learned reward models are prone to reward hacking. The generation-verification gap, where judging good outputs is easier than producing them, offers a potential path to improvement. However, rubric-based judges are often imprecise and stationary, so policies can overfit or reward hack them. We introduce Rubric Curriculum RL (RcRL), which exploits this gap without new data, human annotations, or stronger teacher models. We combine pairwise preferences over rollouts with a curriculum over rubric criteria, yielding a more discriminative signal than absolute scoring and a non-stationary objective. Whereas baselines plateau within a few hundred steps, RcRL preserves output entropy and keeps improving past 1000 steps. On creative writing, RcRL achieves a 70.5% human win rate against the base model, with gains across benchmarks and judges. On HealthBench, RcRL improves over the base model by 14.6% (full set) and 25.4% (consensus subset), outperforming all training baselines, including instance-specific rubrics.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. RARE: Reinforcement Learning with Adaptive Rubric Evolution for Open-Ended Generation

    Oct 18, 2026Xueyun Tian, Bingbing Xu, Chuyue Li +9Rubric-Based RLReward Design for RL

  2. QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards

    Jun 2, 2026Rongzhi Zhang, Rui Feng, Zhihan Zhang +8Reinforcement LearningRL for Language Model Reasoning

  3. Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling

    Jul 2, 2026Dazhi Fu, Jiuding Yang, Yiwen Guo +1Reward ModelingLLM-as-a-Judge