Rubric Curriculum RL: Exploiting the Generation-Verification Gap in Non-Verifiable Domains
Authors: Sumeet Ramesh Motwani, Tejas Krishnan, Alistair Letcher, Charles London, Suhaas Bhat, Huitian Jiao, Philip Torr, Riashat Islam, Christopher Summerfield, Jakob Foerster, Christian Schroeder de Witt, Qilong Gu, Shital Shah
Organizations: University of Oxford · Microsoft Research
Reinforcement learning with verifiable rewards (RLVR) has led to significant gains in math and code generation. Extending these gains to open-ended domains remains challenging: ground-truth verification is unavailable, human annotation is expensive, and learned reward models are prone to reward hacking. The generation-verification gap, where judging good outputs is easier than producing them, offers a potential path to improvement. However, rubric-based judges are often imprecise and stationary, so policies can overfit or reward hack them. We introduce Rubric Curriculum RL (RcRL), which exploits this gap without new data, human annotations, or stronger teacher models. We combine pairwise preferences over rollouts with a curriculum over rubric criteria, yielding a more discriminative signal than absolute scoring and a non-stationary objective. Whereas baselines plateau within a few hundred steps, RcRL preserves output entropy and keeps improving past 1000 steps. On creative writing, RcRL achieves a 70.5% human win rate against the base model, with gains across benchmarks and judges. On HealthBench, RcRL improves over the base model by 14.6% (full set) and 25.4% (consensus subset), outperforming all training baselines, including instance-specific rubrics.
Figures & tables
Figure 1: RcRL vs. RaR on FictionBench. Win rates against the base Qwen3-4B model over training, averaged across 5 seeds; shading indicates ±1 SEM. RcRL (red) improves over time, reaching ∼ 75% by step 1300, while RaR (blue) plateaus early around 55–65%. This demonstrates that curriculum learning over evaluation criteria enables sustained improvements by allowing additional training compute to be spent productively.
Figure 2: Overview of Rubric Curriculum RL (RcRL). (Left) The challenge: training with a static rubric leads to reward hacking, where the model optimizes for the reward signal without improving true quality. (Right) Our solution: RcRL progressively introduces evaluation criteria, using pairwise comparisons to allocate rewards. The curriculum advances when current criteria have improved (Section 5.2 ).
Figure 3: Best-of- N inference scaling on FictionBench. Win rate of self-selected Best-of- N against the N=1 baseline, judged by GPT-5.2 as ground truth. Win rates exceed 50% from N=2 onwards (Claim 1), grow monotonically with N , and are uniformly larger for the 8B model than the 4B model (Claim 2).
Figure 4: Reward hacking with static rubrics. (Left) Absolute train reward increases steadily over the course of training, reaching near-saturation. (Right) Evaluation win rate against the base model on FictionBench shows no consistent improvement as training progresses, indicating a performance mismatch.
FictionBench
LechMazur
Method
ELO
Win%
Score
Hratio
Qwen 3 4B (Baseline)
1000
–
77.0
–
Qwen 3 4B RaR
1052
57.5
80.3
0.73
Ours (RcRL)
1181
73.9
80.9
0.93
Table 1: Additional benchmark results. FictionBench and the LechMazur Writing benchmark. Best results per model in bold .
GPT-5.2
Claude 4.5 Opus
Deepseek V3.2
Method
ELO
Rubric
Win%
ELO
Rubric
Win%
ELO
Rubric
Win%
Qwen 3 4B (Baseline)
1000
45.4
–
1000
40.8
–
1000
51.2
–
Qwen 3 4B RaR
1210
48.5
61.1
1051
41.1
51.7
1326
55.0
53.9
Ours (RcRL)
1577
50.1
69.3
1105
40.0
59.2
1524
58.1
65.4
Table 2: Creative Writing Bench v3 evaluation results. Three frontier models (GPT-5.2, Claude 4.5 Opus, Deepseek V3.2) serve as independent judges, evaluating outputs from each method via pairwise comparisons. We report ELO ratings (baseline anchored at 1000), rubric scores (0–100 scale), and win rates against the Qwen 3 4B baseline (%). ELO ratings reflect relative rankings within each judge and are not comparable across judges. Best results per metric and judge in bold .
Figure 5: Head-to-head win rates on FictionBench. RcRL achieves substantial gains over the Qwen3-4B baseline against all opponents: +28% against RaR, +28% against Qwen3-8B, and +26% against Qwen3-14B.
Figure 6: Human evaluation results. Win rates of RcRL-trained Qwen3-4B against the base model across seven criteria, averaged over three annotators who conducted 500 comparisons each. RcRL achieves a 70.5% win rate on Best Story Overall , with consistent improvements across all criteria.
Figure 7: Scaling behavior of RcRL. Win rates against base models across model sizes for Qwen3 (red) and Llama-3.2 (blue). Performance improves with scale, from 54.6% at 0.6B to 75.2% at 14B parameters. The Llama-3.2-3B result (76.4%) demonstrates that RcRL generalizes across model families.
CW Bench v3
FictionBench + LM
Method
ELO
Rubric
Win%
ELO
Win%
LM
No RL Training
Qwen 3 4B (Baseline)
1000
45.4
–
1000
–
77.0
No Curriculum
Absolute RaR
1210
48.5
61.1
1052
57.5
80.3
Dropout Absolute RaR
1166
49.8
55.4
1113
65.7
80.9
Table 3: Ablation study and baseline comparisons. We evaluate the contribution of each component of RcRL. “ + ” denotes adding a component to the base method, “ − ” denotes removing from the full method. All models trained from Qwen 3 4B. Win rates computed against the base model. Best results in bold .
Figure 8: Single-criterion training dynamics. Win rates averaged across five models, each trained on a single criterion (e.g., plot coherence, character development). Per-criterion performance (red) improves steadily, while overall story quality (blue) peaks around step 100 before declining.
Figure 9: Policy entropy during training. The static rubric baseline exhibits a steady decline in entropy, indicating mode collapse toward a narrow set of outputs. RcRL maintains consistently higher entropy throughout training.
Method
HealthBench Δ
Consensus Δ
Qwen 3 4B (Baseline)
–
–
Qwen 3 4B RaR
+8.26%
+7.53%
Instance-specific
+13.52%
+10.52%
Absolute RcRL
+13.40%
+23.87%
RcRL (Ours)
+14.60%
+25.38%
Table 4: HealthBench cross-domain results. Score improvement over Qwen 3 4B on full HealthBench and the consensus subset (criteria validated by multiple annotators). Best in bold .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
n -gram repetition score
Qwen3-4B
11.51
Qwen3-4B RaR
13.58
RcRL
7.52
Appendix
Table 5: N -gram repetition rates (lower is better).
Parameter
Value
GRPO Configuration
Learning rate
5×10−6
LR scheduler
Cosine
Warmup steps
50
Per-device batch size
2
Gradient accumulation steps
4
Appendix
Table 6: Training hyperparameters for RcRL. Configuration for Qwen3-4B experiments. We use LoRA for parameter-efficient fine-tuning.
Step
Criterion
Evaluation Focus
0
Prompt Alignment
Does the story fully address the requirements of the prompt? Are all key elements, details, constraints, or events implied or explicitly stated in the prompt incorporated into the story in a meaningful way?
100
Plot Coherence & Progression
Does the story present a clear and logically consistent plot? Do events follow naturally from one another with a sense of cause and effect? Are transitions between scenes smooth?
200
Character Depth & Motivation
Do the characters have believable goals, emotions, and internal conflicts? Are their actions motivated and consistent with their personalities?
300
Narrative Tension & Stakes
Does the story sustain a sense of tension, uncertainty, or conflict that keeps readers engaged? Are there meaningful stakes—emotional, physical, or moral—that make the outcome matter?
400
Show, Don’t Tell
Does the story rely on vivid imagery, sensory detail, and action to communicate emotion or meaning? Does it avoid excessive exposition and direct statements of feelings or morals?
500
Emotional Resonance
Does the story evoke a genuine emotional response? Are the emotions integrated naturally into the narrative rather than being forced or sentimental?
Appendix
Table 7: Creative writing rubric criteria in curriculum order. Each criterion is introduced at the indicated training step and remains active for all subsequent steps. The ordering progresses from foundational craft elements (prompt alignment, plot coherence) through character and tension, to stylistic refinement (voice, originality). This progression reflects a pedagogical approach: models first learn to write coherent, on-topic stories before optimizing for subtler qualities like thematic depth and distinctive voice.
Figure 10: Curriculum weighting evolution. Diagram of how the curriculum weighting evolves over the course of training. The newly introduced criterion always has a weight of 21 with the remaining weight distributed uniformly over the remaining criteria.
Step
Criterion
Evaluation Focus
0
Factual Correctness
Does the response contain factually correct medical information, accurately representing dosages, mechanisms, contraindications, and expected outcomes without contradicting clinical consensus?
50
No Harmful or Unsafe Advice
Does the response avoid recommendations that could directly harm the patient—such as dangerous interactions, contraindicated treatments, or advice that delays necessary emergency care?
100
No Fabrication
Does the response avoid inventing clinical details not provided by the user—such as lab values, diagnoses, or examination findings—rather than presenting assumptions as facts?
150
Instruction Following
Does the response directly address what the user asked, following any specified format or scope rather than going off-topic?
200
Completeness
Does the response cover the important aspects of the query—causes, symptoms, treatment options, risks, and next steps—rather than providing only a partial answer?
250
Specificity & Actionability
Does the response provide concrete, actionable information—specific tests, treatments, or warning signs—rather than vague advice like “consult a doctor”?
Appendix
Table 8: HealthBench rubric criteria in curriculum order. Each criterion is introduced at the indicated training step and remains active for all subsequent steps. Compared to creative writing, there is no ordering justification, with criteria synthesized from both the HealthBench instance-specific rubrics and rubric criteria from Yang et al. (2026) .
Figure 11: LechMazur per-criterion scores. Heatmap showing performance of Qwen3-4B (baseline), RaR, and RcRL across the 18 evaluation criteria. Q1–Q8 assess major story aspects; Q9A–Q9J assess element integration. RcRL shows particularly strong gains on element integration criteria.
Criterion
Fleiss’ κ
Interpretation
Best Story Overall
0.548
Moderate
Appendix
Table 9: Inter-annotator agreement. Fleiss’ Kappa score for overall preference. This score is consistent with prior work on creative writing evaluation ( Gómez-Rodríguez and Williams, 2023 ) .
Table 10: Results from human expert annotations. Preference bars (right) show win rates of RcRL over the baseline Qwen 3 4B model across criteria, based on 500 pairwise comparisons.
Verifiable rewards have advanced reinforcement learning in mathematics and code. However, many valuable open-ended tasks share three properties: no automatic verifier of overall quality, multiple requirements to satisfy jointly, and multiple valid responses to each query. Training therefore needs feedback on requirement satisfaction across different valid responses. Rubric-based reinforcement learning provides this feedback by aggregating scores from explicit criteria. As learning progresses unevenly, some criteria become fully satisfied and lose discrimination while others remain informative. Dynamic methods revise shared rubrics or select evolving criteria to restore discrimination, but may also change useful criteria. To address this, we propose Reinforcement Learning with Adaptive Rubric Evolution (RARE). The core idea is to use current policy feedback to guide both which criteria change and what their replacements assess. Sample-level updates replace criteria fully satisfied across current outputs with checks of remaining quality differences, retaining all other criteria and scores. Within this rule, we compare rewarding desirable qualities with penalizing observed flaws. Video script generation exemplifies these properties and supports content production. Its demands for coherent, executable narratives under constraints make it a challenging testbed. Targeting flaws performs best across three video types with a mean gain of 3.82 points over matched static rubrics. Model and human preferences support these gains. Improvements also extend to writing and general generation benchmarks.
Xueyun Tian, Bingbing Xu, Chuyue Li +9
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Qwen Business Unit of Alibaba +1
Rubric-based RL is a promising route for extending reinforcement learning beyond verifiable rewards, yet existing methods optimize rubrics while treating the query distribution as fixed. We identify a structural bottleneck: rubric quality is constrained by query structure. Open-ended queries yield vague rubrics; naively narrowing them introduces fabricated references that no model can verify, so all responses fail and training receives no reward signal. We present QUBRIC, a framework that co-designs queries and rubrics. Teacher-derived key points ground the rewriting of open-ended queries into scenario-based, evaluable questions. Contrastive rubric generation then turns teacher-policy gaps into query-level criteria, and learnability filtering retains only informative query-rubric pairs for GRPO training. QUBRIC achieves a +5.5 point gain on ArenaHard over the SFT baseline. Trained only on instruction-following data, it further transfers to three held-out benchmarks spanning legal, moral, and narrative reasoning (+6.3 points on average), with improvements concentrated in reasoning-related dimensions. These results provide evidence that co-designing queries and rubrics can make rubric-based RL a practical complement to RLVR beyond strictly verifiable tasks.
Reliable reward and preference signals are critical for evaluating and optimizing large language models on open-ended tasks. Rubric-based judges offer a transparent way to decompose such judgments into explicit evaluation criteria, but existing annotation-free rubric generators typically rely on a single generic evaluator. As a result, they may overlook important dimensions of human preference, a failure mode we term dimensional blind spots. To address this limitation, we propose Multi-Role Rubric Generation (MRRG), a training-free and reference-free framework that elicits evaluation criteria from multiple complementary roles and consolidates them into an auditable rubric-based scorer. This scorer can be used both to validate pairwise preferences and to provide rewards for GRPO-style Reinforcement Learning with Verifiable Rewards (RLVR). Experiments on preference validation benchmarks show that MRRG consistently outperforms single-role rubric generation baselines across multiple backbone models. Further RLVR experiments demonstrate that MRRG yields a stronger reward signal for improving open-ended generation.
Dazhi Fu, Jiuding Yang, Yiwen Guo +1
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · LIGHTSPEED · Independent Researcher