Rubric Curriculum RL: Exploiting the Generation-Verification Gap in Non-Verifiable Domains
Authors: Sumeet Ramesh Motwani, Tejas Krishnan, Alistair Letcher, Charles London, Suhaas Bhat, Huitian Jiao, Philip Torr, Riashat Islam, Christopher Summerfield, Jakob Foerster, Christian Schroeder de Witt, Qilong Gu, Shital Shah
Organizations: University of Oxford · Microsoft Research
Reinforcement learning with verifiable rewards (RLVR) has led to significant gains in math and code generation. Extending these gains to open-ended domains remains challenging: ground-truth verification is unavailable, human annotation is expensive, and learned reward models are prone to reward hacking. The generation-verification gap, where judging good outputs is easier than producing them, offers a potential path to improvement. However, rubric-based judges are often imprecise and stationary, so policies can overfit or reward hack them. We introduce Rubric Curriculum RL (RcRL), which exploits this gap without new data, human annotations, or stronger teacher models. We combine pairwise preferences over rollouts with a curriculum over rubric criteria, yielding a more discriminative signal than absolute scoring and a non-stationary objective. Whereas baselines plateau within a few hundred steps, RcRL preserves output entropy and keeps improving past 1000 steps. On creative writing, RcRL achieves a 70.5% human win rate against the base model, with gains across benchmarks and judges. On HealthBench, RcRL improves over the base model by 14.6% (full set) and 25.4% (consensus subset), outperforming all training baselines, including instance-specific rubrics.
Figures & tables
Figure 1: RcRL vs. RaR on FictionBench. Win rates against the base Qwen3-4B model over training, averaged across 5 seeds; shading indicates ±1 SEM. RcRL (red) improves over time, reaching ∼ 75% by step 1300, while RaR (blue) plateaus early around 55–65%. This demonstrates that curriculum learning over evaluation criteria enables sustained improvements by allowing additional training compute to be spent productively.
Figure 2: Overview of Rubric Curriculum RL (RcRL). (Left) The challenge: training with a static rubric leads to reward hacking, where the model optimizes for the reward signal without improving true quality. (Right) Our solution: RcRL progressively introduces evaluation criteria, using pairwise comparisons to allocate rewards. The curriculum advances when current criteria have improved (Section 5.2 ).
Figure 3: Best-of- N inference scaling on FictionBench. Win rate of self-selected Best-of- N against the N=1 baseline, judged by GPT-5.2 as ground truth. Win rates exceed 50% from N=2 onwards (Claim 1), grow monotonically with N , and are uniformly larger for the 8B model than the 4B model (Claim 2).
Figure 4: Reward hacking with static rubrics. (Left) Absolute train reward increases steadily over the course of training, reaching near-saturation. (Right) Evaluation win rate against the base model on FictionBench shows no consistent improvement as training progresses, indicating a performance mismatch.
FictionBench
LechMazur
Method
ELO
Win%
Score
Hratio
Qwen 3 4B (Baseline)
1000
–
77.0
–
Qwen 3 4B RaR
1052
57.5
80.3
0.73
Ours (RcRL)
1181
73.9
80.9
0.93
Table 1: Additional benchmark results. FictionBench and the LechMazur Writing benchmark. Best results per model in bold .
GPT-5.2
Claude 4.5 Opus
Deepseek V3.2
Method
ELO
Rubric
Win%
ELO
Rubric
Win%
ELO
Rubric
Win%
Qwen 3 4B (Baseline)
1000
45.4
–
1000
40.8
–
1000
51.2
–
Qwen 3 4B RaR
1210
48.5
61.1
1051
41.1
51.7
1326
55.0
53.9
Ours (RcRL)
1577
50.1
69.3
1105
40.0
59.2
1524
58.1
65.4
Table 2: Creative Writing Bench v3 evaluation results. Three frontier models (GPT-5.2, Claude 4.5 Opus, Deepseek V3.2) serve as independent judges, evaluating outputs from each method via pairwise comparisons. We report ELO ratings (baseline anchored at 1000), rubric scores (0–100 scale), and win rates against the Qwen 3 4B baseline (%). ELO ratings reflect relative rankings within each judge and are not comparable across judges. Best results per metric and judge in bold .
Figure 5: Head-to-head win rates on FictionBench. RcRL achieves substantial gains over the Qwen3-4B baseline against all opponents: +28% against RaR, +28% against Qwen3-8B, and +26% against Qwen3-14B.
Figure 6: Human evaluation results. Win rates of RcRL-trained Qwen3-4B against the base model across seven criteria, averaged over three annotators who conducted 500 comparisons each. RcRL achieves a 70.5% win rate on Best Story Overall , with consistent improvements across all criteria.
Figure 7: Scaling behavior of RcRL. Win rates against base models across model sizes for Qwen3 (red) and Llama-3.2 (blue). Performance improves with scale, from 54.6% at 0.6B to 75.2% at 14B parameters. The Llama-3.2-3B result (76.4%) demonstrates that RcRL generalizes across model families.
CW Bench v3
FictionBench + LM
Method
ELO
Rubric
Win%
ELO
Win%
LM
No RL Training
Qwen 3 4B (Baseline)
1000
45.4
–
1000
–
77.0
No Curriculum
Absolute RaR
1210
48.5
61.1
1052
57.5
80.3
Dropout Absolute RaR
1166
49.8
55.4
1113
65.7
80.9
Table 3: Ablation study and baseline comparisons. We evaluate the contribution of each component of RcRL. “ + ” denotes adding a component to the base method, “ − ” denotes removing from the full method. All models trained from Qwen 3 4B. Win rates computed against the base model. Best results in bold .
Figure 8: Single-criterion training dynamics. Win rates averaged across five models, each trained on a single criterion (e.g., plot coherence, character development). Per-criterion performance (red) improves steadily, while overall story quality (blue) peaks around step 100 before declining.
Figure 9: Policy entropy during training. The static rubric baseline exhibits a steady decline in entropy, indicating mode collapse toward a narrow set of outputs. RcRL maintains consistently higher entropy throughout training.
Method
HealthBench Δ
Consensus Δ
Qwen 3 4B (Baseline)
–
–
Qwen 3 4B RaR
+8.26%
+7.53%
Instance-specific
+13.52%
+10.52%
Absolute RcRL
+13.40%
+23.87%
RcRL (Ours)
+14.60%
+25.38%
Table 4: HealthBench cross-domain results. Score improvement over Qwen 3 4B on full HealthBench and the consensus subset (criteria validated by multiple annotators). Best in bold .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
n -gram repetition score
Qwen3-4B
11.51
Qwen3-4B RaR
13.58
RcRL
7.52
Appendix
Table 5: N -gram repetition rates (lower is better).
Parameter
Value
GRPO Configuration
Learning rate
5×10−6
LR scheduler
Cosine
Warmup steps
50
Per-device batch size
2
Gradient accumulation steps
4
Appendix
Table 6: Training hyperparameters for RcRL. Configuration for Qwen3-4B experiments. We use LoRA for parameter-efficient fine-tuning.
Step
Criterion
Evaluation Focus
0
Prompt Alignment
Does the story fully address the requirements of the prompt? Are all key elements, details, constraints, or events implied or explicitly stated in the prompt incorporated into the story in a meaningful way?
100
Plot Coherence & Progression
Does the story present a clear and logically consistent plot? Do events follow naturally from one another with a sense of cause and effect? Are transitions between scenes smooth?
200
Character Depth & Motivation
Do the characters have believable goals, emotions, and internal conflicts? Are their actions motivated and consistent with their personalities?
300
Narrative Tension & Stakes
Does the story sustain a sense of tension, uncertainty, or conflict that keeps readers engaged? Are there meaningful stakes—emotional, physical, or moral—that make the outcome matter?
400
Show, Don’t Tell
Does the story rely on vivid imagery, sensory detail, and action to communicate emotion or meaning? Does it avoid excessive exposition and direct statements of feelings or morals?
500
Emotional Resonance
Does the story evoke a genuine emotional response? Are the emotions integrated naturally into the narrative rather than being forced or sentimental?
Appendix
Table 7: Creative writing rubric criteria in curriculum order. Each criterion is introduced at the indicated training step and remains active for all subsequent steps. The ordering progresses from foundational craft elements (prompt alignment, plot coherence) through character and tension, to stylistic refinement (voice, originality). This progression reflects a pedagogical approach: models first learn to write coherent, on-topic stories before optimizing for subtler qualities like thematic depth and distinctive voice.
Figure 10: Curriculum weighting evolution. Diagram of how the curriculum weighting evolves over the course of training. The newly introduced criterion always has a weight of 21 with the remaining weight distributed uniformly over the remaining criteria.
Step
Criterion
Evaluation Focus
0
Factual Correctness
Does the response contain factually correct medical information, accurately representing dosages, mechanisms, contraindications, and expected outcomes without contradicting clinical consensus?
50
No Harmful or Unsafe Advice
Does the response avoid recommendations that could directly harm the patient—such as dangerous interactions, contraindicated treatments, or advice that delays necessary emergency care?
100
No Fabrication
Does the response avoid inventing clinical details not provided by the user—such as lab values, diagnoses, or examination findings—rather than presenting assumptions as facts?
150
Instruction Following
Does the response directly address what the user asked, following any specified format or scope rather than going off-topic?
200
Completeness
Does the response cover the important aspects of the query—causes, symptoms, treatment options, risks, and next steps—rather than providing only a partial answer?
250
Specificity & Actionability
Does the response provide concrete, actionable information—specific tests, treatments, or warning signs—rather than vague advice like “consult a doctor”?
Appendix
Table 8: HealthBench rubric criteria in curriculum order. Each criterion is introduced at the indicated training step and remains active for all subsequent steps. Compared to creative writing, there is no ordering justification, with criteria synthesized from both the HealthBench instance-specific rubrics and rubric criteria from Yang et al. (2026) .
Figure 11: LechMazur per-criterion scores. Heatmap showing performance of Qwen3-4B (baseline), RaR, and RcRL across the 18 evaluation criteria. Q1–Q8 assess major story aspects; Q9A–Q9J assess element integration. RcRL shows particularly strong gains on element integration criteria.
Criterion
Fleiss’ κ
Interpretation
Best Story Overall
0.548
Moderate
Appendix
Table 9: Inter-annotator agreement. Fleiss’ Kappa score for overall preference. This score is consistent with prior work on creative writing evaluation ( Gómez-Rodríguez and Williams, 2023 ) .
Table 10: Results from human expert annotations. Preference bars (right) show win rates of RcRL over the baseline Qwen 3 4B model across criteria, based on 500 pairwise comparisons.
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Qwen Business Unit of Alibaba +1