Scaling Laws for Generative Reward Models

Angel Raychev; Nathan Lile; Rafael Rafailov

cs.AIOct 22, 2026

Scaling Laws for Generative Reward Models

Authors: Angel Raychev, Nathan Lile, Rafael Rafailov

Organizations: Stanford University · Thinking Machines

Abstract

We study generative reward models (GenRMs) as online preference providers for policy optimization in creative writing. Across Qwen3 reward-model sizes from 0.6B to 14B, Thinking judges trained with GRPO have higher static Gold-label accuracy than Answer-Only judges trained with SFT. This ranking often reverses when the judges train policies through online DPO: Answer-Only rewarders yield higher peak and top-five-average policy Elo in seven of eight matched settings under a fixed, learned 32B Gold evaluator. Intermediate Thinking-judge checkpoints can also outperform later, more statically accurate checkpoints as rewarders. Sweeps over rewarder size, policy size, training duration and DPO regularization show substantial benefits from reward-model capacity, including judges larger than the policy. We analyze trajectories with evaluator-specific Elo arenas, conditional match-resampling intervals and descriptive scaling fits. These results demonstrate an evaluator-rewarder gap in the studied pipelines: fixed-distribution judging accuracy does not determine downstream optimization quality. The comparison changes judge-training objective as well as response format, and the main ranking is conditional on its learned reference rather than a universal human-quality result. A limited follow-up using a different learned judge is inconclusive overall.

Figures & tables

Observation	Evidence in this paper
Static evaluation favors Thinking GenRMs	Thinking judges have higher Gold-label validation accuracy across the studied sizes (Figure 1 ).
Online rewarder quality is different	In online DPO, Answer-Only judges usually give higher peak and top-five Gold Elo than Thinking judges in the matched sweep (Table 3 , Figure 4 ).
Judge capacity matters beyond policy size	Larger Answer-Only GenRMs improve Gold-rated policy quality beyond the fixed 4B policy size; Thinking size effects are less monotone (Figure 5 ).
Static accuracy can be over-trained for reward use	Intermediate Thinking checkpoints can outperform fully-trained Thinking checkpoints as online rewarders (Figure 3 ).

Table 1: A compact map of the paper’s central evidence. The main empirical distinction is between GenRMs as evaluators on a fixed distribution and GenRMs as rewarders inside an online optimization loop.

Axis	Values swept	Default / anchor
Proxy GenRM format	Answer-Only SFT, Thinking GRPO	Both compared
Proxy GenRM size	0.6B, 1.7B, 4B, 8B, 14B	4B unless varied
Policy size	0.6B, 1.7B, 4B, 8B	4B unless varied
Online DPO $\beta$	0.1, 0.02, 0.01, 0.005, 0.002	0.02
Training trajectory	Policy evaluation every 10 steps; intermediate Thinking GenRM checkpoints	Principal exports: about 460 steps

Table 2: Axes studied in matched and fixed-axis online-DPO sweeps, not a full factorial grid. The design separates fixed-distribution accuracy from in-loop rewarder behavior.

Figure 1: Static evaluator accuracy on the Gold Preference dataset. Thinking GenRMs exceed Answer-Only models across the studied sizes, while both trained variants substantially exceed off-the-shelf baselines. The size of the static advantage varies by model.

Figure 2: Trained versus off-the-shelf 4B rewarders training a 4B policy. Points are recorded snapshots connected in policy-training order; squares/circles mark first/last observations, not maxima. Gold is shared within this experiment family, whereas own-proxy ratings are separate evaluator fits. The chronological Gold comparison in Appendix A exposes the late trained/baseline Thinking crossover.

Figure 3: Policies trained with intermediate 4B Thinking-judge checkpoints. The retained historical image uses fitted display curves, not chronological connecting lines; its diagonal is not verdict agreement. The original account specifies 40,000 policy-training prompts and the judge-checkpoint accuracies reported in the text. This separate ablation illustrates that the most accurate static evaluator need not be the strongest online rewarder.

Figure 4: Four representative Gold-rated policy trajectories for Answer-Only SFT (blue) and Thinking GRPO rewarders (orange), including the smallest-policy exception, the 4B anchor and larger-policy/rewarder cases. All panels share the global Gold arena and vertical limits; lines connect recorded training snapshots, not fitted curves. All eight matched configurations are summarized in Table 3 and shown chronologically in Appendix A .

Policy	GenRM	Peak gap [95% CI]	Top-5 gap [95% CI]
0.6B	4B	$-35$ [ $-134$ , $49$ ]	$-19$ [ $-59$ , $25$ ]
1.7B	4B	$+113$ [ $43$ , $176$ ]	$+94$ [ $48$ , $134$ ]
4B	0.6B	$+41$ [ $-32$ , $104$ ]	$+24$ [ $-2$ , $65$ ]
4B	1.7B	$+145$ [ $104$ , $239$ ]	$+159$ [ $133$ , $214$ ]
4B	4B	$+135$ [ $49$ , $241$ ]	$+141$ [ $93$ , $189$ ]
4B	8B	$+41$ [ $-67$ , $143$ ]	$+25$ [ $-18$ , $88$ ]

Table 3: Matched Answer-Only versus Thinking Gold Elo gaps in the global all-configuration arena, reported as Answer-Only minus Thinking. Point gaps and intervals use the lightly regularized refit and 200 bootstrap resamples described in Appendix B.1 ; plotted trajectories use the original exported ratings. Answer-Only wins 7 of 8 matched cells under both summaries; the exception is the smallest-policy setting with a 4B judge.

Panel of Figure 5 — Figure 5: Rewarder-size sweeps with a fixed 4B policy, using the retained family-specific Gold arenas. Points are recorded snapshots connected in training order; squares/circles mark first/last observations. Own-proxy ratings come from separate rewarders and are not one calibrated scale. Larger Answer-Only rewarders improve Gold Elo beyond policy size; Thinking scaling is less monotone. The common-global-arena comparison is in Appendix A .

Configuration	Top-5 avg Gold Elo	Peak Gold Elo	Peak step
GenRM-4B-Ans; Policy 8B	1645	1736	450
GenRM-14B-Ans; Policy 4B	1630	1719	430
GenRM-8B-Ans; Policy 4B	1513	1549	460
GenRM-8B-Think; Policy 4B	1488	1507	460
GenRM-4B-Ans; Policy 4B	1467	1488	420
GenRM-14B-Think; Policy 4B	1437	1451	440

Table 4: Top online-DPO configurations in the global all-configuration arena, ranked by average Gold Elo over each run’s best five checkpoints. The practical frontier is dominated by larger judges and Answer-Only rewarders, with the 8B Thinking judge remaining competitive.

Panel of Figure 6 — Figure 6: Policy-size sweeps with a fixed 4B rewarder, in the retained family-specific arenas. Dots follow training chronology; squares/circles mark first/last observations. Gold and own-proxy Elo are separately fitted, and heights across the two panels are not one common arena. Larger policies improve under Answer-Only supervision in this sweep; larger Thinking-supervised policies reach their Gold peak earlier.

Figure 7: Online-DPO

\beta

sweep under an Answer-Only rewarder. Points are retained snapshots connected in training order, not fitted smoothers; squares/circles mark first/last observations. Proxy and Gold Elo are separate fits. Smaller tested

\beta

values reach higher Gold ratings, without strict monotonicity. The original account specifies 40,000 training prompts for this sweep.

Figure 8: Gold and own-proxy ratings against historically smoothed KL export coordinates for the

\beta

sweep. Lines connect exported observations in training order. The final step-260 rating has no retained KL coordinate and is omitted for each condition; no coordinate is synthesized. Negative exported values remain visible, not interpreted as raw nonnegative-estimator measurements.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Trained versus off-the-shelf rewarders through the retained training horizon. Blue denotes Answer-Only and orange Thinking; solid lines denote trained rewarders and dashed lines off-the-shelf rewarders. The upper panel uses one Gold arena; outlined points mark step 460. Each lower panel uses its own rewarder’s separately fitted ratings, so heights across those panels are not comparable. Policy and judge are 4B. Source: trained_vs_baseline/*.csv ; no smoothing or missing-step interpolation.

Figure 10: Policy-size sweep with a fixed 4B rewarder. All four panels use the common Gold arena, common vertical limits and observed policy-training steps. Blue: Answer-Only SFT rewarders; orange: Thinking GRPO rewarders. Each line represents one retained policy trajectory, not a mean across seeds. Source: all_params/gold.csv .

Figure 11: Additional rewarder-size conditions with a fixed 4B policy. Together with the 4B/4B panel of Figure 10 , these are the five rewarder sizes studied. Colors, arena and axis limits match the preceding figure. The retained curves show measured behavior, not a claim of monotonic improvement at every step or across every rewarder size.

Figure 12: Policy-size conditions against retained KL export coordinates. Gold coordinates are the same common-arena observations as Figure 10 ; only the horizontal coordinate changes. Lines follow the recovered checkpoint order. Each cell has 47 exported coordinates per mode; the two extra continuation snapshots in the 1.7B-policy cell lack a retained KL coordinate and are excluded. Horizontal limits differ across cells; vertical limits and the Gold arena are shared. Original KL preprocessing is not fully reconstructed.

Figure 13: Rewarder-size conditions against retained KL export coordinates. Gold coordinates match Figure 11 . Only uniquely matched exported observations are shown. The two extra continuation snapshots in the 0.6B-rewarder cell are absent from these KL exports and are excluded. Negative exported values remain visible without being interpreted as raw KL measurements. This is not a fitted KL scaling law.

Figure 14: Observed capacity sweeps in the common Gold arena. Left: rewarder size varies while the policy is 4B. Right: policy size varies while the rewarder is 4B. Each point is the average of the five highest recorded Gold-rated snapshots of that trajectory. Parameter-size axes are logarithmic and labeled with actual sizes. Blue: Answer-Only; orange: Thinking. These are descriptive observed summaries, not compute-normalized optima, predictive scaling fits, or independent-seed confidence intervals.

Metric	$R^{2}$	RMSE	Ans. policy	Ans. judge	Think policy	Think judge
Peak Elo	0.954	51.8	186.0	144.7	113.3	113.3
Top-5 Elo	0.952	51.2	175.8	135.7	119.0	115.6

Appendix

Table 5: In-sample log-size fits to the 16 matched global configurations. Slope columns are fitted Gold-Elo differences per parameter doubling along each axis, not observed new doublings, compute-matched gains or out-of-sample predictions. RMSE is in Elo units.

Rewarder size (4B policy)	Answer-Only wins	Comparisons	95% Wilson interval
4B	17	40	28.5–57.8%
14B	25	40	47.0–75.8%
Combined	42	80	41.7–63.1%

Appendix

Table 6: A different learned judge does not establish an overall preference advantage in this limited diagnostic. These descriptive binomial intervals are not prompt-cluster or retraining intervals; the 80 comparisons contain 76 unique prompts, with four prompts shared across settings.

Figure 15: Retained OpenOrca policy-optimization trajectories with off-the-shelf rewarders. Axes show net preference rates, not Elo. Recovered source identifies online DPO from an SFT-initialized policy and a separate 32B Thinking reference evaluator. This is not a trained-GenRM cross-domain replication; exact numerical reconstruction of the retained image remains unavailable.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing
Mar 16, 2026Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji +2Memory-Augmented RL RL for Language Models
School of Information, Renmin University of China, Beijing, China · School of Information, BRAIN, Renmin University of China, Beijing, China
EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
Oct 4, 2026Xuancheng Li, Beining Wang, Haitao Li +7Reward Modeling Process Reward Models
Department of Computer Science and Technology, Tsinghua University, Beijing, China · Tencent, Beijing, China
RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time
Apr 13, 2026Haozhe Wang, Cong Wei, Weiming Ren +3Reward Modeling Multimodal Reward Modeling
HKUST · University of Waterloo · Alibaba

Mar 16, 2026J/K move · Enter open · S save

Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing

Jihao Zhao, Shuaishuai Zu, Zhiyuan Ji, Chunlai Zhou+1

School of Information, Renmin University of China, Beijing, China · School of Information, BRAIN, Renmin University of China, Beijing, China

Memory-Augmented RL RL for Language Models Language Model Generation Evaluation Rubric-Based RL Reinforcement Learning

Open paper AskPDF arXiv

Training models for open-ended writing requires diverse prompts and sufficiently precise specifications of response quality. Constructing such specifications at scale entails recovering task requirements, consolidating overlapping criteria, and defining observable differences between performance levels. We introduce Writer-R1, a writing training pipeline built around Inductive Rubric Generation (IRG), which integrates these operations into an automated rubric construction procedure. The pipeline expands an open writing corpus and uses the resulting prompt-specific rubrics throughout supervised training and reinforcement learning (RL). Each rubric is concatenated with the user request and also defines the reward criteria, making the requirements used for assessment available to the writer before generation. A compact model, RubricGen-4B, learns to supply rubrics for new requests. Writer-R1 improves WritingBench scores by 0.182 and 0.158 over the 4B and 8B backbones, respectively, with gains also on HelloBench. Training ablations favor IRG over direct rubric generation, and further analyses support rubric transfer and the robustness of writer improvements across evaluators. These results support using inductively mined rubrics as both generation guidance and reward specifications for open-ended writing.

Abstract

Reward Modeling

Explore similar work

Writer-R1: Dynamic Rubric Mining for Reinforcement Learning of Open-Ended Writing

EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans

RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time