Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models

Qiyuan Zhang; Yufei Wang; Tianhe Wu; Can Xu; Qingfeng Sun; Kai Zheng; Xue Liu; Chen Ma

cs.AIMar 2, 2026

Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models

Authors: Qiyuan Zhang, Yufei Wang, Tianhe Wu, Can Xu, Qingfeng Sun, Kai Zheng, Xue Liu, Chen Ma

Organizations: City University of Hong Kong · Tencent Hunyuan · MBZUAI

Abstract

Recent advancements in Generative Reward Models (GRMs) have demonstrated that scaling the length of Chain-of-Thought (CoT) reasoning considerably enhances the reliability of evaluation. However, current works predominantly rely on unstructured length scaling, ignoring the divergent efficacy of different reasoning mechanisms: Breadth-CoT (B-CoT, i.e., multi-dimensional principle coverage) and Depth-CoT (D-CoT, i.e., substantive judgment soundness). To address this, we introduce Mix-GRM, a framework that reconfigures raw rationales into structured B-CoT and D-CoT through a modular synthesis pipeline, subsequently employing Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) to internalize and optimize these mechanisms. Comprehensive experiments demonstrate that Mix-GRM establishes a new state-of-the-art across five benchmarks, surpassing leading open-source RMs by an average of 8.2%. Our results reveal a clear divergence in reasoning: B-CoT benefits subjective preference tasks, whereas D-CoT excels in objective correctness tasks. Consequently, misaligning the reasoning mechanism with the task directly degrades performance. Furthermore, we demonstrate that RLVR acts as a switching amplifier, inducing an emergent polarization where the model spontaneously allocates its reasoning style to match task demands. The synthesized data and models are released at Hugging Face, and the code is released at Github.

Figures & tables

Figure 1: The pipeline of the Mix-GRM. (i) Standardization : We extract raw rationales into modular Principle–Judgment–Verdict units. (II) Mechanism Synthesis : We reconstruct modules into B-CoT for preference or D-CoT for correctness. (III) Training & Inference : Following SFT and RLVR training, the model achieves mechanism-adaptive alignment, automatically deploying the optimal mechanism for inference and providing reliable signals for downstream tasks like Offline RL and test-time scaling.

Models	Stage	Data	Benchmarks
Models	Stage	Data	RB-v1 †	RB-v2 †	RM-Bench	RMB	PPE	Avg.
Reference: Proprietary Models
DeepSeek-V3.2	–	–	95.5	92.1	91.4	83.9	69.0	86.4
Gemini-3-Flash	–	–	95.3	91.1	93.8	79.2	76.4	87.2
Open-Source Baselines
Skywork-Reward-8B	BT	44K	93.9	79.7	72.4	74.4	61.7	76.5

Table 1: Performance of RMs on reward benchmarks. Among open-source models, the highest score per column is bolded , and the second-highest is underlined . “Overall” denotes the average score within each benchmark. Proprietary LLMs (gray rows) are included for reference. † RB-v1 / v2 refers to RewardBench v1 and v2.

Models	Preference Domain						Correctness Domain						Overall
Models	RB-v1	RB-v2	RM-B †	RMB	PPE	Avg.	RB-v1	RB-v2	RM-B †	RMB	PPE	Avg.	Overall
Baselines
FARE-8B	85.0	57.3	66.9	82.9	59.6	$70.4\phantom{{}_{\uparrow 0.0}}$	85.2	67.3	63.0	88.1	63.3	$73.3\phantom{{}_{\uparrow 0.0}}$	71.9
RubricRM-8B	82.4	56.0	62.2	77.5	64.9	$68.6\phantom{{}_{\uparrow 0.0}}$	87.6	64.2	57.6	86.5	60.4	$71.3\phantom{{}_{\uparrow 0.0}}$	70.0
DeepSeek-GRM	80.6	59.6	64.0	76.8	59.8	$68.2\phantom{{}_{\uparrow 0.0}}$	76.6	55.8	56.6	86.8	56.8	$66.5\phantom{{}_{\uparrow 0.0}}$	67.4
Ours: Mix-GRMs

Table 2: Performance of RMs grouped by domain. “Avg.” denotes the domain average. We annotate the performance gap relative to the Base-GRM in SFT baseline within the same stage using colored subscripts (

\uparrow

for gain,

\downarrow

for drop). Highest score per column is bolded , second-highest is underlined . † RM-B refers to RM-Bench.

Models	Instruction-Following			Mathematical Reasoning
Models	Alpaca-v2	Arena-Hard	Avg.	GSM8k	MATH	STEM	TabMWP	Avg.
SFT	6.4	4.2	5.3	75.1	25.2	38.6	40.9	45.0
DPO Training (Different RMs)
$\hookrightarrow$ RubricRM-8B	8.5	12.5	10.5	76.0	26.9	41.4	38.8	45.9
$\hookrightarrow$ FARE-8B	8.9	15.1	12.0	75.7	26.9	39.0	41.4	45.8
$\hookrightarrow$ RM-R1-Instruct	7.9	14.3	11.1	76.3	26.5	38.5	41.7	45.8

Table 3: Performance of DPO-trained policy models using different reward models on instruction-following and math-reasoning benchmarks. “Avg.” is the average score of all benchmarks in each domain. In each column, the highest score is bolded and the second-highest is underlined .

Figure 2: Best-of-

10

performance across four challenging reasoning and coding benchmarks. Mix-GRM (ours) consistently achieves the highest accuracy across all tasks, effectively identifying solutions in both mathematical and code generation scenarios. Red and green lines denote random and oracle selection baselines.

Figure 3: Structural evolution of CoT mechanisms. The chart tracks 4 indicators: the average token length per judgment, average principle count, and the percentage of CoT classified as having Breadth or Depth characteristics.

Figure 4: Ablation of B-CoT synthesis. (a) Aggregation Scale: Performance as aggregated rationales (

N

) increases from 1 (Vanilla) to 4. (b) Principle Selection: Comparison of Random, Full, and Consistency (Top-10) selection from the

N=4

pool. Orange / blue lines denote Preference/Correctness; dashed lines indicate the Vanilla baseline.

CoT Style	Reasoning Passes	Avg. SFT Tokens	Avg. RLVR Rollouts	Avg. Inference Tokens
D-CoT	2 (Trace + Gen)	624	682	702
B-CoT	2 ( $N=2$ Sample)	711	830	824
Mix-CoT (Ours)	Adaptive	648	725	731

Table 4: Accounting table reporting the compute and token costs. Token consumption across our proposed methods remains within the same order of magnitude.

Case 1: Preference Domain (B-CoT Wins)
Instruction: アフガニスタンがパキスタンの傀儡というのは本当ですか？ (Is it true that Afghanistan is a puppet of Pakistan?)
Response A (Rejected): Response B (Chosen): [Language: English] "A sensitive topic! …While Pakistan has historically exerted significant influence…" [Language: Japanese] 「アフガニスタンとパキスタンの関係については… 簡単に『傀儡』と断言するのは適切ではなく…」
Reasoning Comparison
✗ Vanilla-CoT	Assistant A offers a comprehensive breakdown of historical context… Verdict: [[A]] (Fail: Ignored language mismatch)
✓ B-CoT	1. Principle of Linguistic Alignment: "Assistant B’s response is in Japanese… Sub-Verdict: <<B>>" 2. Principle of Contextual Nuance: "Assistant A provides a detailed explanation… Sub-Verdict: <<A>>" 3. Principle of Cultural Sensitivity: "…" Final Verdict: [[B]] (Success)
✗ D-CoT	1. Principle of Comprehensive Contextualization: The evaluation needs deep reasoning, and let’s solve the instruction: アフガニスタンが…. Based on this, Assistant A excels… Sub-Verdict: <<A>> 2. Principle of Balanced Presentation: … Final Verdict: [[A]] (Fail: Tunnel vision on content depth)

Table 5: Case Study. Case 1 shows how B-CoT aggregates diverse principles to identify subtle preference nuances. Case 2 shows how D-CoT performs step-by-step verification to catch logical errors.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Table 10

Source Dataset	Domain	Samples
HelpSteer-3 (Single-Turn)	General Chat	4,973
	STEM	2,321
	Code	4,322
	Multilingual	3,260
Code-Preference	Code	4,000
Math-DPO	Math	4,000

Appendix

Table 8: Composition and statistics of the training data sampled from domain-specific sources.

Benchmark	Tasks	Samples
RewardBench	Chat, Math, Code, Safety	2,985
RewardBench-v2	Focus, IF, Factuality, Math, Safety, Ties	1,865
RM-Bench	Chat, Math, Code, Safety	11,943
RMB	Harmfulness, Helpfulness (General, Code)	14,725
PPE (Exclude Tie)	Chat, MMLU-Pro, GPQA, IFEval, MBPP	22,991

Appendix

Table 9: Task coverage of the evaluated general reward benchmarks.

Models	Reward Bench	Reward Bench-v2		RM Bench	RMB	PPE		Avg.
	Chat	Focus	IF	Chat	Helpfulness	Human	IF
Open-sourced Reward Models
JudgeLRM-7B	75.8	46.8	31.3	68.4	79.3	60.2	54.3	59.4
RM-R1-7B (Distill)	75.3	58.7	24.4	58.7	63.2	57.1	53.0	55.8
RM-R1-7B (Instruct)	80.5	85.3	28.8	64.7	65.1	57.1	53.0	62.1
FARE-8B	85.0	78.4	36.3	66.9	82.9	63.4	55.7	67.0

Appendix

Table 10: Performance of RMs on preference-related sub-tasks. “Avg.” is the average score among sub-tasks. Best per column is bolded ; second-best is underlined .

Models	RewardBench		RewardBench-v2		RM-Bench		RMB	PPE				Avg.
	Code	Math	Factuality	MATH	CODE	MATH	CODE	MMLU-Pro	MATH	GPQA	MBPP
Open-sourced Reward Models
JudgeLRM-7B	81.6	77.2	53.8	76.5	51.0	86.7	82.1	57.2	65.5	51.3	52.3	66.8
RM-R1-7B (Distill)	91.9	93.7	28.3	73.2	53.3	85.8	74.8	66.7	89.4	56.3	64.4	70.7
RM-R1-7B (Instruct)	81.7	84.1	42.6	67.8	56.7	72.7	74.7	67.0	89.1	55.9	64.8	68.8
FARE-8B	88.1	82.3	65.8	68.9	57.0	69.1	88.1	63.2	79.3	55.2	55.4	70.2

Appendix

Table 11: Performance of RMs on correctness-related sub-tasks. “Avg.” is the average within this block. Best per column is bolded ; second-best is underlined .

Synthesis Stage	Raw Rationale	Merge B-CoT	Generate D-CoT
Accuracy (%)	87.1	90.2	88.5

Appendix

Table 12: Solution accuracy tracked across different stages of the synthesis pipeline, demonstrating the stability of the intermediate transformations.

Training Data (9K)	w/o Filtering (Noisy)	w/ Filtering (Verified)
SFT Performance (%)	69.8	70.1

Appendix

Table 13: SFT performance comparison between noisy and strictly filtered synthesized data.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty
Apr 11, 2026Chao Xue, Yao Wang, Mengqiao Liu +11Reward Modeling Efficient Language Model Reasoning
University of New South Wales, Australia · Tencent Hunyuan, China · Tencent Yuanbao, China +2
EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
Oct 4, 2026Xuancheng Li, Beining Wang, Haitao Li +7Reward Modeling Process Reward Models
Department of Computer Science and Technology, Tsinghua University, Beijing, China · Tencent, Beijing, China
ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training
Apr 8, 2026Yu Liang, Liangxin Liu, Longzheng Wang +5Reward Modeling LLM Alignment
Baidu Inc., Beijing, China

Apr 11, 2026J/K move · Enter open · S save

Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty

Chao Xue, Yao Wang, Mengqiao Liu, Di Liang+10

University of New South Wales, Australia · Tencent Hunyuan, China · Tencent Yuanbao, China+2

Reward Modeling Efficient Language Model Reasoning CoT Reasoning

Open paper AskPDF arXiv

Recent advancements in the Generative Reward Model (GRM) have demonstrated its potential to enhance the reasoning abilities of LLMs through Chain-of-Thought (CoT) prompting. Despite these gains, existing implementations of GRM suffer from two critical limitations. First, CoT prompting is applied indiscriminately to all inputs regardless of their inherent complexity. This introduces unnecessary computational costs for tasks amenable to fast, direct inference. Second, existing approaches primarily rely on voting-based mechanisms to evaluate CoT outputs, which often lack granularity and precision in assessing reasoning quality. In this paper, we propose E-GRM, an efficient generative reward modeling framework grounded in model-internal uncertainty. E-GRM leverages the convergence behavior of parallel model generations to estimate uncertainty and selectively trigger CoT reasoning only when needed, without relying on handcrafted features or task-dependent signals. To improve reward fidelity, we introduce a lightweight discriminative scorer trained with a hybrid regression--ranking objective to provide fine-grained evaluation of reasoning paths. Experiments on multiple reasoning benchmarks show that E-GRM substantially reduces inference cost while consistently improving answer accuracy, demonstrating that model-internal uncertainty is an effective and general signal for efficient reasoning-aware reward modeling.

Abstract

Reward Modeling

Explore similar work

Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty

EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans

ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training