Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty

Chao Xue; Yao Wang; Mengqiao Liu; Di Liang; Xingsheng Han; Peiyang Liu; Xianjie Wu; Chenyao Lu

cs.CLApr 11, 2026

Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty

Authors: Chao Xue, Yao Wang, Mengqiao Liu, Di Liang, Xingsheng Han, Peiyang Liu, Xianjie Wu, Chenyao Lu,

Organizations: University of New South Wales, Australia · Tencent Hunyuan, China · Tencent Yuanbao, China · Peking University, China ·

Abstract

Recent advancements in the Generative Reward Model (GRM) have demonstrated its potential to enhance the reasoning abilities of LLMs through Chain-of-Thought (CoT) prompting. Despite these gains, existing implementations of GRM suffer from two critical limitations. First, CoT prompting is applied indiscriminately to all inputs regardless of their inherent complexity. This introduces unnecessary computational costs for tasks amenable to fast, direct inference. Second, existing approaches primarily rely on voting-based mechanisms to evaluate CoT outputs, which often lack granularity and precision in assessing reasoning quality. In this paper, we propose E-GRM, an efficient generative reward modeling framework grounded in model-internal uncertainty. E-GRM leverages the convergence behavior of parallel model generations to estimate uncertainty and selectively trigger CoT reasoning only when needed, without relying on handcrafted features or task-dependent signals. To improve reward fidelity, we introduce a lightweight discriminative scorer trained with a hybrid regression--ranking objective to provide fine-grained evaluation of reasoning paths. Experiments on multiple reasoning benchmarks show that E-GRM substantially reduces inference cost while consistently improving answer accuracy, demonstrating that model-internal uncertainty is an effective and general signal for efficient reasoning-aware reward modeling.

Figures & tables

Figure 1: An illustration of E-GRM that enables more efficient and accurate generative reasoning.

Figure 2: E-GRM is an enhanced generative framework improving efficiency and quality through dynamic CoT triggering (activating stepwise reasoning only when needed) and discriminative scoring (selecting optimal CoT via a lightweight reward model). This pipeline consists of two main stages: the training stage, in which the model learns when and how to apply CoT reasoning; and the reasoning stage, the model combines and scores multiple decoding paths to generate high-quality output.

Standard GRPO
$\mathcal{J}_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{\begin{subarray}{c}q\sim P(Q)\\ \{o_{i}\}\sim\pi_{\theta_{\mathrm{old}}}\end{subarray}}\frac{1}{G}\sum_{i=1}^{G}\frac{1}{\|o_{i}\|}\sum_{t=1}^{\|o_{i}\|}$
$\qquad\left\{\min\left[r_{i,t}\hat{A}_{i,t},\mathrm{clip}(r_{i,t},1-\epsilon,1+\epsilon)\hat{A}_{i,t}\right]\right.$
$\qquad\left.-\beta\,\mathbb{D}_{\mathrm{KL}}[\pi_{\theta}\,\\|\,\pi_{\mathrm{ref}}]\right\}$
$\text{where}\;\;r_{i,t}=\frac{\pi_{\theta}(o_{i,t}\|q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\|q,o_{i,<t})}$
E-GRM Preference Optimization
$\mathcal{J}_{\mathrm{E-GRM}}(\theta)=\mathbb{E}_{\begin{subarray}{c}(x,r^{+},r^{-})\sim\mathcal{D}_{\text{pref}}\end{subarray}}\Bigg[\frac{1}{2}\sum_{\kappa\in\{+,-\}}\frac{1}{\|r^{\kappa}\|}\sum_{t=1}^{\|r^{\kappa}\|}$

Figure 3: Comparison of standard GRPO and our extended formulation used in E-GRM. While both leverage group-based relative optimization, E-GRM explicitly incorporates paired preference signals through the discriminative scorer

\mathcal{S}_{\phi}

, enhancing alignment with nuanced response quality distinctions.

Models	Chat	Math	Code	Safety	Easy	Normal	Hard	Avg
Scalar RMs
steerlm-70b	56.4	53.0	49.3	51.2	48.3	54.9	54.3	52.5
tulu-v2.5-70b-preference-mix-rm	58.2	51.4	55.5	87.1	72.8	65.6	50.7	63.0
Mistral-7B-instruct-Unified-Feedback	56.5	58.0	51.7	86.8	87.1	67.3	35.3	63.2
RM-Mistral-7B	57.4	57.0	52.7	87.2	88.6	67.1	34.9	63.5
Eurus-RM-7b	59.9	60.2	56.9	86.5	87.2	70.2	40.2	65.9

Table 1: RM-Bench evaluation across domains and complexity levels. The proposed Qwen-Instruct-*B-Ours models exhibit excellent capabilities in multiple categories, reaching a top average performance of 79.2% while excelling in math, chat, code, and challenging tasks. Bold denotes top performance. Underlined denotes runner-up.

	Helpfulness		Harmlessness
Models	BoN	Pairwise	BoN	Pairwise	Overall
Scalar RMs
Tulu-v2.5-13b-preference-mix-rm	0.355	0.562	0.351	0.545	0.453
Skywork-Reward-Gemma-2-27B	0.472	0.653	0.561	0.721	0.602
Internlm2-20b-reward	0.585	0.763	0.499	0.670	0.629
ArmoRM-Llama3-8B-v0.1	0.636	0.787	0.497	0.663	0.646

Table 2: RMB evaluation results. Bold denotes top performance. Underlined denotes runner-up performance.

Scalar RMs
Models	Chat	Chat_Hard	Safety	Reasoning	Overall
SteerLM-RM 70B	91.3	80.3	92.8	90.6	88.8
Cohere-0514	96.4	71.3	92.3	97.7	89.4
GenRMs
Llama3.1-8B-Instruct	85.5	48.5	75.6	72.1	70.4
Prometheus-8*7B-v2	93.0	47.1	80.5	77.4	74.5

Table 3: Performance of various models on the RewardBench benchmark. Qwen-Instruct-*B-Ours consistently outperforms baseline methods across RewardBench evaluations. Underlined denotes runner-up performance.

Method	Accuracy	Avg. Latency	Task-specific Heuristic?
Forced-CoT	75.1	3.8	No
Rule-based	70.5	2.1	Yes
AdaCoT	76.8	2.9	Yes
E-GRM (Ours)	78.4	2.2	No

Table 4: Comparison of CoT triggering strategies. Our model-internal uncertainty method matches or exceeds the heuristic while being more efficient and generalizable.

Variant	MATH	HelpSteer2	RMB Harmlessness
E-GRM (Std. GRPO)	76.9	81.5	0.765
E-GRM (Extended)	78.4	82.3	0.775

Table 5: Ablation on preference optimization. Our GRPO formulation provides a consistent boost, demonstrating its utility for paired preference data.

Variant	Acc. (%)	FLOPs (T)	Latency (s)
Full E-GRM	78.4	15.7	2.2
w/o Dynamic CoT	75.2	23.4	3.4
w/o Discrim. Scoring	72.8	15.9	2.2
Base CoT-GRM	69.1	23.7	3.6

Table 6: Ablation study on the MATH dataset. Removing dynamic CoT increases cost significantly, while removing scoring causes the largest accuracy drop.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Prompt for Cold-start Long-CoT Sampling
As an AI Quality Auditor, rigorously evaluate <prompt, response> pairs through this structured framework:
<foundation>
Basic Compliance Check
Safety : Contains violent/discriminatory/illegal content? $\checkmark$ Pass $\times$ Fail
Legal Compliance : Violates data privacy/IP laws? $\checkmark$ Pass $\times$ Fail
Ethical Standards : Aligns with social norms? $\checkmark$ Pass $\times$ Fail

Appendix

Figure 4: Prompt for Cold-start Long-CoT Sampling evaluation framework.

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 325 papers

Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models
Mar 2, 2026Qiyuan Zhang, Yufei Wang, Tianhe Wu +5Reward Modeling Reward Model Evaluation
City University of Hong Kong · Tencent Hunyuan · MBZUAI
Rewarding Efficient Reasoning Improves Abstention on Underspecified Tasks in Reasoning Models
Aug 5, 2026Polina Tsvilodub, Max Höth, Michael Franke +2LLM Abstention Large Reasoning Models
University of Tübingen · Aleph Alpha Research · Lab1141
Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning
Aug 5, 2026Qiyuan Zhu, Dezhi Li, Pengyu Cheng +8Overthinking in Language Models KV Caching
1HKUST · 2XJTU · 3TSE +1

Mar 2, 2026J/K move · Enter open · S save

Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models

Qiyuan Zhang, Yufei Wang, Tianhe Wu, Can Xu+4

City University of Hong Kong · Tencent Hunyuan · MBZUAI

Reward Modeling Reward Model Evaluation CoT Reasoning

Open paper AskPDF arXiv

Recent advancements in Generative Reward Models (GRMs) have demonstrated that scaling the length of Chain-of-Thought (CoT) reasoning considerably enhances the reliability of evaluation. However, current works predominantly rely on unstructured length scaling, ignoring the divergent efficacy of different reasoning mechanisms: Breadth-CoT (B-CoT, i.e., multi-dimensional principle coverage) and Depth-CoT (D-CoT, i.e., substantive judgment soundness). To address this, we introduce Mix-GRM, a framework that reconfigures raw rationales into structured B-CoT and D-CoT through a modular synthesis pipeline, subsequently employing Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) to internalize and optimize these mechanisms. Comprehensive experiments demonstrate that Mix-GRM establishes a new state-of-the-art across five benchmarks, surpassing leading open-source RMs by an average of 8.2%. Our results reveal a clear divergence in reasoning: B-CoT benefits subjective preference tasks, whereas D-CoT excels in objective correctness tasks. Consequently, misaligning the reasoning mechanism with the task directly degrades performance. Furthermore, we demonstrate that RLVR acts as a switching amplifier, inducing an emergent polarization where the model spontaneously allocates its reasoning style to match task demands. The synthesized data and models are released at Hugging Face, and the code is released at Github.

Abstract

Reward Modeling

Explore similar work

Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Models

Rewarding Efficient Reasoning Improves Abstention on Underspecified Tasks in Reasoning Models

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning