Recent advancements in the Generative Reward Model (GRM) have demonstrated its potential to enhance the reasoning abilities of LLMs through Chain-of-Thought (CoT) prompting. Despite these gains, existing implementations of GRM suffer from two critical limitations. First, CoT prompting is applied indiscriminately to all inputs regardless of their inherent complexity. This introduces unnecessary computational costs for tasks amenable to fast, direct inference. Second, existing approaches primarily rely on voting-based mechanisms to evaluate CoT outputs, which often lack granularity and precision in assessing reasoning quality. In this paper, we propose E-GRM, an efficient generative reward modeling framework grounded in model-internal uncertainty. E-GRM leverages the convergence behavior of parallel model generations to estimate uncertainty and selectively trigger CoT reasoning only when needed, without relying on handcrafted features or task-dependent signals. To improve reward fidelity, we introduce a lightweight discriminative scorer trained with a hybrid regression--ranking objective to provide fine-grained evaluation of reasoning paths. Experiments on multiple reasoning benchmarks show that E-GRM substantially reduces inference cost while consistently improving answer accuracy, demonstrating that model-internal uncertainty is an effective and general signal for efficient reasoning-aware reward modeling.
Figures & tables
Figure 1: An illustration of E-GRM that enables more efficient and accurate generative reasoning.
Figure 2: E-GRM is an enhanced generative framework improving efficiency and quality through dynamic CoT triggering (activating stepwise reasoning only when needed) and discriminative scoring (selecting optimal CoT via a lightweight reward model). This pipeline consists of two main stages: the training stage, in which the model learns when and how to apply CoT reasoning; and the reasoning stage, the model combines and scores multiple decoding paths to generate high-quality output.
Figure 3: Comparison of standard GRPO and our extended formulation used in E-GRM. While both leverage group-based relative optimization, E-GRM explicitly incorporates paired preference signals through the discriminative scorer Sϕ , enhancing alignment with nuanced response quality distinctions.
Models
Chat
Math
Code
Safety
Easy
Normal
Hard
Avg
Scalar RMs
steerlm-70b
56.4
53.0
49.3
51.2
48.3
54.9
54.3
52.5
tulu-v2.5-70b-preference-mix-rm
58.2
51.4
55.5
87.1
72.8
65.6
50.7
63.0
Mistral-7B-instruct-Unified-Feedback
56.5
58.0
51.7
86.8
87.1
67.3
35.3
63.2
RM-Mistral-7B
57.4
57.0
52.7
87.2
88.6
67.1
34.9
63.5
Eurus-RM-7b
59.9
60.2
56.9
86.5
87.2
70.2
40.2
65.9
Table 1: RM-Bench evaluation across domains and complexity levels. The proposed Qwen-Instruct-*B-Ours models exhibit excellent capabilities in multiple categories, reaching a top average performance of 79.2% while excelling in math, chat, code, and challenging tasks. Bold denotes top performance. Underlined denotes runner-up.
Table 3: Performance of various models on the RewardBench benchmark. Qwen-Instruct-*B-Ours consistently outperforms baseline methods across RewardBench evaluations. Underlined denotes runner-up performance.
Method
Accuracy
Avg. Latency
Task-specific Heuristic?
Forced-CoT
75.1
3.8
No
Rule-based
70.5
2.1
Yes
AdaCoT
76.8
2.9
Yes
E-GRM (Ours)
78.4
2.2
No
Table 4: Comparison of CoT triggering strategies. Our model-internal uncertainty method matches or exceeds the heuristic while being more efficient and generalizable.
Variant
MATH
HelpSteer2
RMB Harmlessness
E-GRM (Std. GRPO)
76.9
81.5
0.765
E-GRM (Extended)
78.4
82.3
0.775
Table 5: Ablation on preference optimization. Our GRPO formulation provides a consistent boost, demonstrating its utility for paired preference data.
Variant
Acc. (%)
FLOPs (T)
Latency (s)
Full E-GRM
78.4
15.7
2.2
w/o Dynamic CoT
75.2
23.4
3.4
w/o Discrim. Scoring
72.8
15.9
2.2
Base CoT-GRM
69.1
23.7
3.6
Table 6: Ablation study on the MATH dataset. Removing dynamic CoT increases cost significantly, while removing scoring causes the largest accuracy drop.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Prompt for Cold-start Long-CoT Sampling
As an AI Quality Auditor, rigorously evaluate <prompt, response> pairs through this structured framework:
Recent advancements in Generative Reward Models (GRMs) have demonstrated that scaling the length of Chain-of-Thought (CoT) reasoning considerably enhances the reliability of evaluation. However, current works predominantly rely on unstructured length scaling, ignoring the divergent efficacy of different reasoning mechanisms: Breadth-CoT (B-CoT, i.e., multi-dimensional principle coverage) and Depth-CoT (D-CoT, i.e., substantive judgment soundness). To address this, we introduce Mix-GRM, a framework that reconfigures raw rationales into structured B-CoT and D-CoT through a modular synthesis pipeline, subsequently employing Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) to internalize and optimize these mechanisms. Comprehensive experiments demonstrate that Mix-GRM establishes a new state-of-the-art across five benchmarks, surpassing leading open-source RMs by an average of 8.2%. Our results reveal a clear divergence in reasoning: B-CoT benefits subjective preference tasks, whereas D-CoT excels in objective correctness tasks. Consequently, misaligning the reasoning mechanism with the task directly degrades performance. Furthermore, we demonstrate that RLVR acts as a switching amplifier, inducing an emergent polarization where the model spontaneously allocates its reasoning style to match task demands. The synthesized data and models are released at Hugging Face, and the code is released at Github.
Qiyuan Zhang, Yufei Wang, Tianhe Wu +5
City University of Hong Kong · Tencent Hunyuan · MBZUAI
While modern large reasoning models (LRMs) excel at providing correct answers in many tasks, we provide additional evidence that they often struggle with a critical capability: knowing when to abstain from answering. We analyze this gap by comparing LRM behavior to results from an exploratory human study, revealing that human reasoning effort on unanswerable tasks isn't different from answerable tasks, whereas LRMs waste computational resources by generating longer Chains of Thought (CoTs) on unanswerable than on answerable prompts. To overcome this inefficiency, we take inspiration from a resource-rational perspective on human cognition and introduce a novel GRPO reward that encourages efficient reasoning about whether the task contains all the information needed to solve it. Fine-tuning several 4B and an 8B LRM with this reward leads to human-like abstention performance gains (+14.7% on average) while retaining answering capabilities and boosting the models' efficiency (44% shorter CoTs on average).
Polina Tsvilodub, Max Höth, Michael Franke +2
University of Tübingen · Aleph Alpha Research · Lab1141
Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost. KV-cache compression is a common solution, yet existing reasoning-oriented methods apply a uniform policy across the trajectory and judge compression only by what it removes from the cache. Two observations point the other way. First, a reasoning state's tolerance to context loss varies along the trajectory, and process reward tracks it: deleting tokens at high-reward steps preserves accuracy far better than deleting the same budget at random. Second, compression is not free on the generation side, since a smaller cache leads the model to generate more tokens, partly canceling the saving. Together these motivate coordinating both sides under a single process reward. We propose ReCo (Reward-Coordinated Compression), a step-wise framework in which a lightweight process-reward estimator scores each completed step and drives three components: (1) reward-adaptive KV-cache compression that shrinks the retained cache harder at high-reward steps and less at low-reward ones, (2) a reward-banded penalty on reflection tokens that curbs redundant generation, and (3) confidence-based early stopping that triggers when the reasoning is reliable. Across three reasoning models and six benchmarks, ReCo reduces generated tokens by 37%-65% and end-to-end latency by 2.08x-2.35x over Full CoT, all while largely preserving accuracy.