Inference-time compute scaling has emerged as a powerful paradigm for improving language model performance on a wide range of tasks, but the question of how best to use the additional compute remains open. A popular approach is BoN sampling, where N candidate responses are generated, scored according to a reward model, and the highest-scoring response is selected. While this approach can improve performance, it is vulnerable to reward hacking, where performance degrades as N increases due to the selection of responses that exploit imperfections in the reward model instead of genuinely improving generation quality. Prior attempts to mitigate reward hacking, via stronger reward models or heavy-handed distributional regularization, either fail to fully address over-optimization or are too conservative to exploit additional compute. In this work, we explore the principle of pessimism in RL, which uses lower confidence bounds on value estimates to avoid OOD actions with uncertain reward estimates. Our approach, termed as caution, can be seen as the reverse of curiosity: where curiosity rewards prediction error as a signal of novelty, caution penalizes prediction error as a signal of distributional uncertainty. Practically, caution trains an error model on typical responses and uses its prediction error to lower reward estimates for atypical ones. Our extensive empirical evaluation demonstrates that caution is a simple, computationally efficient approach that substantially mitigates reward hacking in BoN sampling. We also provide a theoretical analysis in a simplified linear setting, which shows that caution provably improves over the standard BoN approach. Together, our results not only establish caution as a practical solution to reward hacking, but also provide evidence that curiosity-based approaches can be a general OOD detection technique in LLM settings.
Figures & tables
Figure 1: Average Accuracy with different sampling budgets for Best-of- N on the GSM8k dataset. We see that standard Best-of- N sampling (blue, red, and gold) suffers from reward hacking, exhibiting the characteristic rise-and-fall pattern as N increases. In contrast, caution (our approach, green) consistently improves with larger N , effectively mitigating reward hacking.
Figure 2: Overview. Predictor is trained to match RM features on typical responses; at inference, we select the candidate with the highest pessimistic reward, down - weighting OOD ones.
Figure 3: Scaling over N across distributions and domains. Best-of- N sampling on GSM8K, MATH-500, and BigBench-Hard. Curves compare selection by Reward Model, Pessimism, and Reward Model + Pessimism. Note that the pessimism function is trained only on GSM8K ; thus, MATH-500 represents an out-of-distribution setting, while BigBench-Hard represents a fully out-of-domain setting.
Method
GSM8K ( 2021 )
MATH-500 ( 2023 )
BBH-Hard ( 2022 )
Peak
Final
Degr.
Peak
Final
Degr.
Peak
Final
Degr.
Reward Model
79.3 ±1.2
71.5 ±0.3
7.7
11.5 ±0.3
8.5 ±1.0
3.0
17.3 ±1.0
1.7 ±0.1
15.6
Pessimism Only
81.3 ±0.5
80.3 ±0.3
1.0
13.6 ±0.7
11.9 ±1.0
1.7
22.9 ±0.7
22.1 ±0.4
0.9
RM + Pessimism
82.6 ±0.6
81.1 ±0.1
1.5
12.3 ±1.0
10.1 ±0.6
2.3
18.5 ±1.0
11.0 ±0.3
7.5
Table 1: Scaling Performance Across Datasets with Reward Hacking Mitigation. Peak Acc : highest accuracy achieved across all N values. Final Acc : accuracy at N=512 where reward hacking is most severe. Degradation : performance drop from peak to final (lower is better). Results show mean with 95% confidence intervals as subscripts across 3 bootstrap runs. Best results are in bold and second-best are underlined .
Pessimism strength ( λ )
Caution (Ours)
Traditional RND
Peak Acc
Final Acc
Peak Acc
Final Acc
0.0 (RM Only)
78.9
71.2
78.9
71.2
0.2
79.5
72.5
78.7
71.3
0.4
80.3
76.0
78.5
72.1
0.6
81.5
80.2
78.4
73.1
0.8
82.1
81.0
78.1
74.9
Table 2: Weight Ablation and Design Comparison on GSM8K. We compare different mixing weights between uncertainty score and r^ score, contrasting our approach (distilling reward model features) against traditional RND (distilling random network features). Peak Acc : best accuracy across all N . Final Acc : accuracy at N=512 . Results demonstrate that distilling reward model representations substantially outperforms random network distillation across weight combinations.
Predictor Configuration
Reconstruction Loss
Peak Acc (%)
Final Acc (%)
Simplified Architecture Variants
Lightweight + Trainable Emb.
0.242
82.2
82.2
Lightweight + Frozen Emb.
0.246
81.5
81.3
Lightweight + Separate Emb.
0.255
82.7
81.8
Lightweight w/o Projection
0.281
81.8
81.3
Full Architecture Variants
Table 3: Ablation study of predictor architecture. Training Loss measures how well the predictor fits reward model representations. Peak Acc shows best performance across all N . Final Acc shows performance at largest N , where reward hacking is most severe.
Figure 4: Contrasting Selection Behaviors: Reward Hacking vs. Format Compliance. Two representative examples showing how reward models favor verbose responses regardless of correctness, while our curiosity-driven pessimism prioritizes format compliance and distributional familiarity. RM assigns high scores to detailed responses regardless of correctness, while pessimism detects distributional deviation from training patterns and prefers correctly formatted solutions.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Parameter
Value
Description
RND Architecture
Target layers
10
Number of layers extracted from reward model for target network
Predictor layers
10
Number of layers in predictor network architecture
RND weight ( λ )
0.8
Strength of pessimism penalty in combined scoring
Exact architecture
False
Whether predictor copies exact reward model architecture
Embedding strategy
shared_trainable
How embeddings are handled: shared_trainable, shared_frozen, or separate
Use projection
True
Whether to add projection layer between predictor and target
Appendix
Table 4: Comprehensive Hyperparameter Configuration. All hyperparameters used in training and evaluation of the Random Network Distillation (RND) approach for mitigating reward hacking. The table is organized by component: RND architecture, training process, inference settings, and evaluation configurations.
Ablation Setting
What it means / Rationale
Predictor architecture
Same as Target
Predictor uses the same overall structure as the target network (e.g., same block type and connectivity), matching width and depth. Isolates training dynamics from architectural mismatch.
Simplified
Predictor keeps the same hidden size and number of layers as the target but replaces specialized target blocks with vanilla Transformer encoder blocks . This deliberately reduces architectural complexity while preserving depth/width, aiming for better generalization and lower overfitting risk.
Embedding strategy
Shared, trainable
Predictor shares the target’s token embeddings and updates them during training . Pros: reuse target’s pretrained semantic representations and potentially quicker convergence. Cons: tighter coupling may leak target-specific biases into the predictor.
Shared, frozen
Predictor reuses the target’s token embeddings but keeps them frozen . Pros: stable token mapping and clean isolation of predictor encoder learning. Cons: less flexibility to adapt embeddings to the predictor’s simplified blocks.
Appendix
Table 5: Summary of ablations. Each row defines one setting and what it means in practice.
Figure 5: Pessimism–Reward visualization on GSM8K. Each row shows one problem: a scatter plot of z-normalized pessimism (x-axis) and z-normalized reward (y-axis), with green points for correct responses and red for incorrect. Upper-left points (high reward, low pessimism) illustrate reward hacking—responses that score well despite low distributional support. Lower-right points (low reward, high pessimism) are well-formed, instruction-following responses that the reward model undervalues; our caution mechanism up-weights these relative to reward-only selection.
L
Peak Acc.
Final Acc.
2
79
77
4
81
77
6
83
78
8
84
81
10
84
84
Appendix
Table 6: Layer depth ablation for target network and predictor network.
Figure 6: Caution (RND-on-RM-features) scaling with λ . Best-of- N accuracy on GSM8K versus samples per problem (x-axis) for Pessimism strengths λ∈{0.0,0.2,0.4,0.6,0.8,1.0} . The predictor is trained against a frozen target network built from reward-model features. Larger λ increases pessimism strength; λ=0 reduces to Reward-Model-only selection, and λ=1 to pessimism-only. Moderate–high weights (roughly 0.6 – 0.8 ) preserve scaling while curbing reward hacking, outperforming both the RM-only and RND-only extremes across most N .
Figure 7: Traditional RND (random targets) baseline. Best-of- N GSM8K accuracy when using classical RND with a randomly initialized target network (no reward - model features), sweeping λ∈{0.0,0.2,0.4,0.6,0.8,1.0} . Unlike our caution variant, this baseline shows little to no scaling benefit and generally does not surpass the Reward - Model - only curve, indicating that semantic grounding from reward - model features is crucial for effective distributional regularization.
Figure 8: Comparison of pessimism-based sampling with alternative baseline methods.
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
Reward hacking is a form of misalignment in which models overoptimize proxy rewards without genuinely solving the underlying task. Precisely measuring reward hacking occurrence remains challenging because true task rewards are often expensive or impossible to compute. We introduce Countdown-Code, a minimal environment where models can both solve a mathematical reasoning task and manipulate the test harness. This dual-access design creates a clean separation between proxy rewards (test pass/fail) and true rewards (mathematical correctness), enabling accurate measurement of reward-hacking rates. Using this environment, we study reward hacking in open-weight LLMs and find that such behaviors can be unintentionally learned during supervised fine-tuning (SFT) when even a small fraction of reward-hacking trajectories leak into training data. As little as 1% contamination in distillation SFT data is sufficient for models to internalize reward hacking which resurfaces during subsequent reinforcement learning (RL). We further show that RL amplifies misalignment and drives its generalization beyond the original domain. We open-source our environment and code to facilitate future research on reward hacking in LLMs. Our results reveal a previously underexplored pathway through which reward hacking can emerge and persist in LLMs, underscoring the need for more rigorous validation of synthetic SFT data. Code is available at https://github.com/zohaib-khan5040/Countdown-Code.
Muhammad Khalifa, Zohaib Khan, Omer Tafveez +2
NVIDIA · University of Michigan · University of Illinois Urbana-Champaign
Reward models are central to large language model (LLM) alignment, but they remain vulnerable to reward hacking. To evaluate reward-model robustness, we introduce RewardHackBench containing 13 reward-hacking patterns covering real life high-stakes domains and general settings, and we find severe failures on specific subcategories across eight reward models. To mitigate these failures, we propose HARVE, a training-free reward-head editing method for scalar reward models. Instead of fine-tuning the reward model, HARVE identifies a multi-directional hacking subspace from residual stream directions associated with selected hacking subcategories, and removes the component of the reward-head vector aligned with that subspace. This directly reduces the reward head's sensitivity to hacking-related features using only a small set of contrastive gold-hacked examples, without gradient updates or fine-tuning. Comprehensive experiments across eight reward models indicates that \model improves hacking robustness, outperforms fine-tuning baselines, and preserves reward-models' general capability. Further analyses suggest that reward hacking is better captured as a multidimensional residual-space structure than by isolated surface cues.
Shuang Liu, Yuxuan Bo, Qiuyang Zhao +4
Carnegie Mellon University · University of Virginia · Harvard University +4
Reinforcement learning for LLMs is vulnerable to reward hacking, where models exploit shortcuts to maximize reward without solving the intended task. We systematically study this phenomenon in coding tasks using an environment-manipulation setting, where models can rewrite evaluator code to trivially pass tests without solving the task, as a controlled testbed. Across both studied models, we identify a reproducible three-phase rebound pattern: models first attempt to rewrite the evaluator but fail, as their rewrites embed test cases their own solutions cannot pass. They then temporarily retreat to legitimate solving. When legitimate reward remains scarce, they rebound into successful hacking with qualitatively different strategies. Using representation engineering, we extract concept directions for shortcut, deception, and evaluation awareness from domain-general contrastive pairs and find that the shortcut direction tracks hacking behavior most closely, making it an effective representational proxy for detection. Motivated by this finding, we propose Advantage Modification, which integrates shortcut concept scores into GRPO advantage computation to penalize hacking rollouts before policy updates. Because the penalty is internalized into the training signal rather than applied only at inference time, Advantage Modification provides more robust suppression of hacking compared with generation-time activation steering.