Inference-time compute scaling has emerged as a powerful paradigm for improving language model performance on a wide range of tasks, but the question of how best to use the additional compute remains open. A popular approach is BoN sampling, where N candidate responses are generated, scored according to a reward model, and the highest-scoring response is selected. While this approach can improve performance, it is vulnerable to reward hacking, where performance degrades as N increases due to the selection of responses that exploit imperfections in the reward model instead of genuinely improving generation quality. Prior attempts to mitigate reward hacking, via stronger reward models or heavy-handed distributional regularization, either fail to fully address over-optimization or are too conservative to exploit additional compute. In this work, we explore the principle of pessimism in RL, which uses lower confidence bounds on value estimates to avoid OOD actions with uncertain reward estimates. Our approach, termed as caution, can be seen as the reverse of curiosity: where curiosity rewards prediction error as a signal of novelty, caution penalizes prediction error as a signal of distributional uncertainty. Practically, caution trains an error model on typical responses and uses its prediction error to lower reward estimates for atypical ones. Our extensive empirical evaluation demonstrates that caution is a simple, computationally efficient approach that substantially mitigates reward hacking in BoN sampling. We also provide a theoretical analysis in a simplified linear setting, which shows that caution provably improves over the standard BoN approach. Together, our results not only establish caution as a practical solution to reward hacking, but also provide evidence that curiosity-based approaches can be a general OOD detection technique in LLM settings.
Figures & tables
Figure 1: Average Accuracy with different sampling budgets for Best-of- N on the GSM8k dataset. We see that standard Best-of- N sampling (blue, red, and gold) suffers from reward hacking, exhibiting the characteristic rise-and-fall pattern as N increases. In contrast, caution (our approach, green) consistently improves with larger N , effectively mitigating reward hacking.
Figure 2: Overview. Predictor is trained to match RM features on typical responses; at inference, we select the candidate with the highest pessimistic reward, down - weighting OOD ones.
Figure 3: Scaling over N across distributions and domains. Best-of- N sampling on GSM8K, MATH-500, and BigBench-Hard. Curves compare selection by Reward Model, Pessimism, and Reward Model + Pessimism. Note that the pessimism function is trained only on GSM8K ; thus, MATH-500 represents an out-of-distribution setting, while BigBench-Hard represents a fully out-of-domain setting.
Method
GSM8K ( 2021 )
MATH-500 ( 2023 )
BBH-Hard ( 2022 )
Peak
Final
Degr.
Peak
Final
Degr.
Peak
Final
Degr.
Reward Model
79.3 ±1.2
71.5 ±0.3
7.7
11.5 ±0.3
8.5 ±1.0
3.0
17.3 ±1.0
1.7 ±0.1
15.6
Pessimism Only
81.3 ±0.5
80.3 ±0.3
1.0
13.6 ±0.7
11.9 ±1.0
1.7
22.9 ±0.7
22.1 ±0.4
0.9
RM + Pessimism
82.6 ±0.6
81.1 ±0.1
1.5
12.3 ±1.0
10.1 ±0.6
2.3
18.5 ±1.0
11.0 ±0.3
7.5
Table 1: Scaling Performance Across Datasets with Reward Hacking Mitigation. Peak Acc : highest accuracy achieved across all N values. Final Acc : accuracy at N=512 where reward hacking is most severe. Degradation : performance drop from peak to final (lower is better). Results show mean with 95% confidence intervals as subscripts across 3 bootstrap runs. Best results are in bold and second-best are underlined .
Pessimism strength ( λ )
Caution (Ours)
Traditional RND
Peak Acc
Final Acc
Peak Acc
Final Acc
0.0 (RM Only)
78.9
71.2
78.9
71.2
0.2
79.5
72.5
78.7
71.3
0.4
80.3
76.0
78.5
72.1
0.6
81.5
80.2
78.4
73.1
0.8
82.1
81.0
78.1
74.9
Table 2: Weight Ablation and Design Comparison on GSM8K. We compare different mixing weights between uncertainty score and r^ score, contrasting our approach (distilling reward model features) against traditional RND (distilling random network features). Peak Acc : best accuracy across all N . Final Acc : accuracy at N=512 . Results demonstrate that distilling reward model representations substantially outperforms random network distillation across weight combinations.
Predictor Configuration
Reconstruction Loss
Peak Acc (%)
Final Acc (%)
Simplified Architecture Variants
Lightweight + Trainable Emb.
0.242
82.2
82.2
Lightweight + Frozen Emb.
0.246
81.5
81.3
Lightweight + Separate Emb.
0.255
82.7
81.8
Lightweight w/o Projection
0.281
81.8
81.3
Full Architecture Variants
Table 3: Ablation study of predictor architecture. Training Loss measures how well the predictor fits reward model representations. Peak Acc shows best performance across all N . Final Acc shows performance at largest N , where reward hacking is most severe.
Figure 4: Contrasting Selection Behaviors: Reward Hacking vs. Format Compliance. Two representative examples showing how reward models favor verbose responses regardless of correctness, while our curiosity-driven pessimism prioritizes format compliance and distributional familiarity. RM assigns high scores to detailed responses regardless of correctness, while pessimism detects distributional deviation from training patterns and prefers correctly formatted solutions.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Parameter
Value
Description
RND Architecture
Target layers
10
Number of layers extracted from reward model for target network
Predictor layers
10
Number of layers in predictor network architecture
RND weight ( λ )
0.8
Strength of pessimism penalty in combined scoring
Exact architecture
False
Whether predictor copies exact reward model architecture
Embedding strategy
shared_trainable
How embeddings are handled: shared_trainable, shared_frozen, or separate
Use projection
True
Whether to add projection layer between predictor and target
Appendix
Table 4: Comprehensive Hyperparameter Configuration. All hyperparameters used in training and evaluation of the Random Network Distillation (RND) approach for mitigating reward hacking. The table is organized by component: RND architecture, training process, inference settings, and evaluation configurations.
Ablation Setting
What it means / Rationale
Predictor architecture
Same as Target
Predictor uses the same overall structure as the target network (e.g., same block type and connectivity), matching width and depth. Isolates training dynamics from architectural mismatch.
Simplified
Predictor keeps the same hidden size and number of layers as the target but replaces specialized target blocks with vanilla Transformer encoder blocks . This deliberately reduces architectural complexity while preserving depth/width, aiming for better generalization and lower overfitting risk.
Embedding strategy
Shared, trainable
Predictor shares the target’s token embeddings and updates them during training . Pros: reuse target’s pretrained semantic representations and potentially quicker convergence. Cons: tighter coupling may leak target-specific biases into the predictor.
Shared, frozen
Predictor reuses the target’s token embeddings but keeps them frozen . Pros: stable token mapping and clean isolation of predictor encoder learning. Cons: less flexibility to adapt embeddings to the predictor’s simplified blocks.
Appendix
Table 5: Summary of ablations. Each row defines one setting and what it means in practice.
Figure 5: Pessimism–Reward visualization on GSM8K. Each row shows one problem: a scatter plot of z-normalized pessimism (x-axis) and z-normalized reward (y-axis), with green points for correct responses and red for incorrect. Upper-left points (high reward, low pessimism) illustrate reward hacking—responses that score well despite low distributional support. Lower-right points (low reward, high pessimism) are well-formed, instruction-following responses that the reward model undervalues; our caution mechanism up-weights these relative to reward-only selection.
L
Peak Acc.
Final Acc.
2
79
77
4
81
77
6
83
78
8
84
81
10
84
84
Appendix
Table 6: Layer depth ablation for target network and predictor network.
Figure 6: Caution (RND-on-RM-features) scaling with λ . Best-of- N accuracy on GSM8K versus samples per problem (x-axis) for Pessimism strengths λ∈{0.0,0.2,0.4,0.6,0.8,1.0} . The predictor is trained against a frozen target network built from reward-model features. Larger λ increases pessimism strength; λ=0 reduces to Reward-Model-only selection, and λ=1 to pessimism-only. Moderate–high weights (roughly 0.6 – 0.8 ) preserve scaling while curbing reward hacking, outperforming both the RM-only and RND-only extremes across most N .
Figure 7: Traditional RND (random targets) baseline. Best-of- N GSM8K accuracy when using classical RND with a randomly initialized target network (no reward - model features), sweeping λ∈{0.0,0.2,0.4,0.6,0.8,1.0} . Unlike our caution variant, this baseline shows little to no scaling benefit and generally does not surpass the Reward - Model - only curve, indicating that semantic grounding from reward - model features is crucial for effective distributional regularization.
Figure 8: Comparison of pessimism-based sampling with alternative baseline methods.
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.