What do Reward Models Memorize?

Ivo Verhoeven; Pushkar Mishra; Ekaterina Shutova

cs.LGJul 27, 2026

What do Reward Models Memorize?

Authors: Ivo Verhoeven, Pushkar Mishra, Ekaterina Shutova

Organizations: ILLC, University of Amsterdam, The Netherlands · Google DeepMind, London, United Kingdom

Abstract

This paper studies what discriminatively trained reward models (RMs) memorize by measuring counterfactual memorization on two human preference datasets. We show that RMs 1) misallocate memorization to easy, high margin preference pairs, 2) memorize dataset-specific shortcuts (e.g., model identity, user sampling strategy), and 3) overgeneralize simple heuristic correlates of human preference (e.g., length, compliance) when confronted with unseen preference pairs. Overall, our findings indicate that discriminative training of RMs from human preference data results in biased RMs not yet capable of judging response quality in context-dependent scenarios.

Explore similar work

Discretizing Reward Models
Jun 19, 2026Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika +4Overconfidence Responses
RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
May 3, 2026Yangyang Zhou, Yi-Chen Li
Patterning in Practice: Debiasing Reward Models with Susceptibilities
Sep 1, 2026George Wang, Elizabeth Donoway, Daniel MurfetProcess Reward Modeling Implicit Bias

Jun 19, 2026 · cs.LGJ/K move · Enter open · S save

Discretizing Reward Models

Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika, Chirag Nagpal+3

1Carnegie Mellon University · 2Meta Superintelligence Labs

Despite their widespread use, the role of reward models in shaping reinforcement learning is poorly understood. Reward models offer a tempting promise: they automatically estimate response quality in the absence of verifiers or human judges. Unlike "verifiable rewards" which typically produce binary scores, reward models typically produce continuous scores, allowing them to be sensitive to fine-grained differences in responses. However, we show this apparent strength is a serious weakness: many popular reward models are oversensitive, assigning different scores to equally good responses. Theoretically, we show that seemingly perfect reward models can be highly oversensitive; empirically, this oversensitivity can lead to bad policies. In place of existing notions of "reward model accuracy," we propose evaluating reward models using distinct measures of "discriminative ability" and "specificity" (the complement of oversensitivity). As a solution, we describe a training-free algorithm that uses Monte Carlo dropout on any neural reward model to produce discrete reward clusters. Theoretically, we prove there exist discretizations that reduce oversensitivity at minimal expense of discriminative ability; empirically we show, in both controlled and natural RL settings, that discretizing rewards leads to less reward hacking and better policies than training on the original rewards.

Overconfidence Responses

Open paper Ask about it PDF

Abstract

Explore similar work

Discretizing Reward Models

RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences

Patterning in Practice: Debiasing Reward Models with Susceptibilities