Rater State Bias in RLHF Preference Data: An Audit Framework

Elena Kopteva; Vitaliy Hlynianyi-Zhuk

cs.AIApr 14, 2026

Rater State Bias in RLHF Preference Data: An Audit Framework

Authors: Elena Kopteva, Vitaliy Hlynianyi-Zhuk

Organizations: The Grainger College of Engineering, Department of Physics & Illinois Center for Advanced Studies of the Universe, University of Illinois Urbana-Champaign, Urbana, Illinois 61801, USA · Faculty of Applied Mathematics, Oles Honchar Dnipro National University, Dnipro, 49045, Ukraine · Department of Clinical Psychology, Kyiv Institute of Modern Psychology and Psychotherapy, Kyiv, 01133, Ukraine

Abstract

We identify a structured confound in Reinforcement Learning from Human Feedback (RLHF). Pairwise preference labels are intended to reflect the compared outputs, but they may also reflect the rater's state during annotation. Under sustained stressful or distressing conditions, raters' preferences may shift over time, so that preference data encode rater state alongside judgments about response quality. We argue that, if present, such shifts would differ from ordinary disagreement or random label noise. They would be state dependent, could be shared across annotators under similar conditions, and would not necessarily cancel during aggregation, reward modeling, and policy optimization. We propose rater state shift as a plausible and testable source of structured bias in RLHF preference data. This paper develops a hypothesis and an audit framework for studying this source of bias. We define rater state shift, rater state confound, and correlated rater state bias. We also propose survival level emotional authenticity as a candidate output signature, defined by lexical, pragmatic, discourse, and safety features whose reliability and validity remain to be demonstrated. We show that systematic rater state bias can survive aggregation and may enter the learned reward signal. We state five testable predictions, together with effect size thresholds for an initial audit, and note which require proprietary data. Finally, we present an audit protocol and pilot study plan that can be applied to publicly available instruction tuned models. We do not infer the training history of any specific deployed model.

Figures & tables

$f$	$\delta$	$\rho$	$D_{\mathrm{eff}}$	$N_{\mathrm{eff}}$	$f\cdot\delta$
0.10	0.10	0.00	1.0	1000	0.010
0.10	0.10	0.10	20.9	48	0.010
0.20	0.15	0.00	1.0	1000	0.030
0.20	0.15	0.10	20.9	48	0.030
0.20	0.15	0.20	40.8	25	0.030

Table 1: Illustrative estimates of rater state bias entry.

f

: fraction of annotations made under shifted conditions.

\delta

: preference probability shift under those conditions.

\rho

: intraclass correlation.

D_{\mathrm{eff}}

: design effect (

n=200

N_{\mathrm{eff}}

: effective independent sample size (

N=1{,}000

). The mean shift

f\delta

is invariant to

\rho

; the effective sample size is not.

Stage	Mechanism	Effect on bias
Preference collection	Rater state shift (Eq. 5 )	The aggregate preference signal shifts by $f\delta(x)$ . Positive correlation reduces $N_{\mathrm{eff}}$ , increasing uncertainty in estimates of the shift.
Reward model training	Bradley-Terry fitting (Eq. 7 )	A structured shift in preference probabilities changes the fitted reward differences between responses.
Policy optimization	Reward weighting with a KL penalty (Eq. 9 )	An absorbed reward shift changes relative response odds by $\exp(\Delta r/\lambda)$ .

Table 2: Propagation of rater state bias through RLHF. Each stage can preserve or amplify the shift introduced at the previous stage.

Explore similar work

Three Models of RLHF Annotation: Extension, Evidence, and Authority
Apr 28, 2026Steve CoyneReinforcement Learning From Human Feedback Human Annotators
University of Toronto, Canada · University of Toronto, Toronto, Canada
Mitigating Cognitive Bias in RLHF by Altering Rationality
May 7, 2026Tiffany Horter, Andrew Markham, Niki Trigoni +1Reinforcement Learning From Human Feedback Rationality
University of Oxford · Brown University
Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
May 26, 2026Dongyoon Hahm, Dylan Hadfield-Menell, Kimin LeeLarge Language Model Alignment Reinforcement Learning From Human Feedback
1KAIST · 2MIT

Apr 28, 2026, cs.CYJ/K move · Enter open · S save

Three Models of RLHF Annotation: Extension, Evidence, and Authority

Steve Coyne

University of Toronto, Canada · University of Toronto, Toronto, Canada

Reinforcement Learning From Human Feedback Human Annotators Large Language Model Alignment Annotations Social Norms

Open paper AskPDF arXiv

Preference-based alignment methods, most prominently Reinforcement Learning with Human Feedback (RLHF), use the judgments of human annotators to shape large language model behaviour. However, the normative role of these judgments is rarely made explicit. I distinguish three conceptual models of that role. The first is extension: annotators extend the system designers' own judgments about what outputs should be. The second is evidence: annotators provide independent evidence about some facts, whether moral, social or otherwise. The third is authority: annotators have some independent authority (as representatives of the broader population) to determine system outputs. I argue that these models have implications for how RLHF pipelines should solicit, validate and aggregate annotations. I survey landmark papers in the literature on RLHF and related methods to illustrate how they implicitly draw on these models, describe failure modes that come from unintentionally or intentionally conflating them, and offer normative criteria for choosing among them. My central recommendation is that RLHF pipeline designers should decompose annotation into separable dimensions and tailor each pipeline to the model most appropriate for that dimension, rather than seeking a single unified pipeline.

Abstract

Explore similar work

Three Models of RLHF Annotation: Extension, Evidence, and Authority

Mitigating Cognitive Bias in RLHF by Altering Rationality

Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases