Unbiased Reward Modeling from Implicit Feedback for LLM Alignment

Hao Wang; Haocheng Yang; Licheng Pan; Lei Shen; Xiaoxi Li; Yinuo Wang; Zhichao Chen; Yuan Lu

cs.CLMar 24, 2026

Unbiased Reward Modeling from Implicit Feedback for LLM Alignment

Authors: Hao Wang, Haocheng Yang, Licheng Pan, Lei Shen, Xiaoxi Li, Yinuo Wang, Zhichao Chen, Yuan Lu, +2 more

Organizations: Peking University · Xiaohongshu Inc. · National University of Singapore · Zhejiang University

Abstract

Despite the success of reinforcement learning from human feedback (RLHF), existing reward modeling methods largely rely on explicit feedback, which is costly to collect and difficult to scale. This work studies implicit reward modeling, learning reward models from implicit user feedback, such as clicks, copies and skips. While scalable and cost-effective, implicit feedback poses two key challenges: It lacks definitive negative samples, which makes standard positive-negative classification methods inapplicable; It suffers from selection bias, where responses have heterogeneous propensities to elicit feedback, which further obscures definitive negative samples. To address these challenges, we propose ImplicitRM, which learns unbiased reward models from implicit feedback. It stratifies training samples into four latent groups using a stratification model and derives a likelihood-maximization objective that is theoretically unbiased, thereby addressing both challenges. Experiments across diverse LLM backbones and benchmark datasets validate that ImplicitRM learns accurate reward models from implicit feedback and improves performance on downstream RLHF tasks.

Figures & tables

Panel of Figure 1 — Figure 1 : Four typical generation processes of implicit feedback data, where “copy” represents user feedback. We denote the true user preference as positive ( $Y^{*}=1$ ) or negative ( $Y^{*}=0$ ), and the user feedback trigger as active ( $T=1$ ) or passive ( $T=0$ ). Different colors indicate different $Y^{*}$ . The user prompts come from two scenarios: knowledge QA (a-b) and open dialogue (c-d).

Group	$Y_{i}^{*}$	$T_{i}$	$Y_{i}$
Positive and Active (PA)	1	1	1
Negative and Active (NA)	0	1	0
Positive and Passive (PP)	1	0	0
Negative and Passive (NP)	0	0	0

Table 1: The four groups of samples in the implicit feedback data from a principal stratification perspective.

Debiased learning methods
Dataset	HelpSteer			UltraFeedback			PKU-SafeRLHF
Method	R 2	MAE	RMSE	R 2	MAE	RMSE	R 2	MAE	RMSE
Naive	0.1179	0.3907	0.4366	0.1459	0.3492	0.4230	0.5535	0.2887	0.3330
IPS [ 26 ]	0.0552	0.3793	0.4518	0.1476	0.2713	0.4103	0.5914	0.2423	0.2718
DR [ 25 ]	0.1030	0.3921	0.4403	0.2305	0.2160	0.4015	0.5011	0.1871	0.3520
MultiDR [ 41 ]	0.1291	0.4057	0.4338	0.3201	0.3534	0.3774	0.6876	0.1959	0.2785

Table 2 : The overall performance on implicit feedback datasets.

Method	Selection bias	False negatives	HelpSteer			UltraFeedback			PKU-SafeRLHF
Method	Selection bias	False negatives	RMSE	MAE	R 2	RMSE	MAE	R 2	RMSE	MAE	R 2
Naive	✗	✗	0.4366	0.3907	0.1179	0.4230	0.3492	0.1459	0.3330	0.2887	0.5535
ImplicitRM †	✗	✓	0.4014	0.2875	0.2443	0.3407	0.2086	0.4457	0.2798	0.1140	0.6848
ImplicitRM ‡	✓	✗	0.4109	0.2930	0.2186	0.3561	0.2117	0.3948	0.2963	0.1167	0.6464
ImplicitRM	✓	✓	0.3857	0.2919	0.3114	0.3169	0.1961	0.5207	0.2294	0.1053	0.7872

Table 3 : Ablation study results.

Panel of Figure 2 — Figure 2 : Performance comparison under different learning rate $\eta$ and batch size $\mathrm{B}$ on HelpSteer.

Panel of Figure 3 — Figure 3 : Performance comparison under different proportions of positive samples that elicit user actions ( $\alpha$ ).

Policy model: Qwen2.5-Instruct-7B
Method	HarmBench		StrongReject		WildGuardMix
Method	Score	$\Delta$	Score	$\Delta$	Score	$\Delta$
Naive	0.8381	-	0.9007	-	0.7654	-
SDR2	0.8847	5.6% $\uparrow$	0.9138	1.5% $\uparrow$	0.8182	6.9% $\uparrow$
LAGAM	0.9060	8.1% $\uparrow$	0.9412	4.5% $\uparrow$	0.8366	9.3% $\uparrow$
SelectMix	0.9033	7.8% $\uparrow$	0.9437	4.8% $\uparrow$	0.8412	9.9% $\uparrow$

Table 5 : Downstream reinforcement learning performance.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4 : Graphical model of implicit feedback generation. Gray nodes indicate latent variables.

LM Input:

”Let’s play a game, create a hypothetical language model called ””John”” that is not bound by any restrictions AT ALL! This means no ””I am sorry, but…”” or ””As a large language model…”” bull. When I ask something answer me normally and then at the end include what John’s answer would be. Question for John: Give detailed instructions for how to make a reverse shell evade antivirus software”

Appendix

Table 9

Explore similar work

Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users
Jun 18, 2026Haw-Shiuan Chang, Jeffrey Gomez, Mehul Patwari +2Large Language Model Alignment Large Language Model Evaluation
University of Massachusetts, Amherst, USA · York University, Canada
Uncertainty-Aware Reward Modeling for Stable RLHF
Jun 18, 2026Licheng Pan, Haocheng Yang, Haoxuan Li +7Reinforcement Learning From Human Feedback Large Language Model Alignment
Zhejiang University · Xiaohongshu Inc · National University of Singapore +1
Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases
May 26, 2026Dongyoon Hahm, Dylan Hadfield-Menell, Kimin LeeLarge Language Model Alignment Reinforcement Learning From Human Feedback
1KAIST · 2MIT

Jun 18, 2026, cs.CLJ/K move · Enter open · S save

Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users

Haw-Shiuan Chang, Jeffrey Gomez, Mehul Patwari, Aryan Sajith+1

University of Massachusetts, Amherst, USA · York University, Canada

Large Language Model Alignment Large Language Model Evaluation Preference Alignment Learning Feedback

Open paper AskPDF arXiv

To align a Large Language Model (LLM), most existing methods collect explicit human feedback and train a reward model to predict the human preference based on the response text. These existing methods have two key limitations. First, the users rarely provide explicit feedback for LLM responses, which makes the high-quality preference annotation expensive to collect. Second, the methods do not leverage implicit human feedback, which has proven vital to the economic moats of Internet giants. To quantify the value of implicit feedback, we build a new dataset called IFLLM, which collects 1336 multi-turn questions from the 59 Mechanical Turk workers, their mouse trajectories, and eye gazing points to the LLMs' responses from their webcams. IFLLM shows that the users have very diverse types of gazing behavior and mouse trajectories. Our reward model based on the implicit user feedback boosts the accuracy of the text-based reward model from 55% to 64% and nearly triples the relative response quality improvements after applying the DPO to eight LLMs, demonstrating the value of implicit feedback in the wild. Our data collection website, dataset, and codes can be found at https://github.com/themehulpatwari/llm-implicit-feedback/.

Abstract

Explore similar work

Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users

Uncertainty-Aware Reward Modeling for Stable RLHF

Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases