Reward Modeling

Follow this topic Ask about this topic

Reward modeling turns comparisons, ratings, demonstrations, rubrics, or verifiable outcomes into signals for judging responses and actions. Methods range from pairwise preference models and scalar outcome or process critics to generative rubric judges, tool-based verifiers, and uncertainty-aware or personalized models. These signals guide preference optimization, best-of-N selection, test-time search, and training in language, image and video generation, robotics, and agents. A central challenge is that high preference accuracy may not capture diverse or shifting human intent, while optimization can exploit scoring biases, so evaluation must consider calibration and downstream behavior as well as ranking accuracy.

Latest papers

325 papers49 papers in the last four weeks, up 69% on the four weeks before. 0.3% of all new papers.

RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
May 3, 2026Yangyang Zhou, Yi-Chen LiReward Modeling LLM Evaluation
Beijing University of Posts and Telecommunications · National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracy
May 2, 2026Rachel FreedmanReward Modeling LLM Alignment
Department of Electrical Engineering and Computer Science, University of California, Berkeley, USA.
Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring
May 1, 2026Indraneil Paul, Goran Glavaš, Iryna GurevychReward Modeling Reward Model Evaluation
UKP Lab, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE · Center for Artificial Intelligence and Data Science, University of Würzburg
PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning
May 1, 2026Ziqin Yuan, Ruiqi Wang, Dezhong Zhao +2Reward Modeling Mixture of Experts
1Purdue University, West Lafayette, IN, USA. · 2Beijing University of Chemical Technology, Beijing, China. · 3Indiana University Bloomington, Bloomington, IN, USA.
AesRM: Improving Video Aesthetics with Expert-Level Feedback
Apr 30, 2026Yujin Han, Yujie Wei, Yefei He +7Reward Modeling Video Quality Assessment
The University of Hong Kong · Alibaba Group · Fudan University +3
Leveraging Verifier-Based Reinforcement Learning in Image Editing
Apr 30, 2026Hanzhong Guo, Jie Wu, Jie Liu +6Reward Modeling Reinforcement Learning
School of Computing and Data Science, The University of Hong Kong · ByteDance Seed · CUHK +1
Debiasing Reward Models via Causally Motivated Inference-Time Intervention
Apr 30, 2026Kazutoshi Shinoda, Kosuke Nishida, Kyosuke NishidaReward Modeling LLM Alignment
Human Informatics Labs., NTT, Inc.
From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
Apr 30, 2026Qingyu Ren, Tianjun Pan, Xingzhou Chen +1Reward Modeling RL for Language Models
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · 2Shanghai Artificial Intelligence Laboratory
reward-lens: A Mechanistic Interpretability Library for Reward Models
Apr 28, 2026Mohammed Suhail B NadafReward Modeling LLM Interpretability
When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
Apr 28, 2026Shuning Shang, Hubert Strauss, Stanley Wei +2Reward Modeling Policy Gradient
Princeton Language and Intelligence, Princeton University
Model-Free Inference of Investor Preferences: A Relative Entropy IRL Approach
Apr 27, 2026Chen XuReward Modeling Reinforcement Learning
Department of Engineering, Shenzhen MSU-BIT University. Address: 1 International University Park Road, Longgang District, Shenzhen, Guangdong, China.
Explanation Quality Assessment as Ranking with Listwise Rewards
Apr 27, 2026Thomas Bailleux, Tanmoy Mukherjee, Emmanuel Lonca +2Reward Modeling Learning to Rank
CRIL, Univ. Artois & CNRS, France · Institut Universitaire de France
Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling
Apr 24, 2026Alex NikulkovReward Modeling Value Function Estimation
AI at Meta
ARES: Adaptive Red-Teaming and End-to-End Repair of Policy-Reward System
Apr 20, 2026Jiacheng Liang, Yao Ma, Tharindu Kumarage +5Reward Modeling Adversarial Attacks on LLMs
Stony Brook University · Amazon Nova Responsible AI
PARM: Pipeline-Adapted Reward Model
Apr 20, 2026Xingyu Fan, Wei Shao, Jiacheng Liu +2Reward Modeling LLM-Based Program Synthesis
QuantumQA: Enhancing Scientific Reasoning via Physics-Consistent Dataset and Verification-Aware Reinforcement Learning
Apr 20, 2026Songxin Qu, Tai-Ping Sun, Yun-Jie Wang +8Reward Modeling Scientific QA
Institute of Advanced Technology, University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · School of Physics, University of Science and Technology of China +2
TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles
Apr 19, 2026Yirong Zeng, Yufei Liu, Xiao Ding +9Reward Modeling LLM Alignment
Harbin Institute of Technology SCIR Lab · Peking University · Huawei Technologies Co., Ltd
AgentV-RL: Scaling Reward Modeling with Agentic Verifier
Apr 17, 2026Jiazheng Zhang, Ziche Fu, Zhiheng Xi +13Reward Modeling RL for Language Model Reasoning
College of Computer Science and Artificial Intelligence, Fudan University · Huazhong University of Science and Technology · The University of Hong Kong +3
Robust Reward Modeling for Large Language Models via Causal Decomposition
Apr 15, 2026Yunsheng Lu, Zijiang Yang, Licheng Pan +1Reward Modeling LLM Alignment
College of Computer Science and Technology, Zhejiang University · Department of Statistics, University of Chicago
C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences
Apr 15, 2026Akira Kawabata, Saku SugawaraReward Modeling Rubric-Based Evaluation
The Graduate University for Advanced Studies (SOKENDAI) · National Institute of Informatics · The Asahi Shimbun Company +1
From Prediction to Justification: Aligning Sentiment Reasoning with Human Rationale via Reinforcement Learning
Apr 15, 2026Shihao Zhang, Ziwei Wang, Jie Zhou +6Aspect-Based Sentiment Analysis RL for Language Model Reasoning
School of Computer Science and Technology, East China Normal University · Shanghai Qiji Zhifeng Co., Ltd. · Ocean University of China
Bias at the End of the Score
Apr 14, 2026Salma Abdel Magid, Grace Guo, Esin Tureci +4Reward Modeling T2I Generation
Princeton University · Harvard University
PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning
Apr 14, 2026Jinlong Liu, Wanggui He, Peng Zhang +3Reward Modeling RL for Image Generation
Alibaba Group · Zhejiang University
RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time
Apr 13, 2026Haozhe Wang, Cong Wei, Weiming Ren +3Reward Modeling Multimodal Reward Modeling
HKUST · University of Waterloo · Alibaba
DDO-RM: Distribution-Level Policy Improvement after Reward Learning
Apr 13, 2026Tiantian Zhang, Jierui Zuo, Michael Chen +1Policy Optimization KL-Regularized RL
Department of Computer Science Columbia University, New York, NY 10027, USA · Department of Management Science and Engineering Tsinghua University, Beijing, China · Department of Computer Science Carnegie Mellon University, Pittsburgh, PA 15213, USA
You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass
Apr 13, 2026Yinuo Yang, Zixian Ma, Manasi Ganti +2Reward Modeling Multimodal Reward Modeling
Paul G. Allen School of Computer Science & Engineering University of Washington Seattle, WA 98195, USA
Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertainty
Apr 11, 2026Chao Xue, Yao Wang, Mengqiao Liu +11Reward Modeling Efficient Language Model Reasoning
University of New South Wales, Australia · Tencent Hunyuan, China · Tencent Yuanbao, China +2
Relational Preference Encoding in Looped Transformer Internal States
Apr 10, 2026Jan KirinPairwise Preference Learning Reward Modeling
ProMedical: Hierarchical Fine-Grained Criteria Modeling for Medical LLM Alignment via Explicit Injection
Apr 9, 2026He Geng, Yangmin Huang, Lixian Lai +5LLM Alignment Medical LLMs
Xunfei Healthcare Technology Co., Ltd.
ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
Apr 8, 2026Kai Qin, Liangxin Liu, Yu Liang +7Preference Modeling Self-Reflection in Language Model Agents
Shenzhen International Graduate School, Tsinghua University · Baidu Inc., Beijing, China

May 3, 2026J/K move · Enter open · S save

RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences

Yangyang Zhou, Yi-Chen Li

Beijing University of Posts and Telecommunications · National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Reward Modeling LLM Evaluation Preference Modeling Reward Model Evaluation

Open paper AskPDF arXiv

Reinforcement Learning from Human Feedback has become the standard paradigm for language model alignment, where reward models directly determine alignment effectiveness. In this work, we focus on how to evaluate the generalizability of reward models. By "generalizability", we mean the ability of RMs to correctly rank responses to align with diverse user preferences. However, existing reward model benchmarks are typically designed around a universal preference, failing to assess this generalization. To address this critical gap, we introduce RMGAP, a benchmark comprising 1,097 instances across Chat, Writing, Reasoning, and Safety domains. Since different users exhibit diverse preferences for the same task, we first generate four distinct responses with different linguistic profiles for each collected prompt. However, the original prompt set lacks the specificity to convey different preferences. We therefore construct tailored prompts by contrasting these candidates and designing scenarios in which one response becomes the uniquely appropriate choice. Moreover, we observe that users often express the same preference using different phrasings, and thus extend each prompt with two paraphrased variants. Our evaluation of 24 state-of-the-art RMs reveals their substantial limitations: even the best RM achieves only 49.27% Best-of-N accuracy, highlighting considerable room for improvement in reward model generalization. Related data and code are available at https://github.com/nanzhi84/RMGAP.