Reward Modeling

Follow this topic Ask about this topic

Reward modeling turns comparisons, ratings, demonstrations, rubrics, or verifiable outcomes into signals for judging responses and actions. Methods range from pairwise preference models and scalar outcome or process critics to generative rubric judges, tool-based verifiers, and uncertainty-aware or personalized models. These signals guide preference optimization, best-of-N selection, test-time search, and training in language, image and video generation, robotics, and agents. A central challenge is that high preference accuracy may not capture diverse or shifting human intent, while optimization can exploit scoring biases, so evaluation must consider calibration and downstream behavior as well as ranking accuracy.

Latest papers

325 papers49 papers in the last four weeks, up 69% on the four weeks before. 0.3% of all new papers.

ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training
Apr 8, 2026Yu Liang, Liangxin Liu, Longzheng Wang +5Reward Modeling LLM Alignment
Baidu Inc., Beijing, China
Personalizing Text-to-Image Generation to Individual Taste
Apr 8, 2026Anne-Sofie Maerten, Juliane Verwiebe, Shyamgopal Karthik +3Personalized Reward Modeling Personalized Image Aesthetics Assessment
Tübingen AI Center, University of Tübingen · Department of Brain and Cognition, KU Leuven
UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning
Apr 7, 2026Xiaolong Wei, Zerun Zhu, Simin Niu +9Text Generation RL for Language Models
Beihang University · Baidu Inc. · Renmin University of China
OmniDiagram: Advancing Unified Diagram Code Generation via Visual Interrogation Reward
Apr 7, 2026Haoyue Yang, Xuanle Zhao, Xuexin Liu +2Code Generation RL for Code Generation
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Zhejiang University
IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents
Apr 6, 2026Rongqian Chen, Yu Li, Zeyu Fang +3Computer-Use Agent Evaluation Reward Modeling
George Washington University, Washington D.C., USA
From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism
Apr 6, 2026Zhuohao Yu, Zhiwei Steven Wu, Adam BlockReward Hacking Reward Over-Optimization
Carnegie Mellon University · Columbia University
ARM: Advantage Reward Modeling for Long-Horizon Manipulation
Apr 3, 2026Yiming Mao, Zixi Yu, Weixin Mao +5Reward Modeling Long-Horizon Robotic Manipulation
LimX Dynamics · Beijing University of Posts and Telecommunications · Zhejiang University
Mitigating Reward Hacking in RLHF via Advantage Sign Robustness
Apr 3, 2026Shinnosuke Ono, Johannes Ackermann, Soichiro Nishimori +2Reward Hacking RL from Human Feedback
The University of Tokyo · RIKEN AIP
Beyond Semantic Manipulation: Token-Space Attacks on Reward Models
Apr 3, 2026Yuheng Zhang, Mingyue Huo, Minghao Zhu +2Adversarial Attacks Adversarial Text Attacks
UIUC · Independent Researcher · University of Massachusetts Amherst
Reinforcement Learning from Human Feedback: A Statistical Perspective
Apr 2, 2026Pangpang Liu, Chengchun Shi, Will Wei SunRL from Human Feedback LLM Alignment
Department of Biostatistics, Yale University. · Department of Statistics, London School of Economics and Political Science. · Mitch Daniels School of Business, Purdue University.
Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences
Apr 1, 2026Simona-Vasilica Oprea, Adela BâraReward Modeling Pairwise Preference Learning
Department of Economic Informatics and Cybernetics, Bucharest University of Economic Studies, Bucharest, Romania
Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization
Apr 1, 2026Gyuseok Lee, Wonbin Kweon, Zhenrui Yue +3Personalized Language Models Personalized Language Model Alignment
University of Illinois Urbana-Champaign · Korea University
Extending MONA in Camera Dropbox: Reproduction, Learned Approval, and Design Implications for Reward-Hacking Mitigation
Mar 31, 2026Nathan HeathReward Hacking Reward Modeling
Independent Researcher
MotionRFT: Unified Reinforcement Fine-Tuning for Text-to-Motion Generation
Mar 28, 2026Xiaofeng Tan, Wanjiang Weng, Hongsong Wang +3Text-to-Motion Generation Human Motion Generation
School of Computer Science and Engineering, Southeast University, Nanjing 211189, China, and also with Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · State Key Laboratory for Novel Software Technology, School of Intelligence Science and Technology, Nanjing University, Nanjing 210023, China · New Laboratory of Pattern Recognition (NLPR), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA), and also with School of Artificial Intelligence, University of Chinese Academy of Sciences
Towards Reward Modeling for AI Tutors in Math Mistake Remediation
Mar 25, 2026Kseniia Petukhova, Ekaterina KochmarReward Modeling Intelligent Tutoring Systems
Mohamed bin Zayed University of Artificial Intelligence
Unbiased Reward Modeling from Implicit Feedback for LLM Alignment
Mar 24, 2026Hao Wang, Haocheng Yang, Licheng Pan +7Reward Modeling LLM Alignment
Peking University · Xiaohongshu Inc. · National University of Singapore +1
Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling
Mar 23, 2026Young Hyun Cho, Will Wei SunPrivacy-Preserving ML Privacy-Preserving Language Models
Purdue University
SpatialReward: Verifiable Spatial Reward Modeling for Fine-Grained Spatial Consistency in Text-to-Image Generation
Mar 23, 2026Sashuai Zhou, Qiang Zhou, Junpeng Ma +9Reward Modeling T2I Generation
Zhejiang University · Alibaba Group · Fudan University
Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction
Mar 21, 2026Haoyu Wang, Yuxin Chen, Liang Luo +3RL for Language Models Process Reward Models
Georgia Institute of Technology · Meta AI
OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards
Mar 19, 2026Zehao Li, Zhenyu Wu, Yibo Zhao +11Reward Modeling RL for GUI Agents
University of Science and Technology of China · Shanghai AI Laboratory · CUHK MMLab +2
Reasoning over mathematical objects: on-policy reward modeling and test time aggregation
Mar 19, 2026Pranjal Aggarwal, Marjan Ghazvininejad, Seungone Kim +18LLM Mathematical Reasoning Mathematical Reasoning Benchmarks
FAIR at Meta · CMU · UCLA
CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks
Mar 19, 2026Hao Wang, Licheng Pan, Zhichao Chen +7Reward Modeling RL from Human Feedback
Peking University · Xiaohongshu Inc. · Zhejiang University
Efficient Exploration at Scale
Mar 18, 2026Seyed Mohammad Asghari, Chris Chute, Vikranth Dwaracherla +5RL from Human Feedback Data-Efficient Learning
The Efficient Agent Team, Google DeepMind
Adapting Technical-Service LLM Agents with Latent Logic Augmentation, Robust Noise Reduction, and Hybrid Reward Modeling
Mar 18, 2026Junzhuo Ma, Chenghuang Shen, Yi Yu +15LLM Agent Training LLM Post-Training
School of Mathematical Sciences, Fudan University, Shanghai, China · Alibaba Group, Hangzhou, China · Shanghai Center for Mathematical Sciences, Fudan University, Shanghai, China +3
Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models
Mar 17, 2026Pierre Krack, Tobias Jülg, Wolfram Burgard +1Reward Modeling Reward Design for RL
Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg, Germany. · TUM School of Computation, Information and Technology, Technical University of Munich, Germany
SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness
Mar 16, 2026Jingyu Lu, Yuhan Wang, Fan Zhuo +7Reward Modeling Spoken Dialogue Systems
Zhejiang University
MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels
Mar 13, 2026Tianyang Luo, Tao Feng, Zhigang Hua +4RL for Language Model Reasoning RL Fine-Tuning
University of Illinois Urbana-Champaign · Meta
Visual-ERM: Reward Modeling for Visual Equivalence
Mar 13, 2026Ziyu Liu, Shengyuan Ding, Xinyu Fang +7Reward Modeling RL for VLMs
Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University +1
Long-form RewardBench: Evaluating Reward Models for Long-form Generation
Mar 13, 2026Hui Huang, Yancheng He, Wei Liu +7Reward Modeling Reward Model Evaluation
Faculty of Computing, Harbin Institute of Technology, Harbin, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · School of Computer Science and Technology, Harbin Institute of Technolgy, Shenzhen, China
SteerRM: Debiasing Reward Models via Sparse Autoencoders
Mar 13, 2026Mengyuan Sun, Zhuohao Yu, Weizheng Gu +2Reward Modeling Debiased ML
National Engineering Research Center for Software Engineering, Peking University

Apr 8, 2026J/K move · Enter open · S save

ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training

Yu Liang, Liangxin Liu, Longzheng Wang, Yan Wang+4

Baidu Inc., Beijing, China

Reward Modeling LLM Alignment Self-Training for Language Models RL Fine-Tuning

Open paper AskPDF arXiv

Generative reward models (GRMs) have emerged as a promising approach for aligning Large Language Models (LLMs) with human preferences by offering greater representational capacity and flexibility than traditional scalar reward models. However, GRMs face two major challenges: reliance on costly human-annotated data restricts scalability, and self-training approaches often suffer from instability and vulnerability to reward hacking. To address these issues, we propose ConsistRM, a self-training framework that enables effective and stable GRM training without human annotations. ConsistRM incorporates the Consistency-Aware Answer Reward, which produces reliable pseudo-labels with temporal consistency, thereby providing more stable model optimization. Moreover, the Consistency-Aware Critique Reward is introduced to assess semantic consistency across multiple critiques and allocates fine-grained and differentiated rewards. Experiments on five benchmark datasets across four base models demonstrate that ConsistRM outperforms vanilla Reinforcement Fine-Tuning (RFT) by an average of 1.5%. Further analysis shows that ConsistRM enhances output consistency and mitigates position bias caused by input order, highlighting the effectiveness of consistency-aware rewards in improving GRMs. Our implementation is available at https://github.com/yuliangCarmelo/ConsistRM.