Reward Modeling

Follow this topic Ask about this topic

Reward modeling turns comparisons, ratings, demonstrations, rubrics, or verifiable outcomes into signals for judging responses and actions. Methods range from pairwise preference models and scalar outcome or process critics to generative rubric judges, tool-based verifiers, and uncertainty-aware or personalized models. These signals guide preference optimization, best-of-N selection, test-time search, and training in language, image and video generation, robotics, and agents. A central challenge is that high preference accuracy may not capture diverse or shifting human intent, while optimization can exploit scoring biases, so evaluation must consider calibration and downstream behavior as well as ranking accuracy.

Latest papers

325 papers49 papers in the last four weeks, up 69% on the four weeks before. 0.3% of all new papers.

Learning Ordinal Probabilistic Reward from Preferences
Feb 13, 2026Longze Chen, Qi Deng, Lu Wang +7Reward Modeling LLM Alignment
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Ritzz-AI +1
Next-token Preference: Scaling Unsupervised Reward Modeling
Feb 11, 2026Jingxuan Fan, Yueying Li, Zhenting Qi +4Reward Modeling Preference Learning
Harvard University · Cornell University · Microsoft Research +1
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
Feb 10, 2026Pei-Chi Pan, Yingbin Liang, Sen LinReward Modeling Reward Hacking
Department of Computer Science University of Houston · The Ohio State University · University of Houston
Bayesian Preference Learning for Test-Time Steerable Reward Models
Feb 9, 2026Jiwoo Hong, Shao Tang, Zhipeng WangReward Modeling Preference Modeling
LinkedIn Corporation · Nubank
Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models
Feb 7, 2026Yankai Yang, Yancheng Long, Hongyang Wei +12Reward Modeling Multimodal Reward Modeling
Harbin Institute of Technology, Shenzhen. · Kuaishou Technology.
CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models
Feb 4, 2026Xiao Zhu, Xinyu Zhou, Boyu Zhu +5RL for Code Generation Reward Modeling
LARK, HKUST(GZ) · Kuaishou Technology · UCL +3
Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis
Jan 31, 2026Zicheng Kong, Dehua Ma, Zhenbo Xu +9Reward Modeling RL for Language Models
Beijing University of Posts and Telecommunications, Beijing, P.R. China · ShiFang Technology Inc., Hangzhou, P.R. China · Tsinghua University, Beijing, P.R. China +1
P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist
Jan 6, 2026Kwangwook Seo, Dongha LeeReward Modeling Personalized Reward Modeling
Department of Artificial Intelligence,Yonsei University
HAL: Inducing Human-likeness in LLMs with Alignment
Jan 6, 2026Masum Hasan, Junjie Zhao, Ehsan HoqueReward Modeling LLM Alignment
University of Rochester
IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models
Jan 2, 2026Haonan Song, Qingchen Xie, Huan Zhu +12Reward Modeling Preference Modeling
HUJING Digital Media & Entertainment Group (XingYun Lab), Beijing, China · Department of Automation, Tsinghua University, Beijing, China · Beihang University, Beijing, China +2
AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens
Dec 19, 2025Tung-Ling Li, Yuhao Wu, Hongliang LiuLLM-as-a-Judge Adversarial Attacks on LLMs
Palo Alto Networks
Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
Dec 7, 2025Denis Belomestny, Alexey Naumov, Artemy Rubtsov +1Reward Modeling Reinforcement Learning
UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking
Nov 30, 2025Lingling Fu, Yongfu XueReward Modeling Reward Hacking
Guangxi University
RubricRL: Simple Generalizable Rewards for Text-to-Image Generation
Nov 25, 2025Xuelu Feng, Yunsheng Li, Ziyu Wan +4Reward Modeling T2I Generation
University at Buffalo · Microsoft AI · Nikola Tesla STEM High School
OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning
Oct 28, 2025Ziyou Hu, Zhengliang Shi, Minghang Zhu +5Reward Modeling RL for Language Models
Leiden University, Leiden, Netherland · Shandong University, Qingdao, China · Tsinghua University, Beijing, China
TripScore: Aligning LLMs for Real-World Travel Planning via Expert-Calibrated Reward
Oct 10, 2025Yincen Qu, Huan Xiao, Feng Li +5Reward Modeling LLM Planning
Trip.com Group · Chongqing University · Stanford University
Rethinking Reward Models for Multi-Domain Test-Time Scaling
Oct 1, 2025Dong Bok Lee, Seanie Lee, Sangwoo Park +12Reward Modeling LLM Evaluation
KAIST · TH Nürnberg · Microsoft Research Asia
Adaptive Margin RLHF via Preference over Preferences
Sep 26, 2025Yaswanth Chittepu, Prasann Singhal, Greg Durrett +1Pairwise Preference Learning Reward Modeling
University of Massachusetts Amherst · University of Texas at Austin
Aligning Audio Captions with Human Preferences
Sep 18, 2025Kartik Hegde, Rehana Mahfuz, Yinyi Guo +1Reward Modeling Audio-Language Models
Qualcomm Technologies, Inc., India · Qualcomm Technologies, Inc., USA
MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning
May 30, 2025Jingyan Shen, Jiarui Yao, Rui Yang +5Pairwise Preference Learning Reward Modeling
New York University · University of Illinois Urbana-Champaign · Rice University
R3: Robust Rubric-Agnostic Reward Models
May 19, 2025David Anugraha, Zilu Tang, Lester James V. Miranda +5Reward Modeling LLM Evaluation
Department of Computer Science, Stanford University · Department of Computer Science, Boston University · Allen Institute for AI +4
Bias Fitting to Mitigate Length Bias of Reward Model in RLHF
May 19, 2025Kangwen Zhao, Jianfeng Cai, Jinhua Zhu +5Reward Modeling LLM Alignment
University of Science and Technology of China
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
May 15, 2025Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +12Reward Modeling LLM Alignment
NLP Group, Fudan University · Hong Kong Baptist University · LLM Team, Meituan Inc
Supervised Reward Inference
Feb 25, 2025Will Schwarzer, Jordan Schneider, Philip S. Thomas +1Reward Modeling Reinforcement Learning
University of Massachusetts · Anthropic · Work done while at the University of Texas.
AgentRM: Enhancing Agent Generalization with Reward Modeling
Feb 25, 2025Yu Xia, Jingru Fan, Weize Chen +7Reward Modeling LLM Agents
Department of Computer Science and Technology, Tsinghua University · Gaoling School of Artificial Intelligence, Renmin University of China

Feb 13, 2026J/K move · Enter open · S save

Learning Ordinal Probabilistic Reward from Preferences

Longze Chen, Qi Deng, Lu Wang, Renke Shan+6

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Ritzz-AI+1

Reward Modeling LLM Alignment

Open paper AskPDF arXiv

Reward models are crucial for aligning large language models (LLMs) with human values and intentions. Existing approaches follow either Generative (GRMs) or Discriminative (DRMs) paradigms, yet both suffer from limitations: GRMs typically demand costly point-wise supervision, while DRMs produce uncalibrated relative scores that lack probabilistic interpretation. To address these challenges, we introduce a novel reward modeling paradigm: Probabilistic Reward Model (PRM). Instead of modeling reward as a deterministic scalar, our approach treats it as a random variable, learning a full probability distribution for the quality of each response. To make this paradigm practical, we present its closed-form, discrete realization: the Ordinal Probabilistic Reward Model (OPRM), which discretizes the quality score into a finite set of ordinal ratings. Building on OPRM, we propose a data-efficient training strategy called Region Flooding Tuning (RgFT). It enables rewards to better reflect absolute text quality by incorporating quality-level annotations, which guide the model to concentrate the probability mass within corresponding rating sub-regions. Experiments on various reward model benchmarks show that our method improves accuracy by

\textbf{2.9%}\sim\textbf{7.4%}

compared to prior reward models, demonstrating strong performance and data efficiency. Analysis of the score distribution provides evidence that our method captures not only relative rankings but also absolute quality.