Reward Model Evaluation

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 22

All topics

Holdout Best-of-N: Unbiased Evaluation and Its Cost
Oct 6, 2026Shrey Shah, Yinheng LiBest-of-N Sampling Policy Evaluation
When Residualization Helps an Audit: Format Effects, Slice Gains, and Their Limits
Sep 21, 2026Daein Weon, Dong Ho KangLanguage Model Bias Evaluation Language Model Auditing
Kookmin University · UStechlab
F $^{2}$ DR: A Fine-Grained Full-Pipeline Reward Framework for DeepSearch Workflows
Sep 17, 2026Bojian Xiong, Wentao Ding, Yujing Lu +11Reward Modeling LLM Evaluation
Tianjin University · TJUNLP Lab, Tianjin University, Tianjin, China · Baidu Inc. +1
Evaluator Ensembles Under Reward Hacking: Covariance Geometry and Finite-Search Guarantees
Aug 8, 2026Fariya Afrin, Ibne Farabi ShihabReward Hacking LLM Evaluation
Department of Computer Science, Kalinga Institute of Industrial Technology · Department of Computer Science, Iowa State University
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
Jul 30, 2026Qiushi Sun, Kanzhi Cheng, Yian Wang +20VLM Evaluation Computer-Use Agent Benchmarks
The University of Hong Kong · Nanjing University · National University of Singapore +4
SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents
Jul 25, 2026Yang Wan, Zhenhao Zhang, Jierui Wang +1Reward Modeling LLM-as-a-Judge
1Zhejiang University · University of Rochester
Discretizing Reward Models
Jun 19, 2026Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika +4Reward Modeling Reinforcement Learning
1Carnegie Mellon University · 2Meta Superintelligence Labs
Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modeling
Jun 5, 2026Xing Yue, Linjuan Wu, Daoxin Zhang +2Reward Modeling LLM Evaluation
Zhejiang University · Xiaohongshu Inc.
Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill
Jun 2, 2026Tao Chen, Gangwei Jiang, Pengyu Cheng +10Reward Modeling Reward Model Evaluation
1Qwen Large Model Application Team, Alibaba · 2Sun Yat-sen University · 3The Chinese University of Hong Kong +3
When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming
Jun 2, 2026Zelalem Abahana, David Evans, Satish Mahadevan Srinivasan +1Reward Hacking LLM Evaluation
College of Engineering, The Pennsylvania State University, University Park, PA, USA · Department of Computer Science, University of Virginia, Charlottesville, VA, USA · 3Penn State Great Valley, The Pennsylvania State University, Malvern, PA, USA +1
HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models
Jun 2, 2026Shuang Liu, Yuxuan Bo, Qiuyang Zhao +4Reward Modeling Reward Hacking
Carnegie Mellon University · University of Virginia · Harvard University +4
Expected Value Alignment for Generative Reward Modeling in Formal Mathematics Verification
May 31, 2026Shihao Ji, Haotao Tan, Zihui Song +1Reward Modeling Automated Theorem Proving
Position: Good Embodied Reward Models Need Bad Behavior Data
May 31, 2026Ran Tian, Yilin Wu, Andrea BajcsyReward Modeling Preference Alignment
UC Berkeley · NVIDIA Research · Carnegie Mellon University
EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing
May 30, 2026Ibne Farabi Shihab, Fariya Afrin, Sanjeda Akter +1Process Reward Models Language Model Calibration
Department of Computer Science, Iowa State University · Department of Computer Science, Kalinga Institute of Industrial Technology · Department of Civil, Construction & Environmental Engineering, Iowa State University
FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Models
May 11, 2026Zeynel A. Uluşan, Burak S. Akbudak, Can S. Erer +1Reward Modeling Automated Theorem Proving
Koç University, Department of Computer Science and Engineering · Codeway Studios · Bo˘gaziçi University, Department of Computer Engineering +1
Misaligned by Reward: Socially Undesirable Preferences in LLMs
May 6, 2026Gayane Ghazaryan, Esra DönmezReward Modeling Language Model Safety Evaluation
Institute for Natural Language Processing, University of Stuttgart · 2Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart
StoryAlign: Evaluating and Training Reward Models for Story Generation
May 6, 2026Haotian Xia, Hao Peng, Yunjia Qi +4Reward Modeling Language Model Generation Evaluation
Department of Computer Science and Technology, Tsinghua University
RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences
May 3, 2026Yangyang Zhou, Yi-Chen LiReward Modeling LLM Evaluation
Beijing University of Posts and Telecommunications · National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring
May 1, 2026Indraneil Paul, Goran Glavaš, Iryna GurevychReward Modeling Reward Model Evaluation
UKP Lab, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE · Center for Artificial Intelligence and Data Science, University of Würzburg
From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
Apr 30, 2026Qingyu Ren, Tianjun Pan, Xingzhou Chen +1Reward Modeling RL for Language Models
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · 2Shanghai Artificial Intelligence Laboratory
When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
Apr 28, 2026Shuning Shang, Hubert Strauss, Stanley Wei +2Reward Modeling Policy Gradient
Princeton Language and Intelligence, Princeton University
Rethinking Reward Models for Multi-Domain Test-Time Scaling
Oct 1, 2025Dong Bok Lee, Seanie Lee, Sangwoo Park +12Reward Modeling LLM Evaluation
KAIST · TH Nürnberg · Microsoft Research Asia

Oct 6, 2026, cs.CLJ/K move · Enter open · S save

Holdout Best-of-N: Unbiased Evaluation and Its Cost

Shrey Shah, Yinheng Li

Best-of-N Sampling Policy Evaluation Minimax Estimation Reward Model Evaluation

Open paper AskPDF arXiv

Reusing the scores that select a Best-of-

N

winner can overstate its expected reward. We study evaluation from a fixed matrix of

K

independent scores per candidate for a policy that selects using

J

fresh scores. A single estimator based only on this matrix is exactly unbiased for expected judge reward under every independent, stable collection of candidate-specific score laws if and only if

J<K

, for every pool size

M\ge N\ge2

. At

J=K-1

, the selector deepens as

K

grows. For independent Gaussian scores with common variance and fixed

M\ge N\ge2

, the unbiased minimax risk in this regime is of order

σ^2/\sqrt K

, attained by Holdout; allowing bias improves the rate to

σ^2/K

. For two candidates, we derive the minimum-variance unbiased estimator at known variance and the sharp asymptotic unbiased minimax constant

1/(π\sqrt2)

, which Holdout attains without knowing the variance. The cyclic average over subsets and ties can be computed in

O(MK\log M)

operations. At fixed selector depth, cyclic evaluation of bounded scores has

O(K^{-1})

risk uniformly in pool size. The impossibility result concerns the fixed matrix: one additional fresh winner score permits unbiased evaluation of the all-

K

policy.

Momentum

Related topics

Latest papers 22