Reward Modeling

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 118% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 197

All topics

SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation
Jun 9, 2026Qianzhong Chen, Hau Zheng, Justin Yu +8RL for Robotics Reward Modeling
Stanford University · UC Berkeley · Shanghai Jiao Tong University
Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short
Jun 8, 2026Han Zhou, Adam X. Yang, Laurence Aitchison +2Pairwise Preference Learning Reward Modeling
University of Cambridge · 2Mistral AI · *Work done as an AI Scientist Intern at Mistral AI.
Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions
Jun 8, 2026Xin Jin, Huanqia Cai, Zhen Li +9Reward Modeling Reasoning Distillation
Z-Image Team, Alibaba Group · VCIP, CS, Nankai University
A Unifying Lens on Reward Uncertainty in RLHF
Jun 8, 2026Ely Hahami, Yoel Zimmermann, Ray Zhou +1Reward Modeling KL-Regularized RL
Harvard College, Harvard University, Cambridge, MA, USA · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity
Jun 8, 2026Fengyuan Liu, Yongliang Miao, Zirui He +3Pairwise Preference Learning Reward Modeling
The Chinese University of Hong Kong, Shenzhen · New Jersey Institute of Technology · Institute of Computing Technology, CAS
Support Vector Rubrics: Closing the Gap Between Self-Generated and Human Rubrics
Jun 6, 2026Mengyuan Sun, Yu Li, Zhuohao Yu +2Reward Modeling Rubric-Based Evaluation
National Engineering Research Center for Software Engineering, Peking University · University of Science and Technology of China
PAFO: Pareto Fairness Optimization for Personalized Reward Modeling
Jun 6, 2026Xiaoyan Zhao, Haoting Ni, Yang Zhang +3Reward Modeling LLM Alignment
National University of Singapore · University of Science and Technology of China · Peking University
Beyond Rubrics: Exploration-Guided Evaluation Skills for Reward Modeling
Jun 5, 2026Xing Yue, Linjuan Wu, Daoxin Zhang +2Reward Modeling LLM Evaluation
Zhejiang University · Xiaohongshu Inc.
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
Jun 3, 2026Saket Reddy, Ke Yang, ChengXiang ZhaiReward Modeling RL for Language Models
University of Illinois - Urbana-Champaign
Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling
Jun 2, 2026Yifan Wang, Jinyi Mu, Mayank Jobanputra +5Pairwise Preference Learning Reward Modeling
Saarland University · Independent Researcher · Bielefeld University +2
Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill
Jun 2, 2026Tao Chen, Gangwei Jiang, Pengyu Cheng +10Reward Modeling Reward Model Evaluation
1Qwen Large Model Application Team, Alibaba · 2Sun Yat-sen University · 3The Chinese University of Hong Kong +3
Mitigating False Credit Propagation: Probabilistic Graphical Reward Aggregation for Rubric-Based Reinforcement Learning
Jun 2, 2026Can Lv, Mingju Chen, Heng Chang +1Reward Modeling RL for Language Models
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University · Tsinghua University
ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents
Jun 2, 2026Zheng Liu, Longxiang Zhang, Xintong Wang +8Reward Modeling Process Reward Models
Tsinghua University · Alibaba Group · Peking University +1
Follow-Your-Preference++: Rethinking Preference Alignment for Image Inpainting
Jun 2, 2026Junkun Yuan, Yutao Shen, Toru Aonishi +2Pairwise Preference Learning Reward Modeling
Zhejiang University · The University of Tokyo · Tsinghua University
HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models
Jun 2, 2026Shuang Liu, Yuxuan Bo, Qiuyang Zhao +4Reward Modeling Reward Hacking
Carnegie Mellon University · University of Virginia · Harvard University +4
Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
Jun 1, 2026Seojeong Park, Jiho Choi, Junyong Kang +3Reward Modeling LLM-as-a-Judge
Graduate School of Artificial Intelligence, KAIST, Republic of Korea · KRAFTON
Expected Value Alignment for Generative Reward Modeling in Formal Mathematics Verification
May 31, 2026Shihao Ji, Haotao Tan, Zihui Song +1Reward Modeling Automated Theorem Proving
Position: Good Embodied Reward Models Need Bad Behavior Data
May 31, 2026Ran Tian, Yilin Wu, Andrea BajcsyReward Modeling Preference Alignment
UC Berkeley · NVIDIA Research · Carnegie Mellon University
From Empathy to Personalized Empathy: Adapting Empathetic Strategies to Individual Users
May 30, 2026Wuqiang Zheng, Chengbing Wang, Yilin Yang +6Reward Modeling LLM Personalization
University of Science and Technology of China · 2Huawei Technologies · 3China Academy of Cyber
Latent Reward Steering: An Adaptive Inference-Time Framework that Implicitly Promotes Cognitive Behaviors in Reasoning LLMs
May 30, 2026Jiakang Li, Guanyu Zhu, Can Jin +8Reward Modeling Language Model Steering
Rutgers University · South China Agricultural University · Columbia University +4
CARE-RL: Capability-Aware Reinforcement Learning for Mitigating Cross-Domain Conflicts
May 30, 2026Rui Zhang, Xinle Wu, Yao LuReward Modeling Reinforcement Learning
National University of Singapore Singapore
The Representation-Rationalizability Tradeoff in Reward Learning
May 29, 2026Jing Dong, Yaoliang Yu, Pascal PourpartPairwise Preference Learning Reward Modeling
Vector Institute · University of Waterloo & Vector Institute
The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
May 29, 2026Xiaobo Wang, Tong Wu, Min Tang +3Reward Modeling LLM Alignment
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · State Key Laboratory of General Artificial Intelligence, BIGAI · University of Science and Technology of China +1
Reward Learning from Best-of- $N$ Preference Data: Targets, Tradeoffs, and Design Principles
May 28, 2026Rattana Pukdee, Maria-Florina Balcan, Pradeep RavikumarPairwise Preference Learning Reward Modeling
Machine Learning Department Carnegie Mellon University
Configurable Reward Model for Balanced Safety Alignment
May 28, 2026Zhengping Jiang, Mehran Khodabandeh, Akash Bharadwaj +5Reward Modeling Language Model Safety Evaluation
Johns Hopkins University · Meta · Meta Superintelligence Labs
Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modeling
May 28, 2026Yuchen Liu, Yingjie Feng, Lixiong Qin +5Reward Modeling Agentic RL
Beijing University of Posts and Telecommunications · Li Auto Inc.
Prompt-Level Reward Specifications for Open-Ended Post-Training
May 28, 2026Zijun Weng, Xiaohui Hu, Shuangyong Song +3Open-Ended Generation Reward Modeling
1Fudan University · 2Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd.
RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains
May 27, 2026Haoxiang Jiang, Zihan Dong, Tianci Liu +5Reward Modeling RL for Language Models
University at Albany · 2Rutgers University · 3Purdue University +3
Refining Multidimensional Video Reward Models via Disentangled Influence Functions
May 27, 2026Muyao Wang, Zeke Xie, Hideki NakayamaReward Modeling Training Data Selection
The University of Tokyo · HKUST (Guangzhou)
StoryLens: Preference-Aligned Story Rewriting via Context-Aware Narrative Enrichment
May 27, 2026Hanwen Cui, Yuting Mei, Yuhang Fu +2Reward Modeling Personalized Text Generation
Beijing University of Posts and Telecommunications · AIM3 Lab, Renmin University of China · Nanyang Technological University

Jun 9, 2026, cs.ROJ/K move · Enter open · S save

SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulation

Qianzhong Chen, Hau Zheng, Justin Yu, Suning Huang+7

Stanford University · UC Berkeley · Shanghai Jiao Tong University

RL for Robotics Reward Modeling Long-Horizon Robotic Manipulation Robot Skill Learning Multi-Task Robotic Manipulation Multimodal Reward Modeling+2

Open paper AskPDF arXiv

Fine-tuning vision-language-action (VLA) policies for long-horizon manipulation still relies heavily on behavior cloning, which requires costly high-quality demonstrations and keeps policies near the demonstration distribution. Reward models can reduce this dependence by reweighting demonstrations and providing dense supervision for on-robot reinforcement learning (RL), but they must be dense, accurate, and general. Existing methods fall short: task-specific stage-aware models are accurate but require per-task annotations, while general vision-language-model (VLM) reward models are broadly applicable but too coarse for fine-grained long-horizon progress. We introduce RM, a multi-task stage-aware reward model that combines an action-primitive-based stage estimator with a multi-gate Mixture-of-Experts (MMoE) value head to produce dense per-step rewards across manipulation tasks. Building on RM, we further propose SPIRAL (Self-Policy Improvement via Reward-Aligned Learning), an on-policy reward-guided framework that improves VLA policies from cheap autonomous rollouts. On a 10-task benchmark, RM reduces value-estimation MSE by 80% over the strongest baselines; when used in SPIRAL, it improves task success from around 50% to near-perfect performance on Folding Shorts (58% to 100%) and Cleaning Whiteboard (50% to 90%), showing that high-quality dense rewards are key to a stable robot data flywheel. Project website: https://qianzhong-chen.github.io/sarm2.github.io/.

Momentum

Related topics

Latest papers 197