Reward Design for RL

RL: Reinforcement Learning

Reward design studies how to turn task goals, preferences, and constraints into learning signals, especially when success is sparse or objectives compete. Approaches include potential-based shaping and reward machines, preference and rubric models, visual progress rewards, intrinsic exploration bonuses, and automated reward search using language models. These methods are used in settings from robot manipulation and locomotion to language-model reasoning, tool use, and multi-agent coordination. A central tension is that denser or adaptive rewards can improve credit assignment but also mislead policies, destabilize training, or invite reward hacking, so evaluation must test intended outcomes rather than proxy scores alone.

Latest papers

261 papers44 papers in the last four weeks, up 83% on the four weeks before. 0.3% of all new papers.

Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning
Oct 27, 2026Saeed Hedayatian, Harsh Shah, Celine Mneimneh +1Reinforcement Learning Behavioral Diversity in RL
Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, USA · Independent Researcher
Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO
Oct 26, 2026Shurun Li, Michael Hua Wang, Haibo Zeng +1Reward Hacking Reinforcement Learning with Verifiable Rewards
REASONS Lab, Institute for Network Sciences and Cyberspace, Tsinghua University · Virginia Tech
Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization
Oct 26, 2026Sohyun Son, Yerim Hong, Jaeyun Park +1Reinforcement Learning RL for Language Model Reasoning
Kyung Hee University, Yongin, Republic of Korea
From Human Intent to In-Orbit Action: A Self-Learning Large Language Model Agent for 6G Satellite Networks
Oct 25, 2026Zhengqian Xu, Yasheng Zhang, Chenhua Sun +2LEO Satellite Networks 6G Networks
The 54th Research Institute of China Electronics Technology Group Corporation (CETC-54)
Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL
Oct 24, 2026Hyun Ryu, Jong-Heon Kim, Yiqing Xie +3Rubric-Based RL Reward Design for RL
KAIST AI · Carnegie Mellon University Language Technologies Institute
$R^2E:$ A Role-driven Reward Evolutionary Framework for Automated Reward Function Design
Oct 23, 2026Shouhao Chang, Xuan Liu, Hongye Zhu +2Reward Design for RL LLM-Guided RL
Hunan University · Central South University
Effectiveness of Curriculum Learning Depends on Reward Sparsity and Competing Optima
Oct 22, 2026John J. Vastola, Ann Huang, Satpreet H. Singh +2Curriculum RL Reinforcement Learning
Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Department of Psychology and Center for Brain Science, Harvard University
MPBRS: Multi-Potential-Based Reward Shaping for Unified Reward Configuration in Humanoid Locomotion
Oct 21, 2026Van Chung Nguyen, Hung Manh LaReward Shaping Reward Design for RL
are with the Advanced Robotics and Automation (ARA) Lab, Department of Computer Science and Engineering, University of Nevada, Reno, NV 89557, USA.
The Predictive--Optimization Gap in GuanDan Self-Play
Oct 21, 2026Qicheng Zou, Biao Yuan, Siyu Song +2Predict-Then-Optimize Self-Play RL
the University of California San Diego, La Jolla, CA, USA. · Chabiyue (Shanghai) Information Technology Co., Ltd., Shanghai, China.
Rewards for Budget-Aware Multi-Turn Agents That Adapt to Arbitrary Test-Time Constraints
Oct 20, 2026Tom A. Lamb, Adam Golinski, Eleonora Gualdoni +2LLM Agent Efficiency Reward Design for RL
Apple · University of Oxford
Cooperative Multi-Agent Reinforcement Learning for Market Making on Prediction-Market Order Books
Oct 19, 2026Skander LejmiAlgorithmic Trading Cooperative MARL
Department of Statistics, London School of Economics and Political Science
Intra-Rubric Reward Divergence: Mitigating Rubric Hacking through Latent Dimension Recovery and Balancing
Oct 19, 2026Jianqiang Xia, Han Xu, Dianxi ShiReward Hacking Rubric-Based RL
Shanghai Jiao Tong University · Baidu, Inc. · Intelligent Game and Decision Lab
HarnessRL: Co-Evolving Policies with a Just-in-Time Training Harness for Open-Ended RL
Oct 19, 2026Xiaoqiang Wang, Ting-En Lin, Yuchuan Wu +7Rubric-Based RL Reward Design for RL
Alibaba Token Hub, Alibaba Group · Université de Montréal · Mila - Quebec AI Institute
RARE: Reinforcement Learning with Adaptive Rubric Evolution for Open-Ended Generation
Oct 18, 2026Xueyun Tian, Bingbing Xu, Chuyue Li +9Rubric-Based RL Reward Design for RL
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Qwen Business Unit of Alibaba +1
Expert-Guided Rubrics as Dense Reward Signals for Reinforcement Learning in Non-Verifiable Domains
Oct 17, 2026Ashish Bansal, Hossam Amer, Keerat Kaur Guliani +3RL for LLM Agents Reward Design for RL
The Vanguard Group Malvern, Pennsylvania, USA · The Vanguard Group Toronto, Ontario, Canada
Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards
Oct 15, 2026Helena Merker, Nick Walker, Andreea BobuActive Preference Learning Preference Learning
Massachusetts Institute of Technology
Specification Gaming and a Constrained Reinforcement Learning Mitigation in a Cleaning Gridworld
Oct 15, 2026Hilda Adwubi OseiReward Hacking Reward Design for RL
Department of Industrial Engineering, Kwame Nkrumah University of Science and Technology, Kumasi, Ghana
Progress Reward Model for Reinforcement Learning via Large Language Models
Oct 15, 2026Xiuhui Zhang, Ning Gao, Xingyu Jiang +4Reward Design for RL LLM-Guided RL
Beihang University, 37 Xueyuan Road, Haidian District, Beijing
Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense
Oct 15, 2026Yushu Li, Jiin Woo, Xiaoxiao Li +1Reward Hacking RL for Language Models
University of British Columbia · Amazon Annapurna Labs
Judging the Judges: Confidence and Disagreement in Rubrics as Rewards
Oct 15, 2026Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell +1Rubric-Based RL Reward Design for RL
UC Berkeley
Design Space Exploration of Reinforcement Learning Configurations for Post-Legalization Placement
Oct 14, 2026Foteini Oikonomou, Maria Pantazi-Kypraiou, George Goudroumanis +5RL for Combinatorial Optimization Reward Design for RL
Department of Electrical and Computer Engineering, University of Thessaly, Volos, Greece · School of Computing Science, University of Glasgow, UK · Department of Electronic and Electrical Engineering, Trinity College Dublin, Ireland
FARL: A Fully Automated Pipeline of RL Interface and Solver Design
Oct 11, 2026Tianyu Jin, Liuyu Xiang, Chaowen Yang +3Reinforcement Learning Robotic RL
Beijing University of Posts and Telecommunications, Beijing, China
Harnessing VLM in Embodied RL: From Reward Regressor to Success Detector via Exploration-Augmented Key-Observation Gating
Oct 11, 2026Xin Liu, Yutong Jiang, Jian Su +4Reward Design for RL Reinforcement Learning
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of the Chinese Academy of Sciences · AGIBOT
GraftOOD: Grafting Out-of-Distribution Exemplars into Advantage-Weighted Rectification for Video Reward Optimization
Oct 9, 2026Zheming Fu, Ruizhe He, Wenbin Ning +7RL for Video Generation RL for Generative Models
Beihang University · Zhongguancun Academy · Joy Future Academy +2
MORAD: Reinforcement Learning with Multi-Objective Rewards for RNA Inverse Design
Oct 9, 2026Jixin Tang, Ji Guo, Jun ZhaoInverse Design Reward Design for RL
Fudan University
Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents
Oct 8, 2026Yiran Wu, Andrew Zhao, Yurui Chang +2Software Engineering Agents Reinforcement Learning
Pennsylvania State University · Microsoft Research · AG2AI
Beyond Outcome Rewards: Constructing and Assigning Retrieval Credit for Search Agents
Oct 7, 2026Wenyu Huang, Xinyu Hou, Pavlos Vougiouklis +2Reward Design for RL RL for IR
University of Edinburgh, UK · Huawei Technologies Research & Development (UK) Limited
LocusRL: Diagnosing LLM Reward and Policy Interventions in Competitive Games
Oct 3, 2026Chengyu Luan, Bo Xin, Songyan Guo +4RL for Game AI Reinforcement Learning
Harbin Institute of Technology
A Bird's-Eye View of Iterative Reward Design
Oct 3, 2026Logan Mondal Bhamidipaty, Lauren Robson, Linda Petrini +2Reward Design for RL LLM-Guided RL
Anthropic Fellows Program · University of Edinburgh · Anthropic
ROOT: Discovering Rewards for User-Specified Embodied Behaviors
Oct 3, 2026Eren Sadikoglu, Aditya Taparia, Xinyuan Liu +1Reward Design for RL Robotic RL
Robotics and Autonomous Systems, Arizona State University, Tempe, Arizona, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, Arizona, USA

Oct 27, 2026J/K move · Enter open · S save

Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning

Saeed Hedayatian, Harsh Shah, Celine Mneimneh, Stefanos Nikolaidis

Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, USA · Independent Researcher

Reinforcement Learning Behavioral Diversity in RL Reward Design for RL Proximal Policy Optimization

Open paper AskPDF arXiv

The goal of Quality-Diversity Reinforcement Learning (QD-RL) is to find a population of high-performing policies that solve a decision-making problem in diverse ways. Despite recent progress, QD-RL methods predominantly rely on evolutionary operators--including black-box optimizers, mutations, and crossovers--to facilitate exploration and induce diversity. Scaling these methods becomes increasingly costly as policies and behavior spaces grow larger. Building on the Soft QD view of quality-diversity optimization, we introduce Repulsive Population Policy Optimization (RePPO), which adapts PPO to jointly optimize a compact population of diverse, high-quality policies using a population-dependent, dynamically shaped reward. On standard continuous-control tasks, RePPO achieves a strong quality-diversity tradeoff compared with state-of-the-art QD-RL methods. Ablations further characterize this tradeoff by examining how each component affects individual quality and collective diversity. To assess RePPO's scalability, we extend our evaluation to learning from pixels and fine-tuning large language models. In pixel-based tasks, where large policies make most traditional QD-RL methods infeasible, RePPO maintains quality on par with standard PPO while achieving substantially higher diversity. Its successful application to language model fine-tuning further demonstrates, for the first time, that QD-RL methods can scale to this domain. Together, these results show that directly optimizing compact populations with RePPO provides a scalable approach to QD-RL across increasingly complex domains.