Reward Design for RL
RL: Reinforcement Learning
Overview
Reward design studies how to turn task goals, preferences, and constraints into learning signals, especially when success is sparse or objectives compete. Approaches include potential-based shaping and reward machines, preference and rubric models, visual progress rewards, intrinsic exploration bonuses, and automated reward search using language models. These methods are used in settings from robot manipulation and locomotion to language-model reasoning, tool use, and multi-agent coordination. A central tension is that denser or adaptive rewards can improve credit assignment but also mislead policies, destabilize training, or invite reward hacking, so evaluation must test intended outcomes rather than proxy scores alone.
Latest papers
290 papers37 papers in the last four weeks, down 16% on the four weeks before. 0.3% of all new papers.
CODA: Difficulty-Aware Compute Allocation for Adaptive Reasoning
Decoupling Task and Behavior: A Two-Stage Reward Curriculum in Reinforcement Learning for Robotics
ToolRLA: Multiplicative Reward Decomposition for Tool-Integrated Agents
RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models
Keyframe-Guided Structured Rewards for Reinforcement Learning in Long-Horizon Laboratory Robotics
Adaptive Correlation-Weighted Intrinsic Rewards for Reinforcement Learning
RF-Agent: Automated Reward Function Design via Language Agent Tree Search
SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probing
How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1
Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications
PRISM: Parallel Reward Integration with Symmetry for MORL
Fairness over Equality: Correcting Social Incentives in Asymmetric Sequential Social Dilemmas
Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment
SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learning
The Trajectory Alignment Coefficient in Two Acts: From Reward Tuning to Reward Learning
Dual-Evolving Reinforcement Learning for Automated Reward Design
Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines
Gain Tuning Is Not What You Need: Reward Gain Adaptation for Constrained Locomotion Learning
Towards Shutdownable Agents via Stochastic Choice
Discounted Reward for Same-Length Trajectories (DReST)' reward function to train agents to (1) pursue goals effectively conditional on each trajectory-length (be USEFUL'), and (2) choose stochastically between different trajectory-lengths (be `NEUTRAL' about trajectory-lengths). In this paper, we propose evaluation metrics for USEFULNESS and NEUTRALITY. We use a DReST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL. Our results thus provide some initial evidence that DReST reward functions could train advanced agents to be USEFUL and NEUTRAL. Our theoretical work suggests that these agents would be useful and shutdownable.