Reward Design for RL

RL: Reinforcement Learning

Reward design studies how to turn task goals, preferences, and constraints into learning signals, especially when success is sparse or objectives compete. Approaches include potential-based shaping and reward machines, preference and rubric models, visual progress rewards, intrinsic exploration bonuses, and automated reward search using language models. These methods are used in settings from robot manipulation and locomotion to language-model reasoning, tool use, and multi-agent coordination. A central tension is that denser or adaptive rewards can improve credit assignment but also mislead policies, destabilize training, or invite reward hacking, so evaluation must test intended outcomes rather than proxy scores alone.

Latest papers

261 papers44 papers in the last four weeks, up 83% on the four weeks before. 0.3% of all new papers.

From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning
May 22, 2026Ranxu zhang, zeyang li, Jiacheng Huang +5Agentic RL Reinforcement Learning
University of Science and Technology of China · Alibaba Group
Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework
May 21, 2026Shourov Joarder, Diganta Sikdar, Ahsan Habib Akash +2KL-Regularized RL Intrinsic Reward Methods
Bangladesh University of Engineering and Technology, Bangladesh · West Virginia University, USA · University of Aberdeen, UK +2
CIG: Exploration via Conditional Information Gain
May 20, 2026Tim Joseph, Marcus Fechner, Philipp Stegmaier +2Intrinsic Reward Methods Intrinsic Motivation
FZI · KIT
Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
May 19, 2026Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5Reward Modeling Rubric-Based RL
Scale AI · Persona
ARC-RL: A Reinforcement Learning Playground Inspired by ARC Raiders
May 19, 2026Carlo Romeo, Andrew D. BagdanovRL Benchmarks Robotics Simulation
Media Integration and Communication Center – University of Florence
AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning
May 18, 2026Peilin Wu, Xinlu Zhang, Kun Wan +4RL for Language Models Reinforcement Learning
The University of Texas at Dallas · Department of Computer Science, University of California, Santa Barbara · Adobe Inc.
Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation
May 18, 2026Guining Cao, Jiaxin Peng, Chu Zeng +3Pairwise Preference Learning Open-Ended Generation
Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · School of Software and Microelectronics, Peking University · Tsinghua University
Step-wise Rubric Rewards for LLM Reasoning
May 17, 2026Weichu Xie, Haozhe Zhao, Wenpu Liu +15Reward Modeling Credit Assignment in RL
Peking University · JD Explore Academy · Shanghai Jiao Tong University +2
TIER: Trajectory-Invariant Execution Rewards for Multi-Step Tool Composition
May 16, 2026Anay Kulkarni, ChiaEn Lu, Dheeraj Mekala +3Reward Modeling Reinforcement Learning with Verifiable Rewards
UC San Diego · Cisco Research
Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization
May 13, 2026Yang Bai, Kaiyuan Liu, Ziyuan Zhuang +5Multi-Objective Reinforcement Learning RL for Language Models
Meituan, China
ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization
May 12, 2026Nirmal Patel, Fei Wang, Inderjit S. DhillonRL for Language Models Reinforcement Learning
University of Texas at Austin · Google
Reward Hacking in Rubric-Based Reinforcement Learning
May 12, 2026Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang +3Reward Hacking Reinforcement Learning
Scale AI
EvoNav: Evolutionary Reward Function Design for Robot Navigation with Large Language Models
May 12, 2026Zhikai Zhao, Chuanbo Hua, Federico Berto +4Evolutionary Optimization Robot Navigation
1KAIST · 2Radical Numerics · 3UC Riverside +1
GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillation
May 12, 2026Sijia Li, Yuchen Huang, Zifan Liu +7RL for Language Model Reasoning Credit Assignment in RL
Hong Kong University of Science and Technology · Microsoft Research Asia
Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning
May 8, 2026Manish Bhattarai, Ismael Boureima, Nishath Rajiv Ranasinghe +2Reinforcement Learning RL for Language Model Reasoning
Los Alamos National Laboratory, Los Alamos, NM, USA.
ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression
May 8, 2026Tingcheng Bian, Yuzhe Zhang, Jing Jin +5Reward Shaping RL for Language Model Reasoning
1Baidu Inc. · 2Shenzhen University · 3Peking University +2
MARBLE: Multi-Aspect Reward Balance for Diffusion RL
May 7, 2026Canyu Zhao, Hao Chen, Yunze Tong +3Multi-Objective Reinforcement Learning Diffusion Model Alignment
Zhejiang University · Work done during an internship at HiThink. · HiThink +1
Reward Shaping and Action Masking for Compositional Tasks using Behavior Trees and LLMs
May 7, 2026Nicholas Potteiger, Ankita Samaddar, Taylor T. Johnson +1Reinforcement Learning Reward Shaping
Vanderbilt University, Nashville, TN
RVPO: Risk-Sensitive Alignment via Variance Regularization
May 7, 2026Ivan Montero, Tomasz Jurczyk, Bhuwan DhingraRL for Language Models Risk-Sensitive RL
Apple
Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards
May 5, 2026Tianyang Han, Hengyu Shi, Junjie Hu +3RL for Code Generation Process Reward Models
1D4 Lab · 2Independent Researcher
Discovering Reinforcement Learning Interfaces with Large Language Models
May 5, 2026Akshat Singh Jaswal, Ashish Baghel, Paras ChopraReinforcement Learning LLM-Guided RL
Lossfunk
Binary Rewards and Reinforcement Learning: Fundamental Challenges
May 4, 2026Marc DymetmanReinforcement Learning KL-Regularized RL
∗Scientific Consultant
Intervention Complexity as a Canonical Reward and a Measure of Intelligence
May 4, 2026Brendan McCaneAgent Evaluation Reward Design for RL
School of Computing University of Otago Dunedin, New Zealand
Enhanced LLM Reasoning by Optimizing Reward Functions with Search-Driven Reinforcement Learning
May 3, 2026Arash Ahmadi, Sarah Sharif, Yaser +1Reinforcement Learning Large Language Model-Guided Optimization
Mike · School of Electrical and Computer Engineering, University of Oklahoma, Norman, OK 73019 USA · Intelligent Neuromorphic and Quantum Understanding for Innovative Research and Engineering (INQUIRE) Laboratory, University of Oklahoma, Norman, OK, USA
MAGIC: Multi-Step Advantage-Gated Causal Influence for Multi-agent Reinforcement Learning
May 3, 2026Haohan Yu, Jinmiao Cong, Shengzhi Wang +2Multi-Agent Coordination Intrinsic Reward Methods
1Dalian University of Technology · 2Microsoft, Beijing, China
Exploring Pass-Rate Reward in Reinforcement Learning for Code Generation
May 1, 2026Xin-Ye Li, Ren-Biao Liu, Yun-Ji Zhang +3RL for Code Generation Reinforcement Learning
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China
RHyVE: Competence-Aware Verification and Phase-Aware Deployment for LLM-Generated Reward Hypotheses
Apr 30, 2026Feiyu Wu, Xu Zheng, Zhuocheng Wang +2Reinforcement Learning LLM-Guided RL
School of Cyber Engineering, Xidian University
When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
Apr 28, 2026Shuning Shang, Hubert Strauss, Stanley Wei +2Reward Modeling Policy Gradient
Princeton Language and Intelligence, Princeton University
How Can Reinforcement Learning Achieve Expert-level Placement?
Apr 28, 2026Ruo-Tong Chen, Ke Xue, Chengrui Gao +7Reinforcement Learning Electronic Design Automation
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · Huawei Noah’s Ark Lab, China
Zero Shot Coordination for Sparse Reward Tasks with Diverse Reward Shapings
Apr 28, 2026Keenan Powell, Peihong Yu, Pratap TokekarMulti-Agent Coordination Reward Shaping
Department of Computer Science University of Maryland, College Park

May 22, 2026J/K move · Enter open · S save

From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning

Ranxu zhang, zeyang li, Jiacheng Huang, Rui Zhang+4

University of Science and Technology of China · Alibaba Group

Agentic RL Reinforcement Learning Agent Skill Learning Policy Learning Reward Design for RL Memory-Augmented Agents+2

Open paper AskPDF arXiv

Agentic reinforcement learning (Agentic RL) has achieved strong progress in tasks with clear success signals. However, many real-world agent applications require user-conditioned behavior: the same query may call for different planning strategies and tool-use decisions across users. This setting raises key challenges: generic rewards cannot capture heterogeneous user preferences, observed behaviors are entangled with conformity effects, and flat memories cannot support personalized skill retrieval. To this end, we propose a unified personalized Agentic RL framework that embeds personalization into training-time optimization. At its core is Personalized Anchor Reward-Decoupled Policy Optimization (PARPO), which decouples generic task-quality rewards from personalized preference rewards and uses user-specific anchors to stabilize learning under heterogeneous reward scales. We further introduce a two-stage preference-disentangled reward model and Preference-Aligned Skill Evolution Graph Memory (PSGM) for personalized supervision and preference-aligned skill retrieval. Together, they form a closed loop of preference identification, policy optimization, and structured skill accumulation. Experiments on ETAPP, ETAPP-Hard, and SJAgent show that our framework consistently outperforms strong memory and RL baselines. Code and data are included in the supplementary materials.