Preference-Based RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 77

All topics

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2LLM Alignment Direct Preference Optimization
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA
PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots
Oct 1, 2026Amr Mousa, Rifny Rachman, Neil Karavis +2Multi-Objective Reinforcement Learning Energy-Efficient Robot Locomotion
The University of Manchester, United Kingdom · BAE Systems, United Kingdom · University of Warwick, United Kingdom
PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviors
Sep 30, 2026Seungeun Rho, Wontaek Kim, Danfei Xu +1Robot Policy Learning Robot Policy Adaptation
School of Interactive Computing Georgia Institute of Technology
Patient-Centered Treatment Planning for Chronic Multimorbidity: A Hierarchical Reinforcement Learning Framework for Preference Modeling
Sep 30, 2026Nafiseh Payani, Soham Das, G. Anthony Wilson +1Healthcare Hierarchical RL
Department of Industrial and Systems Engineering at the University of Tennessee, Knoxville, TN, USA · Department of Industrial Engineering at the University of Arkansas, Fayetteville, AR, USA
Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedback
Sep 30, 2026Xinyi Ni, Lifeng LaiRisk-Sensitive RL Robust RL
Department of Electrical and Computer Engineering, University of California, Davis, CA 95616, USA
Revisiting scaling laws for reward optimization
Sep 29, 2026Ali Aouad, Aymane El Gadarri, Vivek F. FariasReward Modeling Reinforcement Learning
MIT
On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning
Sep 22, 2026Baptiste Bonin, Caro Strickland, Audrey DurandMulti-Objective Reinforcement Learning Experience Replay
Mila (Quebec AI Institute) · Université Laval · CIFAR AI Chair
The Operable Pareto Front: Distilling Offline Search into Run-Time Control for Multi-Objective UAV Edge-Computing Scheduling
Sep 16, 2026Qiao Liao, Zhiyong Feng, Bin Wu +1Trajectory Optimization Edge Computing
School of Computer Science and Technology, Tianjin University, Tianjin, China · School of Computer Software, Tianjin University, Tianjin, China · School of Information Science and Engineering, Shandong Agriculture and Engineering University, Shandong, China
Specifying Reward Functions for RL Without Environment Sampling
Sep 14, 2026Stephane Hatgis-Kessell, W. Bradley Knox, Emma BrunskillLLM-Guided RL Reward Design for RL
1Computer Science Department, Stanford University · 2Computer Science Department, The University of Texas at Austin
ALIGN-HOLD: Experience Alignment for Real-Time Hold Control in Large-Scale Ride-Hailing Matching at DiDi
Sep 9, 2026Zuhao Zhang, Xu Liu, Kai Wan +3RL Control Reward Modeling
Didichuxing Co. Ltd Beijing, China · Shanghai Jiao Tong University Shanghai, China
Subspace Inference Enables Efficient Active Reward Learning from Preferences
Sep 3, 2026Yutai Zhou, Erdem BıyıkBayesian RL Preference Learning
Thomas Lord Department of Computer Science University of Southern California, Los Angeles, USA
Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization
Aug 10, 2026Wenxiao Zhao, Shu Wang, Ying Nian WuLLM Alignment Direct Preference Optimization
University of California, Los Angeles · Shanghai AI Laboratory
Multi-Agent Reinforcement Learning via Agent-Specific Preference
Aug 9, 2026Ni Mu, Yao Luan, Yiqin Yang +1Decentralized MARL Cooperative MARL
CFINS, Department of Automation, BNRist, Beijing Key Laboratory of Embodied Intelligence Systems, Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
Why Study Emergent Behavior When You Can Regulate It? Aligning Multi-Agent Systems with Reward Prediction
Aug 7, 2026Assaf Caftory, Almog Zemach, Moshe Butman +1Multi-Agent Reinforcement Learning Emergent Behavior in Multi-Agent Systems
EfiArazi School of Computer Science, Reichman University, Herzliya, Israel · Faculty of Computer Science, The College of Management Academic Studies, Rishon LeZion, Israel · School of Communication, Reichman University, Herzliya, Israel
ORACLE: A Multi-Objective Reinforcement Learning-Based Analog Circuit Design Optimizer with Large Language Models-Guided Exploration
Aug 5, 2026Osei Brempong, Mohammed Ayman Habib, Vivan Poddar +1Multi-Objective Reinforcement Learning Electronic Design Automation
University of Utah, Salt Lake City, UT, USA
SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
Aug 3, 2026Evan Assmus, Qining Zhang, Lei YingReinforcement Learning Robotic RL
University of Michigan, Ann Arbor
Crayotter: Learning Long-Horizon Video Editing Agents via Group-Relative Preference Backpropagation
Aug 3, 2026Lecheng Yan, Jianze Lin, Yichong Zhang +5Credit Assignment in RL Video Editing
University of Science and Technology of China · Beijing Normal University · Jilin University +4
LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
Jul 31, 2026Manith Adikari, Bei Peng, Samuele Vinanzi +1Multi-Objective Reinforcement Learning Reinforcement Learning
Department of Computer Science, University of Manchester · Centre for Robotics & AI, University of Manchester · School of Computer Science, University of Sheffield +1
Learning Latent Reasoning Traces for Scalar Reward Models End-to-End
Jul 31, 2026Sanwoo Lee, Clive Bai, Hsiu-Yuan Huang +3Reward Modeling LLM Alignment
National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · LLM Department, Tencent
AutoPref: Automatic Discovery of Task-Specific Preference Objectives for Neural Combinatorial Optimization
Jul 30, 2026Shengda Gu, Kai Li, Xinyi Ke +3Automated Algorithm Discovery Pairwise Preference Evaluation
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent AI Lab +1
Synchronizing Beliefs with Second-Order Theory-of-Mind in Human-Autonomy Teams (Extended Version)
Jul 29, 2026Jack Mirenzi, Henny AdmoniPairwise Preference Learning Theory of Mind
Robotics Institute, Carnegie Mellon University
Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
Jul 21, 2026Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun HuangReinforcement Learning Offline RL
Nanjing University of Aeronautics and Astronautics, Nanjing, China
Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models
Jul 14, 2026Ilias Kazantzidis, Timothy J. Norman, Yali Du +1Safe RL Reward Design for RL
University of Southampton, Southampton SO17 1BJ, United Kingdom · King’s College London, London WC2R 2LS, United Kingdom
Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences
Jul 14, 2026Taehyung Kim, Gwangmo Lee, Minjun Chang +2Clustering Preference Alignment
Yonsei University, Seoul, South Korea · Georgia Institute of Technology, United States of America · Hyundai Motor Company, Seoul, South Korea
Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability
Jul 13, 2026Simone Drago, Marco Mussi, Leonardo Bianconi +1Pairwise Preference Learning Pairwise Comparison
Politecnico di Milano, Milan, Italy
Integrated Altruistic and Fairness Preference Induces Advanced Mutual Cooperation in Sequential Social Dilemmas
Jul 6, 2026Yu Wei, Yukiko Ogura, Yoshiyuki Ohmura +3Social Dilemmas Cooperative MARL
The University of Tokyo
SPLC: Social Preference Learning for Crowd Robot Navigation
Jul 2, 2026Zixuan Chen, Hao Fu, Haiwen Hu +1Robot Navigation Social Robot Navigation
Wuhan University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Wuhan University of Science and Technology and also with the Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan 430081, China
CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
Jul 2, 2026Hexian Ni, Tao Lu, Yinghao CaiRobotic RL Multimodal Reward Modeling
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China.
VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning
Jul 1, 2026Kuan-Chen Chen, Winston Chen, Wei-Fang Sun +1Multimodal Reward Modeling Reward Design for RL
Department of Computer Science, National Tsing Hua University · NVIDIA AI Technology Center (NVAITC)
Freeform Preference Learning for Robotic Manipulation
Jun 30, 2026Marcel Torne, Anubha Mahajan, Abhijnya Bhat +1Long-Horizon Robotic Manipulation Robot Skill Learning
Stanford University

Oct 1, 2026, cs.CLJ/K move · Enter open · S save

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights

Andreea Dutulescu, Stefan Ruseti, Mihai Masala, Traian Rebedea+1

National University of Science and Technology POLITEHNICA Bucharest · NVIDIA

LLM Alignment Direct Preference Optimization Adaptive Loss Weighting Preference Optimization Preference-Based RL

Open paper AskPDF arXiv

Most preference optimization methods, such as Direct Preference Optimization (DPO), apply preference supervision at the response level, although autoregressive language models are optimized token by token. As a result, all tokens in a rejected response contribute to the negative training signal, including tokens that may encode behavior that is useful for the preferred response. We introduce GAW-PO, a gradient-aligned token reweighting method for DPO that estimates, for each rejected token, whether penalizing it would interfere with the preferred update directions. Tokens whose gradients are strongly aligned with the preferred behavior receive a weaker negative contribution, while conflicting tokens retain a stronger penalty. Our method achieves the highest average performance among the evaluated preference-optimization methods, improving by 0.97 points over standard DPO and 0.65 points over the strongest competing baseline across 11 benchmarks spanning mathematics, reasoning, coding, and question answering. We further show that gradient-aligned weighting is substantially more robust to aggressive preference optimization: as the DPO regularization parameter

β

decreases, standard DPO degrades sharply, whereas GAW-PO continues to improve. These results suggest that accounting for the interaction between rejected-token updates and preferred behavior provides an effective form of token-level credit assignment for preference optimization.

Momentum

Related topics

Latest papers 77