Preference-Based RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 77

All topics

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2LLM Alignment Direct Preference Optimization
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA
PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots
Oct 1, 2026Amr Mousa, Rifny Rachman, Neil Karavis +2Multi-Objective Reinforcement Learning Energy-Efficient Robot Locomotion
The University of Manchester, United Kingdom · BAE Systems, United Kingdom · University of Warwick, United Kingdom
PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviors
Sep 30, 2026Seungeun Rho, Wontaek Kim, Danfei Xu +1Robot Policy Learning Robot Policy Adaptation
School of Interactive Computing Georgia Institute of Technology
Patient-Centered Treatment Planning for Chronic Multimorbidity: A Hierarchical Reinforcement Learning Framework for Preference Modeling
Sep 30, 2026Nafiseh Payani, Soham Das, G. Anthony Wilson +1Healthcare Hierarchical RL
Department of Industrial and Systems Engineering at the University of Tennessee, Knoxville, TN, USA · Department of Industrial Engineering at the University of Arkansas, Fayetteville, AR, USA
Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedback
Sep 30, 2026Xinyi Ni, Lifeng LaiRisk-Sensitive RL Robust RL
Department of Electrical and Computer Engineering, University of California, Davis, CA 95616, USA
Revisiting scaling laws for reward optimization
Sep 29, 2026Ali Aouad, Aymane El Gadarri, Vivek F. FariasReward Modeling Reinforcement Learning
MIT
On Preference Coverage Collapse from Hindsight Relabeling in Multi-Objective Reinforcement Learning
Sep 22, 2026Baptiste Bonin, Caro Strickland, Audrey DurandMulti-Objective Reinforcement Learning Experience Replay
Mila (Quebec AI Institute) · Université Laval · CIFAR AI Chair
The Operable Pareto Front: Distilling Offline Search into Run-Time Control for Multi-Objective UAV Edge-Computing Scheduling
Sep 16, 2026Qiao Liao, Zhiyong Feng, Bin Wu +1Trajectory Optimization Edge Computing
School of Computer Science and Technology, Tianjin University, Tianjin, China · School of Computer Software, Tianjin University, Tianjin, China · School of Information Science and Engineering, Shandong Agriculture and Engineering University, Shandong, China
Specifying Reward Functions for RL Without Environment Sampling
Sep 14, 2026Stephane Hatgis-Kessell, W. Bradley Knox, Emma BrunskillLLM-Guided RL Reward Design for RL
1Computer Science Department, Stanford University · 2Computer Science Department, The University of Texas at Austin
ALIGN-HOLD: Experience Alignment for Real-Time Hold Control in Large-Scale Ride-Hailing Matching at DiDi
Sep 9, 2026Zuhao Zhang, Xu Liu, Kai Wan +3RL Control Reward Modeling
Didichuxing Co. Ltd Beijing, China · Shanghai Jiao Tong University Shanghai, China
Subspace Inference Enables Efficient Active Reward Learning from Preferences
Sep 3, 2026Yutai Zhou, Erdem BıyıkBayesian RL Preference Learning
Thomas Lord Department of Computer Science University of Southern California, Los Angeles, USA
Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization
Aug 10, 2026Wenxiao Zhao, Shu Wang, Ying Nian WuLLM Alignment Direct Preference Optimization
University of California, Los Angeles · Shanghai AI Laboratory
Multi-Agent Reinforcement Learning via Agent-Specific Preference
Aug 9, 2026Ni Mu, Yao Luan, Yiqin Yang +1Decentralized MARL Cooperative MARL
CFINS, Department of Automation, BNRist, Beijing Key Laboratory of Embodied Intelligence Systems, Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China · Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
Why Study Emergent Behavior When You Can Regulate It? Aligning Multi-Agent Systems with Reward Prediction
Aug 7, 2026Assaf Caftory, Almog Zemach, Moshe Butman +1Multi-Agent Reinforcement Learning Emergent Behavior in Multi-Agent Systems
EfiArazi School of Computer Science, Reichman University, Herzliya, Israel · Faculty of Computer Science, The College of Management Academic Studies, Rishon LeZion, Israel · School of Communication, Reichman University, Herzliya, Israel
ORACLE: A Multi-Objective Reinforcement Learning-Based Analog Circuit Design Optimizer with Large Language Models-Guided Exploration
Aug 5, 2026Osei Brempong, Mohammed Ayman Habib, Vivan Poddar +1Multi-Objective Reinforcement Learning Electronic Design Automation
University of Utah, Salt Lake City, UT, USA
SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
Aug 3, 2026Evan Assmus, Qining Zhang, Lei YingReinforcement Learning Robotic RL
University of Michigan, Ann Arbor
Crayotter: Learning Long-Horizon Video Editing Agents via Group-Relative Preference Backpropagation
Aug 3, 2026Lecheng Yan, Jianze Lin, Yichong Zhang +5Credit Assignment in RL Video Editing
University of Science and Technology of China · Beijing Normal University · Jilin University +4
LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
Jul 31, 2026Manith Adikari, Bei Peng, Samuele Vinanzi +1Multi-Objective Reinforcement Learning Reinforcement Learning
Department of Computer Science, University of Manchester · Centre for Robotics & AI, University of Manchester · School of Computer Science, University of Sheffield +1
Learning Latent Reasoning Traces for Scalar Reward Models End-to-End
Jul 31, 2026Sanwoo Lee, Clive Bai, Hsiu-Yuan Huang +3Reward Modeling LLM Alignment
National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · LLM Department, Tencent
AutoPref: Automatic Discovery of Task-Specific Preference Objectives for Neural Combinatorial Optimization
Jul 30, 2026Shengda Gu, Kai Li, Xinyi Ke +3Automated Algorithm Discovery Pairwise Preference Evaluation
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent AI Lab +1
Synchronizing Beliefs with Second-Order Theory-of-Mind in Human-Autonomy Teams (Extended Version)
Jul 29, 2026Jack Mirenzi, Henny AdmoniPairwise Preference Learning Theory of Mind
Robotics Institute, Carnegie Mellon University
Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation
Jul 21, 2026Li-Rong Zhou, Qin-Wen Luo, Sheng-Jun HuangReinforcement Learning Offline RL
Nanjing University of Aeronautics and Astronautics, Nanjing, China
Learning Safe Agent Behaviour from Human Preferences and Justifications via World Models
Jul 14, 2026Ilias Kazantzidis, Timothy J. Norman, Yali Du +1Safe RL Reward Design for RL
University of Southampton, Southampton SO17 1BJ, United Kingdom · King’s College London, London WC2R 2LS, United Kingdom
Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences
Jul 14, 2026Taehyung Kim, Gwangmo Lee, Minjun Chang +2Clustering Preference Alignment
Yonsei University, Seoul, South Korea · Georgia Institute of Technology, United States of America · Hyundai Motor Company, Seoul, South Korea
Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability
Jul 13, 2026Simone Drago, Marco Mussi, Leonardo Bianconi +1Pairwise Preference Learning Pairwise Comparison
Politecnico di Milano, Milan, Italy
Integrated Altruistic and Fairness Preference Induces Advanced Mutual Cooperation in Sequential Social Dilemmas
Jul 6, 2026Yu Wei, Yukiko Ogura, Yoshiyuki Ohmura +3Social Dilemmas Cooperative MARL
The University of Tokyo
SPLC: Social Preference Learning for Crowd Robot Navigation
Jul 2, 2026Zixuan Chen, Hao Fu, Haiwen Hu +1Robot Navigation Social Robot Navigation
Wuhan University of Science and Technology, Wuhan, China · School of Computer Science and Technology, Wuhan University of Science and Technology and also with the Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan 430081, China
CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
Jul 2, 2026Hexian Ni, Tao Lu, Yinghao CaiRobotic RL Multimodal Reward Modeling
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China.
VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning
Jul 1, 2026Kuan-Chen Chen, Winston Chen, Wei-Fang Sun +1Multimodal Reward Modeling Reward Design for RL
Department of Computer Science, National Tsing Hua University · NVIDIA AI Technology Center (NVAITC)
Freeform Preference Learning for Robotic Manipulation
Jun 30, 2026Marcel Torne, Anubha Mahajan, Abhijnya Bhat +1Long-Horizon Robotic Manipulation Robot Skill Learning
Stanford University
MAPL: Multi-Objective Preference Learning for Robot Locomotion
Jun 24, 2026Xiyue Chen, Muhan Lin, Shuyang Shi +1Multi-Objective Reinforcement Learning Reward Modeling
Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47906, USA · Department of Computer Science, Purdue University, West Lafayette, IN 47906, USA · independent researcher
Self-Evolution for Multi-Turn Tool-Calling Agents via Divergence-Point Preference Learning
Jun 22, 2026Jiaqiang TangTool-Using Agents Self-Improving Agents
The Hong Kong University of Science and Technology (Guangzhou)
UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning
Jun 17, 2026Mohamed Nabail, Leo Kaixuan Cheng, Jingmin Wang +1Pairwise Preference Evaluation RL Exploration
Learning, Embodied Autonomy, and Forecasting (LEAF) Lab, University of Toronto
Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning
Jun 15, 2026Tung M. Luu, Hwanhee Kim, Younghwan Lee +1Offline RL Semi-Supervised Learning
Korea Advanced Institute of Science and Technology (KAIST).
PAWS: Preference Learning with Advantage-Weighted Segments
Jun 10, 2026Aleksandar Taranovic, Onur Celik, Niklas Freymuth +6Credit Assignment in RL Robotic RL
Autonomous Learning Robots, Karlsruhe Institute of Technology, Karlsruhe, Germany · Bosch Center for Artificial Intelligence, Renningen, Germany · Institute for Material Handling and Logistics (IFL), Karlsruhe Institute of Technology, Karlsruhe, Germany
A Regret Minimization Framework on Preference Learning in Large Language Models
Jun 8, 2026Suhwan Kim, Taehyun Cho, Geon-Hyeong Kim +4Regret Minimization in RL LLM Alignment
Seoul National University, Seoul, South Korea · LG AI Research, Seoul, South Korea · UNIST, Ulsan, South Korea +1
FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization
Jun 3, 2026Yihao Wu, He Zhang, Junbo Tan +2Offline RL Vision-Language-Action Models
Tencent Robotics X · Futian Laboratory · Tsinghua University
Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation
Jun 2, 2026Zeyi Liu, Guangyao Liu, Yinuo Qu +6Credit Assignment in RL Robotic RL
Central South University · Nanyang Technological University · Zhejiang University
Pave-GRPO: Beyond Instantaneous Guidance through Principled Average Velocity Decomposition
Jun 1, 2026Pengyang Ling, Jiazi Bu, Yujie Zhou +6Diffusion Model Alignment Group Relative Policy Optimization
University of Science and Technology of China, China · Shanghai Jiao Tong University, China · Fudan University, China +3
S-SPPO: Semantic-Calibrated Self-Play Preference Optimization
Jun 1, 2026Xiwen Chen, Wenhui Zhu, Jingjing Wang +13LLM Alignment Self-Play RL
Morgan Stanley, USA · Arizona State University, USA · Clemson University, USA +5
Efficient Exploration for Iterative Nash Preference Optimization
May 31, 2026Tianlong Nan, Xiaopeng Li, Christian Kroer +1Nash Equilibrium LLM Alignment
Columbia University⋄ · The Chinese University of Hong Kong, Shenzhen†
From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
May 31, 2026Jun-Jie Yang, Chia-Heng Hsu, Kui-Yuan Chen +1Pairwise Preference Learning Representation Learning
Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems
May 29, 2026Jonathan Colaço Carr, Prakash Panangaden, Doina Precup +1Pairwise Preference Learning Markov Decision Processes
Stanford University · School of Computer Science, McGill University, Montreal, Quebec, Canada · Mila - Quebec AI Institute, Montreal, Quebec, Canada +1
Learning Kernel-Based MDPs from Episodic Preferential Feedback
May 22, 2026Nikola Pavlovic, Sattar Vakili, Qing ZhaoPairwise Preference Learning Regret Minimization in RL
Cornell University · MediaTek Research
From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning
May 22, 2026Ranxu zhang, zeyang li, Jiacheng Huang +5Agentic RL Reinforcement Learning
University of Science and Technology of China · Alibaba Group
Token-weighted Direct Preference Optimization with Attention
May 21, 2026Chengyu Huang, Zhuohang Li, Sheng-Yen Chou +1Pairwise Preference Learning LLM Alignment
Cornell University · Vanderbilt University
Implicit Safety Alignment from Crowd Preferences
May 20, 2026Qian Lin, Daniel S. BrownSafe RL Preference-Based RL
Kahlert School of Computing, University of Utah
How Much Online RL is Enough? Informative Rollouts for Offline Preference Optimization in RLVR
May 20, 2026Richa Verma, Balaraman RavindranDirect Preference Optimization RL for Language Model Reasoning
TCS Research Department of CSE IIT Madras Chennai, India · Department of Data Science & AI, Wadhwani School of Data Science & AI IIT Madras Chennai, India
PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment
May 20, 2026Richa Verma, Bavish Kulur, Sanjay Chawla +1Direct Preference Optimization Constrained RL
TCS Research, Department of CSE, IIT Madras India · Department of Computing Science, University of Alberta Canada · Qatar Computing Research Institute, Hamad Bin Khalifa University Qatar +1
LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models
May 19, 2026Redacted by arXivRL for Language Model Reasoning Group Relative Policy Optimization
General Preference Reinforcement Learning
May 18, 2026Muhammad Umer, Muhammad Ahmed Mohsin, Ahsan Bilal +5Multi-Objective Language Model Alignment RL Fine-Tuning
Stanford University · The University of Oklahoma
Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation
May 18, 2026Guining Cao, Jiaxin Peng, Chu Zeng +3Pairwise Preference Learning Open-Ended Generation
Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · School of Software and Microelectronics, Peking University · Tsinghua University
ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Tasks
May 17, 2026Tianxiang Xu, Xiaoyan Zhu, Xin Lai +1Reward Modeling Reinforcement Learning
School of Computer Science and Technology, Xi’an Jiaotong University
Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment
May 17, 2026Yucong Huang, Xiucheng Li, Kaiqi Zhao +1Reward Modeling LLM Alignment
Harbin Institute of Technology, Shenzhen, China.
OHP-RL: Online Human Preference as Guidance in Reinforcement Learning for Robot Manipulation
May 15, 2026Yunyang Mo, Jian Li, Qiwei Wu +2Robotic RL Human-in-the-Loop AI
The Hong Kong University of Science and Technology (Guangzhou)
Embedding-perturbed Exploration Preference Optimization for Flow Models
May 15, 2026Sujie Hu, Chubin Chen, Jiashu Zhu +3AI Alignment Group Relative Policy Optimization
1Tsinghua University · ⋆Work done during the internship at AMAP · 2AMAP, Alibaba Group
Distributed Zeroth-Order Policy Gradient for Networked Multi-agent Reinforcement Learning from Human Feedback
May 15, 2026Pengcheng Dai, He Wang, Dongming Wang +2Policy Gradient Decentralized MARL
Singapore University of Technology and Design, Singapore 487372, Singapore · Southeast University, Nanjing 211189, China · University of California, Riverside, CA 92521, USA +2
Driving Intents Amplify Planning-Oriented Reinforcement Learning
May 12, 2026Hengtong Lu, Victor Shea-Jay Huang, Chengmin Yang +4RL for Autonomous Driving Goal-Conditioned RL
Li Auto · Tsinghua University · CUHK MMLab
Semantic Reward Collapse and the Preservation of Epistemic Integrity in Adaptive AI Systems
May 12, 2026William ParrisReward Modeling Reward Hacking
BDB Labs / BagelTech · San Diego, California, USA
RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking
May 11, 2026Andrew Choi, Wei XuQ-Learning Robotic RL
Horizon Robotics

Oct 1, 2026, cs.CLJ/K move · Enter open · S save

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights

Andreea Dutulescu, Stefan Ruseti, Mihai Masala, Traian Rebedea+1

National University of Science and Technology POLITEHNICA Bucharest · NVIDIA

LLM Alignment Direct Preference Optimization Adaptive Loss Weighting Preference Optimization Preference-Based RL

Open paper AskPDF arXiv

Most preference optimization methods, such as Direct Preference Optimization (DPO), apply preference supervision at the response level, although autoregressive language models are optimized token by token. As a result, all tokens in a rejected response contribute to the negative training signal, including tokens that may encode behavior that is useful for the preferred response. We introduce GAW-PO, a gradient-aligned token reweighting method for DPO that estimates, for each rejected token, whether penalizing it would interfere with the preferred update directions. Tokens whose gradients are strongly aligned with the preferred behavior receive a weaker negative contribution, while conflicting tokens retain a stronger penalty. Our method achieves the highest average performance among the evaluated preference-optimization methods, improving by 0.97 points over standard DPO and 0.65 points over the strongest competing baseline across 11 benchmarks spanning mathematics, reasoning, coding, and question answering. We further show that gradient-aligned weighting is substantially more robust to aggressive preference optimization: as the DPO regularization parameter

β

decreases, standard DPO degrades sharply, whereas GAW-PO continues to improve. These results suggest that accounting for the interaction between rejected-token updates and preferred behavior provides an effective form of token-level credit assignment for preference optimization.

Momentum

Related topics

Latest papers 77