Preference-Based RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

10 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 77

All topics

MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
May 11, 2026Rohan Surana, Xintong Li, Sheldon Yu +7Direct Preference Optimization Active Learning
1UC San Diego · 2Adobe Research
Controllability in preference-conditioned multi-objective reinforcement learning
May 11, 2026Pau de las Heras Molins, Beyazit Yalcinkaya, Lasse Peters +2Multi-Objective Reinforcement Learning Preference-Based RL
LTCI, Télécom Paris, Institut Polytechnique de Paris · University of California, Berkeley · TU Delft +1
Reinforcement Learning for Scalable and Trustworthy Intelligent Systems
May 8, 2026Guangchen LanFederated RL Privacy-Preserving Language Models
Elmore Family School of Electrical and Computer Engineering, West Lafayette, Indiana · THE PURDUE UNIVERSITY GRADUATE SCHOOL
Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph
May 8, 2026Ning Liu, Chuanneng Sun, Kristina Klinkner +1Pairwise Preference Learning LLM Alignment
Amazon
Implicit Preference Alignment for Human Image Animation
May 8, 2026Yuanzhi Wang, Xuhua Ren, Jiaxiang Cheng +5Human Motion Generation Human Avatar Animation
Work done during the internship at Tencent Hunyuan. · School of Artificial Intelligence, Beijing Normal University · Tencent Hunyuan
Multi-Objective Constraint Inference using Inverse reinforcement learning
May 7, 2026Syed Ihtesham Hussain Shah, Floris den Hengst, Aneta Lisowska +1Reinforcement Learning Constrained RL
Faculty of Sciences Vrije Universiteit Amsterdam
A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment
May 7, 2026Hao YuPairwise Preference Learning LLM Alignment
Department of Automation, Tsinghua University
Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
May 6, 2026Xin Yu, Liuchen Liao, Yiwen Zhang +3On-Policy Self-Distillation Language Model Distillation
1The Pennsylvania State University · 2TikTok
Data-dependent Exploration for Online Reinforcement Learning from Human Feedback
May 6, 2026Zhen-Yu Zhang, Yuting Tang, Jiandong Zhang +2RL Exploration Sample-Efficient RL
Center for Advanced Intelligence Project, RIKEN · Graduate School of Frontier Sciences, The University of Tokyo · Northeastern University +1
PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning
May 1, 2026Ziqin Yuan, Ruiqi Wang, Dezhong Zhao +2Reward Modeling Mixture of Experts
1Purdue University, West Lafayette, IN, USA. · 2Beijing University of Chemical Technology, Beijing, China. · 3Indiana University Bloomington, Bloomington, IN, USA.
Design-MLLM: A Reinforcement Alignment Framework for Verifiable and Aesthetic Interior Design
Mar 4, 2026Yuxuan Yang, Xiaotong Mao, Jingyao WangLLM Alignment Indoor Scene Generation
Nanjing Forestry University, Nanjing, China · Université de Lorraine, Nancy, France · Laboratoire Réactions et Génie des Procédés, Nancy, France +2
Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
Feb 17, 2026Jialu Wang, Heinrich Peters, Asad A. Butt +6LLM Alignment Group Relative Policy Optimization
1Apple Inc., California, United States.
GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning
Jan 28, 2026Zhiheng Jiang, Yunzhe Wang, Ryan Marr +3Multi-Objective Reinforcement Learning RL Benchmarks
University of California, Los Angeles · USC Institute for Creative Technologies · U.S. Army DEVCOM Army Research Laboratory
Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedback
Oct 2, 2025Derek Shi, Ruben Glatt, Christine Klymko +5Video-Language Models RL Fine-Tuning
Stanford University, CA, USA · Lawrence Livermore National Laboratory, CA, USA · Microsoft, WA, USA
Learning Acrobatic Flight from Preferences
Aug 26, 2025Colin Merk, Ismail Geles, Jiaxu Xing +3RL for Robotics Aerial Robotics
Robotics and Perception Group, University of Zurich, Switzerland · Autonomous Learning and Predictive Intelligence Lab, University of Zurich, Switzerland
Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
Jun 14, 2025Sara Rajaram, R. James Cotton, Fabian H. SinzOffline RL Robust RL
Institute for Computer Science and Campus Institute for Data Science, University of Göttingen, Göttingen, Germany · University Hospital Tübingen, Tübingen, Germany · International Max Planck Research School for Intelligent Systems, Tübingen, Germany +2
Corruption Robust Offline Reinforcement Learning with Human Feedback
Feb 9, 2024Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban +2Offline RL Corruption Robustness
Dept. of Computer Science University of Warwick, UK · Max-Planck Institute for Software Systems, Germany · Independent Researcher London, UK

May 11, 2026, cs.LGJ/K move · Enter open · S save

MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization

Rohan Surana, Xintong Li, Sheldon Yu, Yiran Jenny Shen+6

1UC San Diego · 2Adobe Research

Direct Preference Optimization Active Learning Preference Optimization Preference-Based RL

Open paper AskPDF arXiv

Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and multiple rejected responses. However, optimizing over large negative pools is costly, and many candidates contribute redundant gradients due to their similar effects on policy updates. We introduce MASS-DPO, a multi-negative active sample selection method that derives a PL-specific Fisher-information objective for selecting compact, informative negative subsets within each prompt. The resulting log-determinant objective selects negatives that contribute complementary information for policy updates, yielding compact subsets that retain the full pool's information while reducing redundancy. In practice, this favors negatives whose gradients cover different update directions, reducing redundant signal from near-duplicate candidates while preserving the most useful training information. Across four benchmarks spanning recommendation and multiple-choice QA and three model families, MASS-DPO consistently exceeds or matches existing methods in accuracy, improves Recall/NDCG and margin-based optimization dynamics, and delivers stronger alignment with substantially fewer negatives.

Momentum

Related topics

Latest papers 77