Preference Learning for LLMs

LLM: Large Language Model

Follow this topic Ask about this topic

Latest papers

14 papers6 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

PORT: Preference Optimization via Robust Token-Level Reweighting
Nov 9, 2026Ding Zhu, Xiukun Wei, Tian Xie +3Preference Learning for LLMs LLM Alignment
Department of Computer Science and Engineering, The Ohio State University
TLRS-DPO: Token-Level Risk Scoring for Better Direct Preference Optimization
Nov 8, 2026Suxin Ji, Hungtao WanDirect Preference Optimization Safety-Preserving Fine-Tuning of Language Models
University of Pennsylvania · Independent Researcher
Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment
Nov 7, 2026Qisai Liu, Zhanhong Jiang, Zahid Hasan +1LLM Alignment Preference Optimization
Iowa State University, Ames, USA
AERO: Adaptive Ensemble-Disagreement Routing for Oracle Feedback in Sample-Efficient Online RLHF
Nov 7, 2026Shrey Singhal, Nannan Huang, Haytham M. FayekLLM Alignment RL from Human Feedback
RMIT University Melbourne, VIC, Australia
Alignment by Confident Debate: Preference Learning from Multi-Agent Debate
Nov 6, 2026Nikhil Verma, Dingxian Wang, Mohammad Reza Rezaei +2Preference Learning for LLMs LLM Alignment
Upwork Inc.
Don't Blame The Preferences: Examining The Emergence of Sycophancy in LLM Training
Nov 5, 2026Chao Peter Yang, Catherine Zehr, Pranav Bijith +1LLM Sycophancy LLM Post-Training
Yale University, New Haven, CT, USA · Duke University, Durham, NC, USA
Same Verdict, Different Readers: Personalizing Fact-Checking Explanations from Relative Preferences
Oct 27, 2026Haokun Wen, Xuemeng Song, Mong-Li Lee +1Preference Learning for LLMs Automated Fact-Checking
National University of Singapore Singapore · Southern University of Science and Technology Shenzhen, China
Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents
Aug 28, 2026Nan LiPreference Learning for LLMs LLM Agent Training
Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands
AdaHome: An Adaptive Smart Home Assistant using Local Small Language Models
Jul 20, 2026Eu Jin Lim, Zhaoxing Li, Sebastian SteinPreference Learning for LLMs Adaptive Inference
University of Southampton Southampton, United Kingdom
Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics
May 26, 2026Yuxuan Ye, Raul Santos-Rodriguez, Edwin SimpsonPreference Learning for LLMs Factual Consistency Evaluation
Intelligent System Laboratory University of Bristol United Kingdom
CoAct: Co-Active LLM Preference Learning with Human-AI Synergy
Apr 19, 2026Ruiyao Xu, Mihir Parmar, Tiankai Yang +3Preference Learning for LLMs RL for Language Models
Northwestern University · Google · University of Southern California +1
DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment
Apr 2, 2026Liang Zhu, Feiteng Fang, Yuelin Bai +4Preference Learning for LLMs LLM Alignment
Southern University of Science and Technology · Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Science and Technology of China +1
What Is Missing: Interpretable Ratings for Large Language Model Outputs
Feb 17, 2026Nicholas Stranges, Yimin YangPreference Learning for LLMs LLM Evaluation
Department of Electrical and Computer Engineering, Western University, Canada
Dual Active Learning for Reinforcement Learning from Human Feedback
Oct 3, 2024Pangpang Liu, Chengchun Shi, Will Wei SunRL from Human Feedback Preference Learning for LLMs
Mitchell E. Daniels, Jr. School of Business, Purdue University · Department of Statistics, London School of Economics and Political Science

Nov 9, 2026J/K move · Enter open · S save

PORT: Preference Optimization via Robust Token-Level Reweighting

Ding Zhu, Xiukun Wei, Tian Xie, Zhihui Zhu+2

Department of Computer Science and Engineering, The Ohio State University

Preference Learning for LLMs LLM Alignment Token-Level Loss Weighting

Open paper AskPDF arXiv

Preference optimization has become a central approach for aligning large language models with human values, but its effectiveness depends heavily on the quality of preference annotations. In practice, preference data is often noisy due to annotation errors and ambiguity. Existing robust preference optimization methods primarily operate at the sequence level, implicitly treating entire responses as uniformly correct or incorrect. However, rejected responses may still contain informative reasoning steps, while preferred responses can include subtle errors. In this work, we propose Preference Optimization via Robust Token-level reweighting (PORT), a fine-grained framework for robust alignment under noisy preferences. PORT performs token-level reweighting using the empirical cumulative distribution function (CDF) of token log-probabilities, yielding an efficient forward-pass-only proxy to gradient-norm penalization that selectively suppresses corrupted tokens without additional backward-pass computation. We provide theoretical analysis showing that PORT reduces the gradient bias from noisy preference labels by minimizing an upper bound on the bias term. Extensive experiments across diverse noise settings demonstrate that PORT consistently improves robustness and outperforms existing sequence-level baselines. The code is available at https://github.com/osu-srml/PORT