Preference Optimization

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 30% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 98

All topics

Successive Training Stages and Large Language Model Persuasion: Effects of Misalignment, Supervised Fine-Tuning, and Preference Optimization
Oct 7, 2026Antony Dalmiere, Pascal Marchand, Guillaume Auriol +1Persuasion in Language Models LLM Post-Training
LAAS-TRUST · LAAS-TRUST, INSA Toulouse · LAAS-TSF, LAAS
RobotAPO: Adversarial Physics Preference Optimization for Robotic Manipulation Video Generation
Oct 7, 2026Kerui Li, Zhe Jing, Chenyi Huang +4Physical Consistency in Video Generation Robotic Manipulation
Institute of Automation, Chinese Academy of Sciences · GigaAI · Beijing Institute of Technology
AuraSE: Low-Hallucination Generative Speech Enhancement via Multimodal Flow Matching and Inference Policy Optimization
Oct 5, 2026Yingda Shen, Yao Qian, Yuxuan Hu +7Inference-Time Optimization Audio Diffusion Models
The Chinese University of Hong Kong, Shenzhen · Microsoft Research
Rubric-Based Optimization for Text-to-Music Generation
Oct 2, 2026Ping Wang, Guang Yang, Shao-Rong Su +3Text-to-Music Generation Rubric-Based RL
Paul G. Allen School of Computer Science & Engineering, University of Washington · Department of Electrical and Computer Engineering, University of Washington · Allen Institute for AI
Mitigating Social Sycophancy via Pluralistic Preference Optimization
Oct 1, 2026Stephane Hatgis-Kessell, Myra Cheng, Xiaoxuan Hou +4LLM Sycophancy Preference Optimization
Department of Computer Science, Stanford University · Amazon · Department of Computer Science, University of Washington
GAW-PO: Preference Optimization with Gradient-Aligned Token Weights
Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2LLM Alignment Direct Preference Optimization
National University of Science and Technology POLITEHNICA Bucharest · NVIDIA
Uncertainty-Normalized Margins for Direct Preference Optimization
Sep 29, 2026Sadegh Khorasani, Petrus Mikkola, Matthias GrossglauserPairwise Preference Learning Direct Preference Optimization
School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland · University of Helsinki, Helsinki, Finland
Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement
Sep 28, 2026Hyun-Kurl Jang, Jihun Kim, Kuk-Jin YoonWeakly Supervised Semantic Segmentation Preference Optimization
Visual Intelligence Lab. KAIST
Agentic Multi-Turn Reasoning: A Fairness Approach
Sep 27, 2026Thanh-Dat Truong, Sankalp Pandey, Hugh Churchill +3Preference Optimization Agentic Reasoning
CVIU Lab, University of Arkansas, USA · Dep. of Physics, University of Arkansas, USA · Dep. of Geosciences, University of Arkansas, USA +1
A Zeroth-Order Paradigm for LLM Preference Alignment
Sep 16, 2026Peter Chen, Xi Chen, Wotao Yin +1Pairwise Preference Learning LLM Alignment
Department of Electrical Engineering and Computer Sciences (EECS) University of California, Berkeley Berkeley, CA 94720, USA · Stern School of Business New York University New York, NY 10012, USA · Decision Intelligence Lab (Seattle) DAMO Academy, Alibaba Group U.S. Bellevue, WA 98004, USA +1
StalePO: Anchored Token-Level Preference Optimization using Legacy Post-Edits in Machine Translation
Sep 14, 2026Rohit Dhaipule, Sukhdeep Singh Kharbanda, Prasanth Bathala +2Preference Optimization Language Model Post-Training
Translation Services, Amazon
Direct Preference Density Alignment for Conversational Audio Equalization
Sep 14, 2026Ioannis Stylianou, Sven Ewan Shepstone, Jon Francombe +2RL for Language Models LLM Alignment
Department of Electronic Systems Aalborg University Aalborg, Denmark · Bang & Olufsen A/S Struer, Denmark
Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training
Sep 9, 2026Junwon Ko, Dong-Jae Lee, Minchan Kwon +2Offline RL Preference Optimization
School of Electrical Engineering, KAIST Daejeon, Republic of Korea
Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
Sep 8, 2026Oleksandr Cherednichenko, Roman KlypaLLM Alignment LLM Safety Alignment
Department of Mathematics and Mathematical Statistics Integrated Science Lab, Umeå University Umeå, Sweden · Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK 38000 Grenoble, France
TV-Regulated OPD: Direction Matters in On-Policy Distillation
Sep 8, 2026Han Xiao, Yifan Niu, Dongyi Liu +2Preference Optimization On-Policy Distillation
The Hong Kong University of Science and Technology (Guangzhou) · The University of Edinburgh · The Hong Kong University of Science and Technology
CUNO: Curriculum and Preference Optimization for Stable Graph Unlearning under Mass Deletion
Sep 8, 2026Chenhan Zhang, Ali Braytee, Madhushi Bandara +4Preference Optimization Machine Unlearning
University of Technology Sydney
MARBO: Relational Belief Grounding for LLM Agents in Social Deduction Games
Sep 6, 2026Yechan Hwang, Sangjun Bae, Jeongmo Kim +2Game-Playing Agents Social Deduction Games
Graduate School of Artificial Intelligence, UNIST, Ulsan, South Korea.
MCPO: Modality-Contrastive Preference Optimization for Multimodal Chain-of-Thought Compression
Sep 4, 2026Guangheng Yang, Zhenliang Ni, Zhenkai Wu +4Efficient Multimodal Inference Multimodal Large Language Models
Huawei Technologies Ltd. Shanghai, China
FiMI Banking: A Sovereign Model for Indian Retail Banking
Sep 3, 2026NPCI AI Research Team, Aman Kumar, Asit Desai +15Financial Services RL for Language Models
NPCI AI Research Team
CoMerge: Conflict-Driven Preference Optimization for Multi-Task Model Merging
Sep 2, 2026Mingjie Zheng, Zihao Chen, Wenqing Chen +4Model Merging Preference Optimization
School of Software Engineering, Sun Yat-sen University, Zhuhai, China · The State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · School of Artificial Intelligence, Sun Yat-sen University, Zhuhai, China +1
Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation
Sep 1, 2026Thibaut Thonet, Jos Rozen, Laurent BesacierLLM Alignment TTS Evaluation
NAVER LABS Europe
Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization
Aug 31, 2026Camila Blank, Zhuofan Ying, Christopher Potts +2LLM Sycophancy Emergent Misalignment in Language Models
1Stanford University · 2Columbia University
Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimization
Aug 31, 2026Dishu Yang, Jingjing Liu, Jize LiLLM Alignment Memorization in Language Models
Khoury College of Computer Sciences Northeastern University San Jose, USA · Department of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley, USA · Department of Economics Boston University Boston, USA
VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matching
Aug 13, 2026Wenxiang Guo, Changhao Pan, Ziyue Jiang +2Text-to-Audio Generation Preference Optimization
Zhejiang University, Hangzhou, China
SSPO: Structure-Aware Similarity-Weighted Preference Optimization for Neural Combinatorial Optimization
Aug 12, 2026Yuanyu Li, Jintao Xu, Zijiang Liu +6Combinatorial Optimization Preference Optimization
Supply Chain Tech Team Y, JD.com, Beijing, China · Department of Industrial Engineering, Tsinghua University, Beijing, China · Faculty of Engineering and Faculty of Business and Economics, The University of Hong Kong, Hong Kong, China
TSPORec: Token Selection via Preference Optimization for LLM-Based Sequential Recommendation
Aug 10, 2026Wenqiao Zhu, Chao Xu, Haipang Wu +1LLM Inference Efficiency Sequential Recommendation
Learning Sexism Detection Using Multi-Agent Perspectivist Preference Optimization
Aug 4, 2026Hadi Mohammadi, Tina Shahedi, Robert A. Bagheri +2Annotator Disagreement Preference Optimization
Department of Methodology and Statistics, Utrecht University, The Netherlands · Department of Information and Computing Sciences, Utrecht University, The Netherlands · Field Technology Innovations (FTI) & Robotics, Wageningen University & Research, The Netherlands
Adaptive Human-Robot Collaborative Painting Combining Preference-Based Optimization and Dynamic Motion Primitives
Aug 3, 2026C. Cella, M. Ristic, M. Faroni +2Robot Motion Generation Human-Robot Collaboration
Politecnico di Milano
Cloud-ScPO: Hidden-State Geometry for Semi-Supervised Preference Optimization in LLM Reasoning
Aug 2, 2026Yuzhou Liu, Xiyang HuPreference Optimization Semi-Supervised Learning
University of Southern California · Arizona State University
Rolling With Resistance: Preference-Optimized LLM Counselors Can Trade Goal Persistence for Relational Attunement in Motivational Interviewing
Jul 30, 2026Weiying Chen, Junlong Shen, Zhexuan TangDirect Preference Optimization Preference Optimization
University of Alberta · University of Shanghai for Science and Technology

Oct 7, 2026, cs.AIJ/K move · Enter open · S save

Successive Training Stages and Large Language Model Persuasion: Effects of Misalignment, Supervised Fine-Tuning, and Preference Optimization

Antony Dalmiere, Pascal Marchand, Guillaume Auriol, Vincent Nicomette

LAAS-TRUST · LAAS-TRUST, INSA Toulouse · LAAS-TSF, LAAS

Persuasion in Language Models LLM Post-Training Supervised Fine-Tuning Preference Optimization

Open paper AskPDF arXiv

Large language models (LLMs) can be tuned to influence human attitudes, yet the respective contributions of successive post-training stages remain un-clear. This study examines how three successive training stages affect LLM persuasiveness: (1) misalignment through supervised fine-tuning (SFT) on conspiracy data, (2) additional persuasive SFT on argumentative data, and (3) Identity Preference Optimization (IPO), a preference-optimization method. A total of 835 participants recruited on Prolific were randomly assigned to five between-subject conditions (neutral text, conspiracy-trained model, persuasion-trained model, preference-optimized model, and GPT-4) and were exposed to texts on 10 divisive political issues, personalized from their individual profiles in all model conditions. Attitude change was measured as the difference between pre- and post-exposure positions on continuous Likert scales and analyzed with an analysis of covariance (ANCOVA). A significant condition x baseline-attitude interaction, F (4, 825) = 5.33, p < .001, indicated that training effects depended on participants' initial attitudes. Persuasive SFT produced greater attitude change than conspiracy training alone, d = 0.30, whereas IPO provided no additional benefit, d = 0.03, and GPT-4 did not differ from neutral text, d = --0.01. These results show that targeted supervised training on persuasive data increases LLM persuasiveness, whereas preference optimization yields no significant gains beyond it.

Momentum

Related topics

Latest papers 98