Preference Optimization

Follow this topic Ask about this topic

Momentum

13 papers in the last four weeks, up 30% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 98

All topics

Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion
Jul 30, 2026Henglin Liu, Fangyuan Kong, Jing Wang +7Diffusion Model Alignment Video Diffusion Models
Tsinghua University · Kling Team, Kuaishou Technology · Sun Yat-sen University
AutoPref: Automatic Discovery of Task-Specific Preference Objectives for Neural Combinatorial Optimization
Jul 30, 2026Shengda Gu, Kai Li, Xinyi Ke +3Automated Algorithm Discovery Pairwise Preference Evaluation
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Tencent AI Lab +1
BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences
Jul 29, 2026Ru Peng, Haokai Xu, Xijun Gu +11Pairwise Preference Learning Preference Optimization
Zhejiang University · Inclusion AI, Ant Group · Qwen Team, Alibaba Group +2
TELLER: Dual-Path Iterative Preference Optimization for Table Entity Linking
Jul 29, 2026Yixin Peng, Kehao Li, Stefan DeckerDirect Preference Optimization Preference Optimization
RWTH Aachen, Ahornstraße 55, 52074 Aachen, Germany
Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization
Jul 27, 2026Zhengtao Yao, Runhao Li, Xupeng Chen +12Preference Optimization
University of Southern California · New York University · Columbia University +4
EvoThink: Evolving Thinking in Large Reasoning Models via Self-Pruning and Aha-Moment Preference Optimization
Jul 22, 2026Xinbang Dai, Zheyu Xin, Huikang Hu +7Overthinking in Language Models Self-Training for Language Models
Southeast University · Noah’s Ark Lab
RIMS: Preference Optimization via Smoothed Multi-pair Aggregation for Small-Scale LLM Retrieval-Augmented Generation
Jul 17, 2026Pei Tian, Zihan Dong, Tianci Liu +2Retrieval-Augmented Generation Small Language Models
Department of Computer Science, Columbia University · Department of Statistics, Rutgers University · School of Electrical and Computer Engineering, Purdue University +1
TD-DPO: Difference-Aware Preference Optimization for Mitigating Sycophancy in Clinical Autism Intervention Dialogue
Jul 17, 2026Shuzhong Lai, Junhong Lai, Chenxi Li +5LLM Sycophancy Direct Preference Optimization
1Nanhu Brain-Computer Interface Institute · 2MOE Frontiers Science Center for Brain and Brain-Machine Integration, Zhejiang University · College of Computer Science and Technology, Zhejiang University +3
Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs
Jul 15, 2026Zhixiao Zheng, Zheren Fu, Zhiyuan Yao +3Multimodal Grounding Multimodal Large Language Models
University of Science and Technology of China Hefei, China · Xiaomi Corporation Beijing, China · State Key Laboratory of Communication Content Cognition, People’s Daily Online Beijing, China
Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet
Jul 15, 2026Xi Yang, Guodong Liu, Chuqin Li +10LLM Alignment Small Language Models
Advanced Intelligence, Eli Lilly and Company, Indianapolis, IN, USA · Department of Biostatistics and Health Data Science, Indiana University, Indianapolis, IN, USA · Center for Biomedical Informatics, Regenstrief Institute, Indianapolis, IN, United States +2
Direct Image-to-Modern Vietnamese Translation of Han-Nom Manuscripts via Multimodal RLHF Preference Alignment
Jul 13, 2026Thi Kim Trang Vo, Nghia Hieu Nguyen, Ha Minh TanDirect Preference Optimization Preference Optimization
University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam
PORTS: Preference-Optimized Retrievers for Tool Selection with Large Language Models
Jul 3, 2026Lorenzo Molfetta, Giacomo Frisoni, Nicolò Monaldini +1Preference Optimization LLM Tool Use
Department of Computer Science and Engineering, University of Bologna
KARMA: Knowledge graph-based Automated Reasoning Materialization and Alignment
Jul 3, 2026Jinkyeong Choi, Chaebin Jeong, Donghyeon ParkContrastive Learning Preference Optimization
Sejong University
Distributionally Robust Listwise Preference Optimization
Jul 2, 2026Xudong Wu, Jian Qian, Pangpang Liu +2LLM Alignment Distributionally Robust Optimization
The University of Hong Kong · Hong Kong SAR · Yale University +3
RSICCLLM: A Multimodal Large Language Model for Remote Sensing Image Change Captioning
Jun 26, 2026Yelin Wang, Zijia Song, Shuo Ye +6Remote Sensing Image Understanding VLM Adaptation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Harbin Institute of Technology, Shenzhen, China · Great Bay University, Dongguan, China +1
MORPH: Generative Retrieval via Diffusion Transformer with Metric-Ordered Sequence Training and Hybrid-Policy Preference Optimization
Jun 25, 2026Chenghao Liu, Yu Zhang, Zhongtao Jiang +8Generative Retrieval Preference Optimization
Peking University
OracleAnalyser: Analysing Implicit Semantics of Oracle Bone Scripts through MLLMs with Post-training
Jun 24, 2026Zijia Song, Yelin Wang, Zhengyi Ma +5VLM Reasoning Preference Optimization
National University of Defense Technology · ShanghaiTech University · Great Bay University +1
Against Proxy Optimization
Jun 22, 2026Sven NethPreference Optimization Decision-Making under Uncertainty
University of Pittsburgh
Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings
Jun 18, 2026Pranav Bhandari, Nicolas Fay, Amitava Datta +2Continual Learning for LLMs Direct Preference Optimization
1Network Analysis and Social Influence Modelling (NASIM) Lab · School of Physics Maths and Computing, The University of Western Australia · School of Psychological Science, The University of Western Australia +1
Temporal Preference Optimization for Unsupervised Retrieval
Jun 16, 2026HyunJin Kim, Jaejun Shim, Young Jin Kim +1Temporal IR Preference Optimization
1Sungkyunkwan University, Suwon, South Korea · 2Microsoft, Redmond, USA
Effects of Objective Normalization on Regions of Interest in Preference-Based Evolutionary Multi-Objective Optimization
Jun 15, 2026Ryuichi Mogami, Ryoji TanabeMulti-Objective Evolutionary Optimization Preference Optimization
Yokohama National University, Yokohama, Japan
PVminerLLM2: Improving Structured Extraction of Patient Voice via Preference Optimization
Jun 15, 2026Samah Fodeh, Linhai Ma, Ganesh Puthiaraju +7Healthcare Clinical Information Extraction
Department of Emergency Medicine, Yale School of Medicine, 464 Congress Ave, 06519, CT, USA · Department of Social and Behavioral Sciences, Yale School of Public Health, 60 College Street, 06520, CT, USA · Division of Research, Texas State University, 601 University Dr., 78666, TX, USA
PolyAlign: Conditional Human-Distribution Alignment
Jun 11, 2026L. D. M. S. Sai Teja, Ufaq Khan, Sathira Silva +2Multilingual Language Models LLM Alignment
NIT Silchar, India · MBZUAI, Abu Dhabi, UAE
Emo-LiPO: Listwise Preference Optimization for Fine-Grained Emotion Intensity Control in LLM-based Text-to-Speech
Jun 11, 2026Yihang Lin, Li Zhou, Congwei Cao +4TTS Synthesis Preference Optimization
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · Agency for Science, Technology and Research +2
SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment
Jun 9, 2026Kaiwen Shi, Zheyuan Zhang, Yanfang YeConfidence Estimation in Language Models Preference Optimization
University of Notre Dame
FoA-SR: Faithful or Aesthetic? Profile-Aware Preference Optimization for Real-World Image Super-Resolution
Jun 9, 2026Amjad Mahdi Alqarni, Peizhong JuDiffusion Models for Image Restoration Preference Optimization
Department of Computer Science University of Kentucky Lexington, KY, USA
ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning
Jun 2, 2026Ziyan Liu, Xueda Shen, Yuzhe Gu +7Efficient Language Model Reasoning Preference Optimization
Shanghai Artificial Intelligence Laboratory · University of Science and Technology of China · MMLab, The Chinese University of Hong Kong
Drifting Preference Optimization for One-Step Generative Models
Jun 1, 2026Zhou Jiang, Yandong Wen, Zhen LiuDiffusion Model Alignment One-Step Generative Modeling
Westlake University · The Chinese University of Hong Kong, Shenzhen
From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging
May 29, 2026Qi Sun, Siyue Zhang, Yulin Chen +3Pairwise Preference Learning Low-Rank Adaptation
NYU Shanghai · NTU · NYU +1
Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
May 27, 2026Beiduo Chen, Pingjun Hong, Ziyun Zhang +3LLM-Assisted Annotation Preference Optimization
U MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Æ Munich Center for Machine Learning (MCML), Munich, Germany · [ University of Vienna, Austria +1

Jul 30, 2026, cs.CVJ/K move · Enter open · S save

Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion

Henglin Liu, Fangyuan Kong, Jing Wang, Yizhou Lin+6

Tsinghua University · Kling Team, Kuaishou Technology · Sun Yat-sen University

Diffusion Model Alignment Video Diffusion Models Text-to-Video Generation Temporal Credit Assignment Preference Optimization Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Recent advances in preference alignment for diffusion-based video generation, particularly via Direct Preference Optimization (DPO), have significantly improved visual quality. However, temporally sparse artifacts such as motion collapse, object flickering, and color oversaturation remain a major barrier to perceptual realism. Existing methods struggle with these issues due to two key limitations: (1) the preference attribution bottleneck, where offline human annotations are costly and fail to accurately capture learning dynamics, while online reward signals are rollout-aware but often unstable and biased; and (2) temporal credit misallocation, where uniformly applied supervision cannot effectively target the brief segments in which artifacts occur. To address these challenges, we propose concentrated Implicit Preference Optimization (cIPO), a post-training framework for video diffusion models. cIPO derives implicit preference signals directly from the denoising process: given a real video, the model adds forward noise and reconstructs it via iterative denoising, treating the original as the preferred sample and the reconstruction as the dispreferred one. This formulation captures inference-time errors without requiring human annotations or external reward models. Moreover, frame-level discrepancies between original and reconstructed videos reveal when failures occur. cIPO leverages this by computing temporal reconstruction errors and concentrating optimization on high-error segments, enabling more precise correction of failure-prone regions. Extensive experiments demonstrate that cIPO consistently enhances video authenticity and temporal coherence across multiple datasets, highlighting the effectiveness and efficiency of implicit preference with temporally concentrated optimization.

Momentum

Related topics

Latest papers 98