Offline Preference Optimization

Follow this topic Ask about this topic

Offline preference optimization trains a model to favor preferred outputs over rejected ones using a fixed dataset, rather than collecting feedback through live interaction. Methods such as Direct Preference Optimization (DPO) and its variants adjust preference weights, add penalties, or select training pairs to improve stability and use the data efficiently. The approach is used for language-model responses, tool-use trajectories, and image generation with diffusion or flow models, sometimes deriving preference strengths from ratings or demonstrations. A key challenge is that offline preferences may be noisy or incomplete: methods can improve in-domain results, but their stability and generalization to new settings remain uncertain.

Latest papers

8 papers0 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

LeWAM: Latent evolving World Action Model
Sep 23, 2026Xueji Fang, Boqiang Duan, Hua Wu +2Joint-Embedding Predictive Architecture Action-Conditioned World Models
Zhejiang University · Westlake University · Baidu Inc.
FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
Sep 9, 2026Yansen Han, Shengyi Liao, Peng Sun +4Pairwise Preference Learning Flow Matching
Westlake University · Zhejiang University · Kling Team, Kuaishou Technology
Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories
Jul 12, 2026Yixiong Chen, Alan YuilleDirect Preference Optimization LLM Agents
Johns Hopkins University
Boosting Direct Preference Optimization with Penalization
Jun 10, 2026Pengwei SunPairwise Preference Learning LLM Alignment
Stanford University.
Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs
May 10, 2026Yunhong Lu, Qichao Wang, Hengyuan Cao +2T2I Generation Rectified Flow
Zhejiang University · Shanghai Institute for Advanced Study-Zhejiang University · Shanghai Institute for Mathematics and Interdisciplinary Sciences
Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models
May 7, 2026Zhikai Li, Yue Zhao, Edward Zhongwei Zhang +4Diffusion Model Alignment Preference Optimization
Institute of Automation, Chinese Academy of Sciences · University of California, Berkeley · University of California, Santa Barbara
Intrinsic Mutual Information as a Modulator for Preference Optimization
Apr 27, 2026Peng Liao, Peijia Zheng, Lingbo Li +2LLM Alignment Offline Preference Optimization
1Sun Yat-sen University · University of Warwick · 3Macao Polytechnic University
Not All Preferences Deserve Gradients: Understanding Gradient Utility in Offline Reasoning Alignment
Feb 1, 2026Hui Wu, Hengyi Cai, Jinman Zhao +6Pairwise Preference Learning Training Data Selection
Baidu Inc. · University of Toronto · School of Intelligence Science and Technology, Peking University +1

Sep 23, 2026J/K move · Enter open · S save

LeWAM: Latent evolving World Action Model

Xueji Fang, Boqiang Duan, Hua Wu, Jingdong Wang+1

Zhejiang University · Westlake University · Baidu Inc.

Joint-Embedding Predictive Architecture Action-Conditioned World Models Robotic Manipulation World Action Models Latent World Models Offline Preference Optimization

Open paper AskPDF arXiv

World Action Models (WAMs) jointly model action generation and environment dynamics and are mostly built on pretrained Video Diffusion Models (VDMs). In VDM-based WAMs, observations are first encoded by a VAE, and the resulting compressed latents are then processed by large video diffusion backbones to extract effective features for action generation. However, this paradigm ties WAM performance and training cost to large-scale video generation pretraining, limiting WAM efficiency and scalability. In this paper, we theoretically and empirically investigate how visual representations affect action generation in WAMs. Our results show that predictive embeddings from Joint-Embedding Predictive Architecture (JEPA) encoders better support action generation than compressed VAE latents, with I-JEPA performing best in our encoder comparison. Based on these findings, we propose LeWAM, which conditions action generation on JEPA embeddings and models environment evolution by predicting future embeddings in the same space, without relying on a video diffusion backbone. We further find that imitation learning matches demonstrated actions but does not distinguish better actions from worse ones, even though small action deviations can greatly affect task success. To address this limitation without additional environment interaction or the human oversight required for resets and safety, we introduce Demonstration-Guided DPO (DemoDPO), an offline preference refinement stage that derives preference supervision directly from demonstrations. With only 0.4B trainable parameters, LeWAM achieves an average success rate of 92.28% on RoboTwin 2.0, comparable to that of state-of-the-art VLAs and WAMs, and maintains practical effectiveness on real-world manipulation tasks.