Overview

Preference optimization uses comparisons, rankings, or desirability signals to steer models toward outputs people or task-specific evaluators prefer. Common approaches include reinforcement learning from human feedback, Direct Preference Optimization (DPO) and variants such as KTO, as well as listwise, group-wise, and token-level objectives; newer methods also use online feedback, synthetic preferences, and task-specific rewards. Work spans language-model alignment on instruction-following and reasoning benchmarks, as well as image, audio, recommendation, translation, and robotic systems. A central challenge is that preferences can be noisy, inconsistent, or specific to a user or context, while optimizing proxy rewards can cause unwanted behavior or reduce the model’s existing capabilities.

Latest papers

CardsList

164 papers10 papers in the last four weeks, down 58% on the four weeks before. 0.1% of all new papers.

  1. ParetoM3^3: Learning on the Pareto Set under Preference Guidance via Min-Max-Min Optimization

    Nov 13, 2026Pei Tang, Songtao LuMulti-Objective OptimizationPareto Optimization

  2. Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment

    Nov 7, 2026Qisai Liu, Zhanhong Jiang, Zahid Hasan +1LLM AlignmentPreference Optimization

  3. When Scores Conflict with Preferences: Pre-Alignment Diagnosis and Adaptive Correction of Constraint Drift in DPO

    Nov 5, 2026Ping Liu, Yan YanDirect Preference OptimizationPreference Optimization

  4. Aligning Language Models with General Preferences: An Empirical Study of Direct Nash Optimization

    Nov 3, 2026Austin Tsai, Saurav Nanda, Ganapathy Parthasarathy +2LLM AlignmentPreference Optimization

  5. Aligning Medical Vision Language Models at the Capability Boundary with Entropy Adaptive Rubric Grounding

    Oct 29, 2026Hongyu Bai, Jiaqi Zeng, Chunzheng ZhuMedical VLMsReasoning for VLMs

  6. Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization

    Oct 26, 2026Christian Moya, Purav Matlia, Elliott Thornley +1Preference OptimizationSpurious Correlation Robustness

  7. RubricOpt: Self-Revision of Semantic Rubrics for Preference Optimization

    Oct 21, 2026Wei Chen, Yubing Wu, Shigui Li +7Rubric-Guided GenerationPreference Optimization

  8. TeachBack: Learning to Teach via Bidirectional Preference Optimization for On-Policy Self-Distillation

    Oct 21, 2026Linfei Ji, Jiayun Li, Haozhao Wang +5Language Model DistillationOn-Policy Self-Distillation

  9. Divergence based Ranked Preference Alignment for LLMs

    Oct 18, 2026Rajdeep Haldar, Yue Xing, Qifan SongLLM AlignmentPreference Alignment

  10. SteadyThought: Mitigating LLM Under-Thinking via Thought-Level Preference Optimization

    Oct 18, 2026Zhenyue Peng, Lin Yang, Jie Wang +7Efficient Language Model ReasoningAdaptive Reasoning in Language Models

  11. Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison

    Oct 13, 2026Pratuat Amatya, Vinay Setty, V Venktesh +1LLM AlignmentFactual Knowledge in Language Models

  12. Prompt or Policy? Optimizing Physician Slate Presentations with LLM User Simulators

    Oct 13, 2026Andrei Niculae, Adrian Cosma, Emilian RadoiAutomatic Prompt OptimizationLarge Language Model-Based Recommendation

  13. FrameLite: Decision-Oriented Aesthetic Cropping with a Compact Vision-Language Model

    Oct 13, 2026Jianze Li, Kai Liu, Ziqing Zhang +4Image EditingVision-Language Models

  14. Uniform Information Density-Based Preference Optimization

    Oct 10, 2026Yongyuan Li, Yang XuPreference OptimizationLLM Alignment

  15. Evolving Creatives with Clicks: CTR-Driven Advertising image Editing via Attribute Disentanglement and Preference Alignment

    Oct 9, 2026Xu Zhang, Haohan Wang, Zheng Zhang +9Image EditingText-Guided Image Editing

  16. Emo-BPO: Emotion Bidirectional Preference Optimization for Diffusion-based Emotional TTS

    Oct 9, 2026Jiacheng Shi, Hongfei Du, Xinyuan Song +3Preference Optimization for Diffusion ModelsEmotional Speech Synthesis

  17. RobotAPO: Adversarial Physics Preference Optimization for Robotic Manipulation Video Generation

    Oct 7, 2026Kerui Li, Zhe Jing, Chenyi Huang +4Physical Consistency in Video GenerationRobotic Manipulation

  18. AuraSE: Low-Hallucination Generative Speech Enhancement via Multimodal Flow Matching and Inference Policy Optimization

    Oct 5, 2026Yingda Shen, Yao Qian, Yuxuan Hu +7Inference-Time OptimizationAudio Diffusion Models

  19. Geometry-Aware Preference Optimization for Text-to-Image Diffusion Models

    Oct 4, 2026Lei Wang, Zhen Wang, Yuexiang Xie +1Preference Optimization for Diffusion ModelsDiffusion Model Alignment

  20. Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

    Oct 4, 2026Minjae Lee, Kyunghyun Cho, Sangdon ParkLLM AlignmentMulti-Objective Language Model Alignment

  21. Exploration-Preserving Policy Optimization

    Oct 2, 2026Hangzhan jin, Mohammad Hamdaqa, Doina PrecupRL for Language Model ReasoningRL Exploration

  22. Rubric-Based Optimization for Text-to-Music Generation

    Oct 2, 2026Ping Wang, Guang Yang, Shao-Rong Su +3Text-to-Music GenerationRubric-Based RL

  23. PsyEvo: A Personalized Counseling Agent That Self-Evolves at Test Time

    Oct 2, 2026Yuting Yan, Shihao Xu, Junhao Yu +6LLM Agent Self-ImprovementMental Health Counseling

  24. Mitigating Social Sycophancy via Pluralistic Preference Optimization

    Oct 1, 2026Stephane Hatgis-Kessell, Myra Cheng, Xiaoxuan Hou +4LLM SycophancyPreference Optimization

  25. GAW-PO: Preference Optimization with Gradient-Aligned Token Weights

    Oct 1, 2026Andreea Dutulescu, Stefan Ruseti, Mihai Masala +2LLM AlignmentDirect Preference Optimization