Preference Optimization

Preference optimization uses comparisons, rankings, or desirability signals to steer models toward outputs people or task-specific evaluators prefer. Common approaches include reinforcement learning from human feedback, Direct Preference Optimization (DPO) and variants such as KTO, as well as listwise, group-wise, and token-level objectives; newer methods also use online feedback, synthetic preferences, and task-specific rewards. Work spans language-model alignment on instruction-following and reasoning benchmarks, as well as image, audio, recommendation, translation, and robotic systems. A central challenge is that preferences can be noisy, inconsistent, or specific to a user or context, while optimizing proxy rewards can cause unwanted behavior or reduce the model’s existing capabilities.