Preference Alignment

Preference alignment aims to make AI outputs reflect what people value, from helpful and safe language-model responses to personalized recommendations, generated images, and robot behavior. Common approaches learn reward models from pairwise judgments and optimize with reinforcement learning from human feedback (RLHF), or use direct preference optimization (DPO); other methods steer models at inference time or adapt them to individual users and multiple objectives. Evaluations use human judgments and task-specific benchmarks, including language-model preference tests and image-generation comparisons. A central challenge is that preferences can be noisy, conflicting, culturally varied, or change over time, while optimizing imperfect reward signals can introduce bias, reduce capabilities, or reward behavior people do not actually want.