Human Preference Evaluation

Human preference evaluation studies how people judge AI outputs, from choosing between two responses to assigning scores or explaining what they value. Common approaches include pairwise comparisons aggregated with Bradley–Terry models, rubric-based ratings, and preference data used to train reward models; benchmarks apply these methods to settings such as writing, coding, speech, medicine, and creative work. LLM judges and other automated evaluators can make assessment cheaper, but their agreement with people varies by task, modality, and evaluation criteria. A central challenge is representing genuine differences among people while detecting biases and avoiding treating preference as a substitute for qualities such as factual accuracy or safety.