Human Preference Evaluation

Overview

Human preference evaluation studies how people judge AI outputs, from choosing between two responses to assigning scores or explaining what they value. Common approaches include pairwise comparisons aggregated with Bradley–Terry models, rubric-based ratings, and preference data used to train reward models; benchmarks apply these methods to settings such as writing, coding, speech, medicine, and creative work. LLM judges and other automated evaluators can make assessment cheaper, but their agreement with people varies by task, modality, and evaluation criteria. A central challenge is representing genuine differences among people while detecting biases and avoiding treating preference as a substitute for qualities such as factual accuracy or safety.

Latest papers

CardsList

60 papers6 papers in the last four weeks, down 25% on the four weeks before. 0.0% of all new papers.

  1. Scalable Audio-LLM System for Speech MOS Evaluation: A Diagnostic Case Study and Adaptation Roadmap

    Oct 30, 2026Dimitrios Kokkinis, Panagiotis Kakoulidis, Nikolaos Ellinas +10Speech Quality AssessmentTTS Evaluation

  2. Predicting Human-Gain Curves under Local Proxy Optimization from Repeated Ratings

    Oct 30, 2026Moonwon Choi, Kisung Nam, Seunggeun LeeLanguage Model Performance PredictionHuman Preference Evaluation

  3. Beauty Is Human: When People Love AI-Generated Art, They Think a Human Made It

    Oct 28, 2026Zihan Chen, Di Zhu, Lei Nico ZhengAuthorship AttributionHuman Preference Evaluation

  4. ConformalMOS: Uncertainty-Aware MOS Prediction with Conformal Intervals and Ordinal Modeling

    Oct 26, 2026Kehinde Elelu, Joshua E. Siegel, Mohammadali Saffary +3Speech Quality AssessmentTTS Evaluation

  5. Attractive Faces Skew Text-to-Image Reward Models and Leaderboards

    Oct 20, 2026Harvey Mannering, Zhiwu Huang, Adam Prügel-BennettT2I Generation EvaluationHuman Preference Evaluation

  6. PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas

    Oct 16, 2026Haoming Li, Defu Cao, Xu Hu +23Computer-Use Agent EvaluationComputer-Use Agent Benchmarks

  7. Improving the Evaluation of Biomedical Question Answering with Few Human Preference Signals

    Oct 12, 2026Shakila Mahjabin Tonni, Xiang Dai, Sarvnaz Karimi +3RAG EvaluationAutomated Evaluation

  8. Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization

    Oct 10, 2026Changkun Ou, Sven Mayer, Daniel Buschek +1Human-in-the-Loop AIPreference Elicitation

  9. How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?

    Sep 30, 2026Nahomi Kusunoki, Tsubasa Ochiai, Naohiro Tawara +4Human Preference EvaluationSpeech Quality Assessment

  10. JudgeProfile: Understanding and Steering Subjectivity in LLM Judges

    Sep 29, 2026Qi Cao, Kangning Liu, Xuan Kan +10Human Preference EvaluationLLM Evaluation

  11. Multi-Dimensional Comparative Scale Construction for Efficient Personalized Subjective Judgment in High-Traffic Applications

    Sep 27, 2026Xianglong Shi, Shifeng Liu, Sirui Zhao +2Human Preference EvaluationLearning to Rank

  12. PANEL: An Open-Source, Self-Hosted Web Platform for Human Evaluation of Generative Models

    Sep 25, 2026Matteo Spanio, Andrea Poltronieri, Martín RocamoraHuman-in-the-Loop EvaluationGenerative AI Evaluation

  13. The VoiceMOS Challenge 2026: Evaluating Speech Enhancement, Emotional TTS and Accented TTS Systems

    Sep 12, 2026Wen-Chin Huang, Wei Wang, Marvin Sach +4Speech Quality AssessmentTTS Evaluation

  14. Is Semantics Enough for Speech Mean Opinion Score Prediction?

    Sep 3, 2026Tianyu Lan, Yufei Shi, Yang Ai +3Human Preference EvaluationNeural Audio Codecs

  15. ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs

    Aug 13, 2026Jiale Cui, Yueyao Yuan, Kaixi Zhong +3Human Preference EvaluationAI Agent Benchmarks

  16. How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans

    Aug 10, 2026Hasan Mahmud, Khawaja Abaid Ullah, Mohammad Javad Khojasteh +2Human Preference EvaluationGender Bias in Language Models

  17. Beyond a Single Judge: The Evidence-Grounded, Social-Weighted Persona Panel for Generative UI Evaluation

    Jul 30, 2026Zheng Wu, Yibo Luo, Pu Zhang +2Human Preference EvaluationLLM-as-a-Judge

  18. Human Preference aligned Tabular Similarity

    Jul 27, 2026Frederik Hoppe, Astrid Franz, Marianne Michaelis +2Human Preference EvaluationTabular Representation Learning

  19. Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation

    Jul 20, 2026Jiabing Yang, Yixiang Chen, Yuan Xu +6Human Preference EvaluationPairwise Preference Evaluation

  20. Human-in-the-Loop User Feedback Affects Perceived Accuracy and Trust, but Task Subjectivity Matters

    Jul 20, 2026Donald R. Honeycutt, Mahsan Nourani, Eric D. RaganHuman Preference EvaluationHuman-in-the-Loop Evaluation

  21. Genre Bias or Aesthetic Perception? Identifying and Mitigating Shortcut Learning in Music Evaluation

    Jul 15, 2026Yizhou Zhang, Wangjin Zhou, Yi Zhao +3Human Preference EvaluationAlgorithmic Bias

  22. Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

    Jul 6, 2026Robert Morabito, Tyler McDonald, Charitra Viswanath +4Human Preference EvaluationLLM Evaluation

  23. LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform

    Jul 1, 2026Ruotong Zhao, Zhiyu Chen, Xurui Liu +7Human Preference EvaluationLLM-as-a-Judge

  24. The Human Creativity Benchmark

    Jun 29, 2026Aspen Hopkins, Allison Nulty, Alexandria Minetti +2Human Preference EvaluationCreativity Assessment