Human Preference Evaluation

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 41

All topics

How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?
Sep 30, 2026Nahomi Kusunoki, Tsubasa Ochiai, Naohiro Tawara +4Human Preference Evaluation Speech Quality Assessment
Waseda University, Japan · NTT, Inc., Japan
JudgeProfile: Understanding and Steering Subjectivity in LLM Judges
Sep 29, 2026Qi Cao, Kangning Liu, Xuan Kan +10Human Preference Evaluation LLM Evaluation
Meta · University of California San Diego · The University of Hong Kong +1
PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators
Sep 28, 2026Cheng Chang, Yining Mao, Peng QiHuman Preference Evaluation LLM Evaluation
Uniphore
Multi-Dimensional Comparative Scale Construction for Efficient Personalized Subjective Judgment in High-Traffic Applications
Sep 27, 2026Xianglong Shi, Shifeng Liu, Sirui Zhao +2Human Preference Evaluation Learning to Rank
University of Science and Technology of China · University of Electronic Science and Technology of China
Is Semantics Enough for Speech Mean Opinion Score Prediction?
Sep 3, 2026Tianyu Lan, Yufei Shi, Yang Ai +3Human Preference Evaluation Neural Audio Codecs
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China
Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation
Aug 31, 2026Minsoo Song, Chanwoo Kim, Sugyeong Eo +1Human Preference Evaluation LLM-as-a-Judge
Soongsil University · Yonsei University
ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs
Aug 13, 2026Jiale Cui, Yueyao Yuan, Kaixi Zhong +3Human Preference Evaluation AI Agent Benchmarks
School of Software Technology, Zhejiang University
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
Aug 12, 2026Xi Chen, Jie Mu, Mo Xuan +1Human Preference Evaluation LLM-as-a-Judge
Ant Group
Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
Aug 10, 2026Oluwanifemi Bamgbose, Simon Rosen, Jash Shah +6Audio-Language Model Evaluation Human Preference Evaluation
ServiceNow
How Do Large Language Models Judge Social Attraction? Evidence from Theory-Grounded Persona Ratings Across Multiple LLMs and Humans
Aug 10, 2026Hasan Mahmud, Khawaja Abaid Ullah, Mohammad Javad Khojasteh +2Human Preference Evaluation Gender Bias in Language Models
School of Communication, Rochester Institute of Technology · Department of Electrical and Microelectronic Engineering, Kate Gleason College of Engineering, Rochester Institute of Technology Rochester, NY 14623, USA
Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation
Aug 5, 2026Divyansh SinghHuman Preference Evaluation LLM Evaluation
University of Florida
Beyond a Single Judge: The Evidence-Grounded, Social-Weighted Persona Panel for Generative UI Evaluation
Jul 30, 2026Zheng Wu, Yibo Luo, Pu Zhang +2Human Preference Evaluation LLM-as-a-Judge
School of Computer Science, Shanghai Jiao Tong University · 2ByteDance Inc
Human Preference aligned Tabular Similarity
Jul 27, 2026Frederik Hoppe, Astrid Franz, Marianne Michaelis +2Human Preference Evaluation Tabular Representation Learning
CONTACT Software GmbH, Bremen, Germany
Style over Substance: A Shortcut Audit of Emotion-Description Preference Evaluation
Jul 20, 2026Jiabing Yang, Yixiang Chen, Yuan Xu +6Human Preference Evaluation Pairwise Preference Evaluation
University of Chinese Academy of Sciences Institute of Automation, Chinese Academy of Sciences Beijing, China
Human-in-the-Loop User Feedback Affects Perceived Accuracy and Trust, but Task Subjectivity Matters
Jul 20, 2026Donald R. Honeycutt, Mahsan Nourani, Eric D. RaganHuman Preference Evaluation Human-in-the-Loop Evaluation
University of Florida, USA · Northeastern University, USA
Genre Bias or Aesthetic Perception? Identifying and Mitigating Shortcut Learning in Music Evaluation
Jul 15, 2026Yizhou Zhang, Wangjin Zhou, Yi Zhao +3Human Preference Evaluation Algorithmic Bias
Graduate School of Informatics, Kyoto University, Japan · WXG, Tencent, China
Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance
Jul 6, 2026Robert Morabito, Tyler McDonald, Charitra Viswanath +4Human Preference Evaluation LLM Evaluation
Brock University · Emory University · University of Southern California +2
LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform
Jul 1, 2026Ruotong Zhao, Zhiyu Chen, Xurui Liu +7Human Preference Evaluation LLM-as-a-Judge
Tsinghua University, Beijing, China · Zhongguancun Academy, Beijing, China · Zhongguancun Institute of AI, Beijing, China +2
The Human Creativity Benchmark
Jun 29, 2026Aspen Hopkins, Allison Nulty, Alexandria Minetti +2Human Preference Evaluation Creativity Assessment
Massachusetts Institute of Technology Cambridge, MA, USA · Contra New York, New York, USA
Investigating Human-Model Discrepancies in Speech Quality Assessment via Acoustic and Prosodic Perturbations
Jun 18, 2026Masato Takagi, Masaya Kawamura, Reo Shimizu +1Human Preference Evaluation Speech Generation Evaluation
Nagoya Institute of Technology, Japan · LY Corporation, Japan
Humor Style Drives Laughter, Topic Shapes Acceptability: Evaluating Bilingual Personal and Political Robot-Delivered AI Jokes
Jun 11, 2026Anna-Maria Velentza, Anne-Gwenn BosserHuman Preference Evaluation Human-Robot Interaction
Univ Brest-Bretagne INP, COMMEDIA team, Lab-STICC CNRS UMR 6285, France
LLMs Can Better Capture Human Judgments--With the Right Prompts
Jun 10, 2026Danica Dillion, Chen Cecilia Liu, Baihui Wang +5Human Preference Evaluation LLM Alignment
Complexity Science Hub, Vienna · The Ohio State University · University of Cambridge +2
Re-Centering Humans in LLM Personalization
Jun 4, 2026Lechen Zhang, Jiarui Liu, Tal AugustHuman Preference Evaluation LLM Evaluation
†University of Illinois Urbana-Champaign · ♯Carnegie Mellon University
A Dataset for Dynamic Human Preferences for Vision Language Models
Jun 2, 2026Hannah Gao, Dylan Hadfield-Menell, Rachel MaVLM Evaluation Human Preference Evaluation
Massachusetts Institute of Technology · MIT Cambridge, MA
Community-Aware Assessment of Social Textual Engagement and Resonance: A Human-Centric Perspective on User-Generated Content Evaluation
Jun 1, 2026Tianjiao Li, Kai Zhao, Xiang Li +2Human Preference Evaluation Video Quality Assessment
Bilibili Index LLM Team
Pairwise Reference Alignment as a Model-Level Ordinal Observable
May 29, 2026Mujing LiHuman Preference Evaluation LLM Evaluation
Independent Researcher
Personalized Turn-Level User Conversation Satisfaction Benchmark
May 28, 2026Zhefan Wang, Zhiqiang Guo, Weizhi Ma +3Human Preference Evaluation LLM Personalization
Department of Computer Science and Technology, Tsinghua University, Beijing, China. · Institute for AI Industry Research, Tsinghua University, Beijing, China. · Meituan.
Preferred, Not Safer: Pairwise Preference Is a Poor Proxy for Clinical Safety
May 25, 2026Fay Elhassan, David Sasu, Alexandra Kulinkina +2LLM Safety Benchmarks Human Preference Evaluation
LiGHT Laboratory, EPFL, Lausanne, Switzerland
JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment
May 24, 2026Russell Yang, Ruishi Chen, Pierce Kelaita +6Human Preference Evaluation LLM Evaluation
Stanford University · Snorkel AI · Harvey
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
May 19, 2026Zhenyu Li, Aleksandar Cvejic, Zehui Chen +1Pairwise Preference Learning Human Preference Evaluation
KAUST · USTC

Sep 30, 2026, cs.SDJ/K move · Enter open · S save

How Reliable Are Predicted MOS for Reproducing Human System-Level Preferences in Speech Enhancement?

Nahomi Kusunoki, Tsubasa Ochiai, Naohiro Tawara, Marc Delcroix+3

Waseda University, Japan · NTT, Inc., Japan

Human Preference Evaluation Speech Quality Assessment

Open paper AskPDF arXiv

We investigate whether predicted Mean Opinion Scores (MOS) can reliably support system-level comparisons of speech enhancement (SE) methods by introducing system-level preference accuracy (SPA). Although MOS prediction models are widely used to evaluate SE systems, their performance is typically assessed by correlation with human-rated MOS, which does not guarantee agreement on which system is better. SPA addresses this gap by directly evaluating whether predicted and human-rated MOS yield the same system preferences. Using SPA, we systematically evaluate three settings: single prediction models, ensembling, and domain adaptation. Through experiments, SPA varies substantially across single prediction models, from 9.4% to 76.8%. Even the best model disagrees with human judgments in approximately 23% of system comparisons. Ensembling yields only limited improvement, while domain adaptation tends to substantially improve SPA in the closed condition but brings only modest gains in the more practical open condition, where neither the target systems nor the speakers are known. These results suggest that SPA can reveal errors correlation-based evaluation alone does not expose, and that predicted MOS alone can lead to unreliable conclusions in practical SE system comparison.

Momentum

Related topics

Latest papers 41