Human Preference Evaluation

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 41

All topics

Why Expert Alignment Is Hard: Evidence from Subjective Evaluation
May 6, 2026Tzu-Mi Lin, Wataru Hirota, Tatsuya Ishigaki +2Human Preference Evaluation LLM Alignment
National Yang Ming Chiao Tung University, Taiwan · Stockmark, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan +1
WorldJen: An End-to-End Multi-Dimensional Benchmark for Generative Video Models
May 5, 2026Karthik Inbasekar, Guy Rom, Omer ShlomovitsHuman Preference Evaluation Video Quality Assessment
The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers
Apr 27, 2026Benjamin Minhao Chen, Xinyu XieHuman Preference Evaluation AI Alignment
The University of Hong Kong, Hong Kong, China
Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement
Apr 24, 2026Wataru Hirota, Tomoki Taniguchi, Tomoko Ohkuma +6Human Preference Evaluation Annotator Disagreement
1Stockmark Inc · 2Asahi Kasei Corporation · National Institute of Informatics +1
Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages
Apr 23, 2026Srija Anand, Ashwin Sankar, Ishvinder Sethi +10Pairwise Preference Learning Human Preference Evaluation
Indian Institute of Technology Madras, India · AI4Bharat, India · Josh Talks, India
Rater State Bias in RLHF Preference Data: An Audit Framework
Apr 14, 2026Elena Kopteva, Vitaliy Hlynianyi-ZhukHuman Preference Evaluation Human-in-the-Loop Annotation
The Grainger College of Engineering, Department of Physics & Illinois Center for Advanced Studies of the Universe, University of Illinois Urbana-Champaign, Urbana, Illinois 61801, USA · Faculty of Applied Mathematics, Oles Honchar Dnipro National University, Dnipro, 49045, Ukraine · Department of Clinical Psychology, Kyiv Institute of Modern Psychology and Psychotherapy, Kyiv, 01133, Ukraine
Why That Robot? A Qualitative Analysis of Justification Strategies for Robot Color Selection Across Occupational Contexts
Mar 30, 2026Jiangen He, Wanqi Zhang, Jessica K. BarfieldHuman Preference Evaluation Human-Robot Interaction
The University of Tennessee, Knoxville, Tennessee, USA · University of Kentucky, Lexington, Kentucky, USA
Mediocrity is the key for LLM as a Judge Anchor Selection
Mar 17, 2026Shachar Don-Yehiya, Asaf Yehudai, Leshem Choshen +1LLM-as-a-Judge Pairwise Preference Evaluation
1The Hebrew University of Jerusalem · 2IBM Research · 3MIT +1
Evaluating Alignment of Behavioral Dispositions in LLMs
Feb 11, 2026Amir Taubenfeld, Zorik Gekhman, Lior Nezry +8Human Preference Evaluation LLM Alignment
Google Research · Hebrew University · University of Cambridge
Expert Preference-based Evaluation of Automated Related Work Generation
Aug 11, 2025Furkan Şahinuç, Subhabrata Dutta, Iryna GurevychHuman Preference Evaluation LLM-as-a-Judge
Ubiquitous Knowledge Processing Lab (UKP Lab) · Department of Computer Science and Hessian Center for AI (hessian.AI) · Technical University of Darmstadt +1
Harmonious Color Pairings: Insights from Human Preference and Natural Hue Statistics
Aug 3, 2025Ortensia Forni, Alexandre Darmon, Michael BenzaquenHuman Preference Evaluation Human Preference Modeling
Econophysics Lab, Institut Louis Bachelier, 28 Pl. de la Bourse, Palais Brongniart, 75002 Paris, France · LadHyX UMR CNRS 7646, École Polytechnique, Institut Polytechnique de Paris, 91128 Palaiseau Cedex, France · Art in Research, 33 rue Censier, 75005 Paris, France +1

May 6, 2026, cs.CLJ/K move · Enter open · S save

Why Expert Alignment Is Hard: Evidence from Subjective Evaluation

Tzu-Mi Lin, Wataru Hirota, Tatsuya Ishigaki, Lung-Hao Lee+1

National Yang Ming Chiao Tung University, Taiwan · Stockmark, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan+1

Human Preference Evaluation LLM Alignment Human-in-the-Loop Evaluation

Open paper AskPDF arXiv

Aligning large language models with expert judgment is especially difficult in subjective evaluation tasks, where experts may disagree, rely on tacit criteria, and change their judgments over time. In this paper, we study expert alignment as a way to understand this difficulty. Using expert evaluations and follow-up questionnaires, we examine how different forms of expert information affect alignment and what this reveals about subjective judgment. Our findings show four consistent patterns. First, alignment difficulty varies substantially across experts, suggesting that expert evaluation styles differ widely in their distance from a model's prior behavior. Second, explicit criteria and reasoning do not always improve alignment, indicating that expert judgment is not fully captured by verbalized rules. Third, editing is sensitive to both the number and the identity of examples, with small numbers of edits providing useful but unstable gains. Fourth, alignment difficulty differs across evaluation dimensions: dimensions grounded more directly in proposal content are easier to align, while dimensions requiring external knowledge or value-based judgment remain harder. Taken together, these results suggest that expert alignment is difficult not only because of model limitations, but also because subjective evaluation is inherently heterogeneous, partly tacit, dimension-dependent, and temporally unstable.

Momentum

Related topics

Latest papers 41