Generative AI Evaluation

Follow this topic Ask about this topic

Momentum

29 papers in the last four weeks, up 142% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 211

All topics

Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation
Jun 6, 2026Katelyn Xiaoying Mei, Yi-Li Hsu, Minjoon Choi +5Human-in-the-Loop Evaluation Scientific Reproducibility
University of Washington · National Tsing Hua University · Seoul National University +2
'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions
Jun 3, 2026Nils Dycke, Marina Sakharova, Nico Daheim +1AI-Generated Text Detection Generative AI Evaluation
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany · Zuse School ELIZA
A Systematic Analysis of Linguistic Features in AI-Generated Text Detection Across Domains and Models
Jun 2, 2026Yassir El Attar, Esra Dönmez, Maximilian Maurer +1AI-Generated Text Detection Stylometry
Institute for Natural Language Processing, University of Stuttgart · Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart · GESIS Leibniz Institute for the Social Sciences +1
SenseJudge: Human-Centric Preference-Driven Judgment Framework
Jun 2, 2026Rui Li, Junfeng Liu, Xiangwen Kong +2LLM Evaluation LLM-as-a-Judge
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · StepFun · Xi’an Jiaotong University
AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following
Jun 2, 2026Haitao Li, Tian Tan, Yuguang Yang +2Audio-Language Model Evaluation LLM-as-a-Judge
Zhejiang University · Shanghai Innovation Institute · Shanghai Jiao Tong University +1
LL-Bench: Rethinking Low-Level Vision Evaluation in the Era of Large-Scale Generative Models
Jun 1, 2026Lu Liu, Huiyu Duan, Chenxin Zhu +6Image Restoration Image Quality Assessment
Shanghai Jiao Tong University, Shanghai, China
AI as a Tool for Simulation-Based Experiments in Literary Studies
Jun 1, 2026Matthew WilkensComputational Literary Studies Controllable Text Generation
Department of Information Science Cornell University
"I've Seen How This Goes": Characterizing Diversity via Progressive Conditional Surprise
Jun 1, 2026Matthew Khoriaty, David Williams-King, Shi FengDiverse Text Generation Generative AI Evaluation
ERA Fellowship · George Washington University
Child-directed speech facilitates production, not comprehension, in BabyLMs
May 31, 2026Bastian Bunzeck, Sina ZarrießLanguage Model Pretraining LLM Evaluation
Computational Linguistics, Department of Linguistics Bielefeld University, Germany
Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation
May 31, 2026Zhicheng Zhang, Lei Wang, Yu Zhang +1Talking Head Generation Audio-Driven Facial Animation
School of Business, University of New South Wales (UNSW), Australia. · School of Engineering and Built Environment, Griffith University, Australia. · Data61/CSIRO, Australia.
Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions
May 31, 2026Di WuScholarly Peer Review Automated Peer Review
University of Central Florida Department of ECE Orlando, FL, USA
RWGBench: Evaluating Scholarly Positioning in Related Work Generation
May 30, 2026Anzhe Xie, Weihang Su, Jiaxin Mao +4LLM Evaluation Automated Evaluation
Tsinghua University Beijing, China · Renmin University of China Beijing, China
Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation
May 29, 2026Grégoire Martinon, Ibrahim Merad, Mohammed RakiLLM-Assisted Annotation LLM Agent Evaluation
Emerton Data, 16 Avenue Hoche, 75008, Paris, France.
A Persona-Based Evaluation Framework for Pluralistic Alignment in Generative AI
May 29, 2026Atahan KaragozLLM Alignment Persona Consistency
Department of Computer Science University of Basel Basel, Switzerland
Conformal Reliability: A New Evaluation Metric for Conditional Generation
May 29, 2026Yachen Gao, Xinwei Sun, Yikai Wang +4Uncertainty Quantification Conformal Prediction
Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · School of Data Science, Fudan University, Shanghai, China +2
Generative Models and Statistical Validation
May 28, 2026Sascha Diefenbacher, Sofia Palacios Schweitzer, Gregor KasieczkaGenerative Modeling Density Estimation
Institut für Theoretische Physik, Universität Heidelberg, Germany · Physics Division, Lawrence Berkeley National Laboratory, Berkeley, USA · NHETC, Department of Physics & Astronomy, Rutgers University, Piscataway, NJ, USA +1
Temporal Stability and Few-Shot Prompting in Math Task Assessment
May 28, 2026Danielle S. Fox, Brenda L. Robles, Elizabeth DiPietro Brovey +1Generative AI in Education Few-Shot Prompting
Learning Research and Development Center, University of Pittsburgh · Institute for Learning, University of Pittsburgh
Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation
May 28, 2026Haoyue Yang, Zhangxiao Shen, Fan Ding +8Code Generation Evaluation Automated Evaluation
Baidu Inc., Beijing, China
BEAMS: Benchmarking and Evaluating AI for Modeling and Simulation
May 27, 2026Sara Metcalf, William SchoenbergAI for Science Human-in-the-Loop AI
ISEE Systems
The Cases LJP Never Sees: Prosecution Decision Prediction for More Complete Criminal Liability Assessment
May 27, 2026Junyu Lu, Qi Wei, Peishuo Zheng +6AI-Assisted Decision Making Legal Reasoning
1Beijing Institute of Technology, Zhuhai · 2Osaka University · 3Xi’an University of Technology +2
GRADE: Generalizable Reasoning-Aware Dialogue Evaluation for AI Tutors
May 27, 2026Parth Bhalerao, Jeromy Chang, David Chou +1Intelligent Tutoring Systems Low-Rank Adaptation
Santa Clara University
GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human
May 26, 2026Yihang Lin, Yunze Gao, Zeyang Lin +3Benchmark Design Human-in-the-Loop Evaluation
Amap, Alibaba Group · The Chinese University of Hong Kong, Shenzhen
AI evaluation may bias perceptions: The importance of context in interpreting academic writing
May 26, 2026Shang Wu, Randol YaoAI-Generated Text Detection Generative AI Evaluation
UC Irvine · MIT
Plans for Evaluating Structured Generative Search Summaries
May 26, 2026Tetsuya Sakai, Jina Lee, Hanpei Fang +1Language Model Generation Evaluation Summarization Evaluation
Waseda University/Naver Corporation · Tokyo, Japan · Waseda University +2
LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV
May 25, 2026Tengfei Liu, Yang Shi, Xuanyu Zhu +17Audio-Video Generation Long-Horizon Video Generation
Peking University · Kling Team · Nanjing University +6
AI-Assisted Systematization for Evaluating GenAI Systems
May 25, 2026Dhruv Agarwal, Emily Sheng, Chad Atalla +6Generative AI Evaluation
Cornell University · Microsoft Research
Divergent Recommendations, Convergent Diagnoses: Cross-Provider Failure-Mode Convergence in AI Commercial Recommendation
May 22, 2026Will Jack, Noah Lehman, Keller Maloney +1Generative Engine Optimization Large Language Model-Based Recommendation
Unusual
NLG Evaluation: Past, Present, Future
May 22, 2026Ehud ReiterLLM Evaluation Language Model Generation Evaluation
Dept of Computing Science University of Aberdeen Aberdeen, UK
EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation
May 22, 2026Songlin Yang, Haobin Zhong, Ruilin Zhang +23VLM Evaluation Video Quality Assessment
The Hong Kong University of Science and Technology · Tencent · Tsinghua University +5
UniSRM: A Unified Speech Reward Model for Reasoning-Based Fine-grained Assessment
May 22, 2026Yuanyuan Wang, Dongchao Yang, Yayue Deng +4Reward Modeling Speech Generation Evaluation
The Chinese University of Hong Kong · Tsinghua University · Independent Researcher

Jun 6, 2026, cs.CLJ/K move · Enter open · S save

Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation

Katelyn Xiaoying Mei, Yi-Li Hsu, Minjoon Choi, Zongwan Cao+4

University of Washington · National Tsing Hua University · Seoul National University+2

Human-in-the-Loop Evaluation Scientific Reproducibility Generative AI Evaluation

Open paper AskPDF arXiv

Human evaluation plays a critical role in assessing the quality of generated text. However, the reliability and reproducibility of these evaluations depend on transparent and well-documented protocols -- details that are frequently missing in current practice. In this work, we conduct a large-scale analysis of human evaluation protocols for evaluating long-form generation tasks in *CL conference publications from 2023--2025, including a full manual review of 284 papers and LLM-assisted analysis for another 1.8k+ papers. We define a set of 20 reportable criteria related to reproducibility of human evaluation studies, and apply these criteria to systematically examine reporting norms and practices within the community. We find widespread under-reporting of important aspects of human evaluation study design, leading to ambiguity about what was measured and how, who contributed judgments, and how judgments should be interpreted. Based on these findings, we outline actionable recommendations to support more transparent and reproducible reporting in future research. Our analysis code and annotated dataset can be found at: https://github.com/larchlab/Illusions-of-the-Gold-Standard

Momentum

Related topics

Latest papers 211